| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-01-05 | |||
| 16:29:06 | dtp | no you are good. let me try | |
| 16:29:08 | dtp | thank you | |
| 16:29:59 | openstackgerrit | Alex Szarka proposed openstack/nova: Transform instance.rebuild.error notification https://review.openstack.org/396621 | |
| 16:34:21 | dansmith | dtp: if you click on a line of code it will open you up the option to type, and/or hit reply on someone else's comment | |
| 16:34:45 | dtp | cool. thanks | |
| 16:35:32 | jaypipes | mriedem: mornin. what did I miss from jlvillal? | |
| 16:35:43 | mriedem | jaypipes: oh tons of fun | |
| 16:35:44 | mriedem | sec | |
| 16:35:59 | mriedem | jaypipes: two things, https://review.openstack.org/#/c/416810/ | |
| 16:36:07 | mriedem | and https://review.openstack.org/#/c/416822/ | |
| 16:36:26 | mriedem | we were over in the ironic channel debugging some scheduler/resource tracker failures for a couple of hours last night, | |
| 16:36:29 | mriedem | and those were the resulting fixes | |
| 16:36:52 | jaypipes | mriedem: ok, cool. will review right away. | |
| 16:36:56 | mriedem | the 401 issue was from the keystone v3 only job which has been fixed in devstack already | |
| 16:37:11 | mriedem | but every time the RT ran to talk to the placement API we got a 401 and weren't disabling that check | |
| 16:37:21 | jaypipes | I see | |
| 16:37:39 | mriedem | the other thing was a latent race in the scheduler host manager stuff where it picks up newly created compute nodes from the db before they have free_disk_gb set | |
| 16:37:55 | jaypipes | ok | |
| 16:38:20 | mriedem | seems we shouldn't even pull those types of compute nodes from the database in the first place, but i'm not very familiar with that host manager code | |
| 16:39:19 | jaypipes | mriedem: odd that the scheduler report client would affect compute_nodes.free_disk_gb, considering the scheduler report client never looks at that field. | |
| 16:39:25 | mriedem | it's not that | |
| 16:39:37 | jaypipes | I'm reading your bug report comment. | |
| 16:39:40 | mriedem | update_resource_stats in the scheduler report client calls compute_node.save(), | |
| 16:39:46 | jaypipes | right | |
| 16:39:49 | mriedem | jaypipes: it's probably easier to just read the commit message | |
| 16:39:58 | jaypipes | :) trying to do that now... | |
| 16:39:58 | mriedem | as the bug report is a bit scatterbrained during debug | |
| 16:43:54 | sahid | sfinucan: hum :/ i commented i think we can't remove that code so i would like you to remove the TODO | |
| 16:44:26 | jaypipes | sahid: did you see my review comment about renaming cpu_overhead to cpus_reserved? | |
| 16:44:53 | sahid | jaypipes: yes, yes i did not get time to do it so far | |
| 16:45:15 | jaypipes | sahid: no worries, just wanted to make sure it didn't get lost in the rebases and revisions :) | |
| 16:45:39 | sahid | no, no don't worry - i wanted to ask you if you want me to change the name only for the field or everywhere ? | |
| 16:46:28 | sahid | well i guess everywhere :) | |
| 16:47:28 | mriedem | sdague: you may also enjoy https://review.openstack.org/#/c/416810/ | |
| 16:48:29 | sfinucan | sahid: Ugh, yeah. Whoever thought it was a good idea to put JSON in the database :( I'll do that now | |
| 16:49:54 | sahid | sfinucan: yes it's a particularity of the numa part | |
| 16:50:26 | jaypipes | mriedem: +2 on both patches. nice work with jlvillal and JayF. | |
| 16:50:34 | mriedem | thanks | |
| 16:50:41 | JayF | mriedem did most of the work | |
| 16:50:42 | jaypipes | sahid: yes, everywhere :) | |
| 16:50:44 | JayF | I just followed along in irc | |
| 16:50:46 | JayF | lol | |
| 16:51:07 | jlvillal | jaypipes: Oh I only started the conversation. mriedem and JayF did all the heavy lifting. I had to go do a 1:1 with my manager and then leave for our holiday party :) | |
| 16:51:08 | openstackgerrit | Matt Riedemann proposed openstack/nova-specs: Return uuid from os-aggregates API (spec) https://review.openstack.org/415511 | |
| 16:51:10 | mriedem | sdague: added the example response ^ | |
| 16:51:25 | jlvillal | Big thanks to mriedem and JayF :D | |
| 16:51:47 | mriedem | supporting me talk to myself through debugging a problem is half the battle :) | |
| 16:52:22 | mriedem | johnthetubaguy: want to re-apply your +2 to https://review.openstack.org/#/c/415511/ ? i just added an api response example for GET | |
| 16:52:34 | openstackgerrit | Jiri Suchomel proposed openstack/nova: Retry few times when creating the initial connection to libvirtd. https://review.openstack.org/416928 | |
| 16:53:23 | jaypipes | jlvillal: holiday party definitely more important. | |
| 16:53:28 | jlvillal | heh :) | |
| 16:53:30 | jaypipes | :) | |
| 16:54:32 | JayF | mriedem: quack? | |
| 16:55:56 | openstackgerrit | Stephen Finucane proposed openstack/nova: Fix backwards compatibility for InstanceNUMACell https://review.openstack.org/396184 | |
| 17:02:34 | dansmith | mriedem: houston, we have a problem | |
| 17:02:44 | mriedem | noooooo | |
| 17:02:50 | mriedem | i hope it's just about that regex :) | |
| 17:02:56 | openstackgerrit | Derek Higgins proposed openstack/nova: libvirt: fix nova can't delete the instance with nvram https://review.openstack.org/357190 | |
| 17:03:37 | dansmith | mriedem: so, I need to go dig, but.. there's something about cellsv1 that means we don't have hosts in the database to do the cell mappings when we're doing the rest of it | |
| 17:03:51 | dansmith | I remember something very faintly about it, but I don't know the details.. | |
| 17:04:06 | dansmith | I'm thinking it's something like we won't get hosts reported to the api cell until after we've been running for a bit or something | |
| 17:04:14 | dansmith | which will be a major wrench in the works | |
| 17:04:43 | mriedem | so this is for the nova-manage discover_hosts command? | |
| 17:04:49 | mriedem | by hosts do you mean compute node records? | |
| 17:05:17 | dansmith | well, map_cell_and_hosts, but same difference yeah | |
| 17:06:04 | mriedem | do we get the cell mapping record? | |
| 17:06:15 | dansmith | no, it fails early because no hosts | |
| 17:06:30 | dansmith | https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L1244-L1247 | |
| 17:06:39 | jaypipes | looks like dansmith needs https://twitter.com/jaypipes/status/801422826402148353 | |
| 17:07:15 | mriedem | jaypipes: if that's intel nfv ci it's a known broken i think | |
| 17:07:15 | dansmith | won't help unless I'm rechecking to see if anyone else has fixed it | |
| 17:07:34 | jaypipes | mriedem: intel-nfv-ci, xen ci, the list goes on :) | |
| 17:08:05 | mriedem | wznoinsk: i've removed intel nfv ci from voting on nova, let me know when i can reenable it | |
| 17:08:19 | jaypipes | dansmith, mriedem: here, this'll cheer you up: https://twitter.com/jaypipes/status/801233017993719808 | |
| 17:08:23 | dansmith | mriedem: I'm almost wondering if we never get those records up in the api node, actually | |
| 17:08:25 | dansmith | johnthetubaguy: around? | |
| 17:09:51 | dansmith | mriedem: I guess I should see how bad it would be to continue to support the old way of listing just for cellsv1 and measure that against the pain of a real fix | |
| 17:10:48 | mriedem | ok i see http://logs.openstack.org/85/416585/4/experimental/gate-tempest-dsvm-cells-ubuntu-xenial/fb5df4f/logs/devstacklog.txt.gz#_2017-01-05_15_37_14_035 | |
| 17:10:48 | jaypipes | bauzas: around? you handling the rebase of https://review.openstack.org/392569? If not, happy to do it. just lemme know. | |
| 17:11:09 | dansmith | mriedem: yeah | |
| 17:11:13 | bauzas | jaypipes: ooooh :( | |
| 17:11:26 | bauzas | jaypipes: I was literrally clicking on submitting the follow-up changre | |
| 17:11:39 | bauzas | jaypipes: hold on, rebasing then | |
| 17:11:44 | jaypipes | bauzas: no worries. | |
| 17:11:58 | mriedem | dansmith: i guess it doesn't matter if we're using nova.conf or nova-cell.conf in this context because they both point to the same dbs | |
| 17:12:17 | dansmith | mriedem: do they? | |
| 17:12:19 | mriedem | i know that devstack starts the cells stuff in a different order, i'm wondering if we're running this before the child cell has started? | |
| 17:12:27 | mriedem | http://logs.openstack.org/85/416585/4/experimental/gate-tempest-dsvm-cells-ubuntu-xenial/fb5df4f/logs/etc/nova/nova.conf.txt.gz | |
| 17:12:31 | mriedem | [database] connection = mysql+pymysql://root:secretmysql@127.0.0.1/nova?charset=utf8 | |
| 17:12:39 | mriedem | http://logs.openstack.org/85/416585/4/experimental/gate-tempest-dsvm-cells-ubuntu-xenial/fb5df4f/logs/etc/nova/nova-cells.conf.txt.gz | |
| 17:12:44 | mriedem | [database] connection = mysql+pymysql://root:secretmysql@127.0.0.1/nova_cell?charset=utf8 | |
| 17:12:48 | mriedem | oh i guess not | |
| 17:12:52 | mriedem | so... | |
| 17:12:55 | dansmith | yeah, so that's worth a try | |
| 17:13:11 | dansmith | especially since laski's comment in there says to use nova-cell.conf | |
| 17:13:13 | mriedem | we should be using nova-cells.conf when running nova-manage right? to get the compute nodes from the child cell | |
| 17:13:28 | mriedem | comment in where? | |
| 17:13:30 | dansmith | mriedem: houston, we may have a problem, stand by | |
| 17:13:46 | dansmith | mriedem: https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L1305 | |
| 17:13:48 | mriedem | dansmith: in this scenario, are you tom hanks or kevin bacon? | |