| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-02-02 | |||
| 13:25:10 | bauzas | but fixing that could be difficult given we already have allocations | |
| 13:25:17 | sfinucan | trinaths: Not sure if you got a response to your question or not. Have you tried building an instance with a NUMA topology but without hugepages? | |
| 13:28:10 | mriedem | gcb: sfinucan: mdbooth: we shouldn't be putting things into the gate or check queue that don't need to be, py3 unit test cleanup included | |
| 13:28:41 | sfinucan | mriedem: Morning. That's what we thought | |
| 13:28:57 | gcb | mriedem: ack | |
| 13:31:32 | mdbooth | lyarwood: I'm just going to grab coffee. Want to talk about swap_volume in a few minutes? | |
| 13:31:43 | lyarwood | mdbooth: sure | |
| 13:31:48 | mriedem | bauzas: is that related to the thing that johnthetubaguy has a patch up for? | |
| 13:32:06 | mriedem | https://review.openstack.org/#/c/428120/ | |
| 13:32:47 | sfinucan | mriedem, bauzas: That's the one, yes | |
| 13:33:04 | bauzas | mriedem: not really but the same problme | |
| 13:33:20 | bauzas | mriedem: honestly, MHO is that we should not use GBs for disk resources but rather bytes | |
| 13:33:37 | bauzas | or KB | |
| 13:33:43 | bauzas | if that's too big | |
| 13:33:54 | bauzas | but, we have lots of ceilings like the one I wrote | |
| 13:34:11 | mriedem | dansmith: melwitt: hmm, lyarwood's patch failed in the gate on the cells inst delete/create race thing http://logs.openstack.org/67/427267/2/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/e4e67b3/console.html#_2017-02-02_11_48_07_755243 | |
| 13:34:28 | bauzas | mriedem: https://github.com/openstack/nova/blob/master/nova/scheduler/filter_scheduler.py#L148 | |
| 13:34:49 | mriedem | https://review.openstack.org/#/c/427782/ | |
| 13:35:07 | mriedem | which merged at 7:34pm | |
| 13:35:29 | mriedem | and lyarwood's change failed at 2017-02-02 11:48:07.755243 | |
| 13:35:51 | lyarwood | mriedem: twice, I think the previous run also hit this a few hours before that | |
| 13:36:07 | mriedem | lyarwood: well i could see it hitting before the other fix was merged, | |
| 13:36:14 | mriedem | but i'm worried if it hit after the fix was merged | |
| 13:36:37 | mriedem | i think ES is backed up too so we probably can't tell if the hits have dropped off after the fix merged | |
| 13:36:49 | mriedem | yeah we can't tell http://status.openstack.org//elastic-recheck/index.html | |
| 13:36:52 | mriedem | everything is backed up | |
| 13:37:25 | lyarwood | mriedem: http://logs.openstack.org/67/427267/2/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/cc0ed47/console.html#_2017-02-02_05_45_00_341186 was the other hit | |
| 13:39:18 | mriedem | yeah that's before the fix was merged | |
| 13:39:21 | mriedem | so not a surprise | |
| 13:39:32 | mriedem | but failing 3 hours after the merge worries me | |
| 13:39:41 | lyarwood | mriedem: ah did it land at 7:34am? | |
| 13:39:47 | mriedem | pm | |
| 13:39:51 | lyarwood | mriedem: you said pm above | |
| 13:39:58 | lyarwood | mriedem: that's 05:45am | |
| 13:40:02 | mriedem | oh yeah | |
| 13:40:12 | lyarwood | anyway | |
| 13:41:18 | nmathew- | hi all | |
| 13:41:35 | nmathew- | can anybody point to a material on flavors | |
| 13:41:38 | mriedem | lyarwood: apparently the times in the zuul log don't match up with what's in the gerrit ui | |
| 13:41:55 | mriedem | nmathew-: http://docs.openstack.org/admin-guide/compute-flavors.html | |
| 13:42:00 | mriedem | docs.o.o has a search box | |
| 13:44:07 | nmathew- | mriedem, say I not enabling hugepages in flavors, I can stil enable that by kernel parameters to that vm kernel | |
| 13:44:31 | nmathew- | mriedem, what is the difference between specifying in flavors and not specifying in flavors? | |
| 13:44:54 | lyarwood | nmathew-: see the topic, if you have questions like that #openstack is a better place to ask :) | |
| 13:45:17 | nmathew- | ok | |
| 13:45:25 | nmathew- | oh sorry | |
| 13:45:33 | lyarwood | nmathew-: np at all | |
| 13:46:06 | mriedem | so looking at my fix for the delete/create race bug, | |
| 13:46:07 | mriedem | https://review.openstack.org/#/c/427775/1/nova/conductor/manager.py | |
| 13:46:23 | mriedem | i see that debug log message hit 3 times in the n-cond logs on a run of that patch http://logs.openstack.org/75/427775/1/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/c906c15/logs/screen-n-cond.txt.gz#_2017-02-01_16_14_03_273 | |
| 13:46:31 | mriedem | so it's obviously catching the issue | |
| 13:46:51 | mriedem | dansmith: bauzas: ^ so i'm thinking we should probably just get that in given the time | |
| 13:53:10 | cdent | mriedem: unless you're already on it, I'll fix the nits in https://review.openstack.org/#/c/427367 | |
| 13:53:48 | mriedem | i'll get it quick | |
| 13:55:41 | openstackgerrit | Matt Riedemann proposed openstack/nova master: doc: add upgrade notes to the placement devref https://review.openstack.org/427367 | |
| 13:55:42 | mriedem | sfinucan: bauzas: ^ fixed the typo | |
| 13:56:20 | bauzas | mriedem: okay, let's merge https://review.openstack.org/#/c/427775/1 then | |
| 13:57:23 | sfinucan | mriedem: Job. Done. | |
| 13:57:26 | mriedem | i wonder if it would be better to destroy the build request right before creating the instance? | |
| 13:57:53 | mriedem | but, i don't really want to ponder that today | |
| 13:58:17 | bauzas | mriedem: possibly | |
| 13:58:24 | bauzas | mriedem: the main issue I see here is timing | |
| 13:58:45 | bauzas | so let's merge your change is quite impactless and think about the best approach for Pike | |
| 14:00:41 | johnthetubaguy | mriedem: FWIW, I think its best to create the instance first, so we don't "loose" the instance, although marking the build request in someway that stops deletes might make sense, maybe thats the cell mapping entry? | |
| 14:01:24 | johnthetubaguy | hmm, I can't 100% remember the ordering now | |
| 14:02:07 | johnthetubaguy | mriedem: this one the openstack ansible folks I spotted feels nasty, but we totally could back port it later, I believe | |
| 14:02:47 | mriedem | johnthetubaguy: https://review.openstack.org/#/c/428120/ right? | |
| 14:02:50 | mriedem | i left some comments in there | |
| 14:03:00 | mriedem | there is a bug report i think that fits the same thing already | |
| 14:03:03 | mriedem | i've marked it rc1 | |
| 14:06:36 | alex_xu | johnthetubaguy: I think gmann is right on https://review.openstack.org/#/c/428071, we agreed on we have filter which match the name in the API representation, but that will be done in the future with microversion. | |
| 14:11:54 | johnthetubaguy | mriedem: the bug is different I think, the bug talks about the allocation, this my bug is about the inventory (due to the reservation) | |
| 14:12:09 | jroll | mriedem: fyi this one looks fun https://bugs.launchpad.net/nova/+bug/1661258 | |
| 14:12:09 | openstack | Launchpad bug 1661258 in OpenStack Compute (nova) "Deleted ironic node has an inventory in nova_api database" [Undecided,New] | |
| 14:13:41 | mriedem | jroll: damn | |
| 14:13:52 | jroll | it might actually be easy | |
| 14:13:53 | mriedem | jroll: will need cdent, edleafe and/or jaypipes to look at that one | |
| 14:14:02 | jroll | mriedem: thank you for pinging them :P | |
| 14:14:17 | mriedem | i was thinking the periodic update would remove the inventory for the removed node but vdrok said in the bug that it didn't | |
| 14:14:22 | cdent | joy | |
| 14:15:34 | edleafe | mriedem: I can take a look at that | |
| 14:15:57 | jroll | right, I assume we're missing something like this for inventory https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L6577 | |
| 14:18:50 | mriedem | jroll: yeah that looks reasonable | |
| 14:19:02 | mriedem | jaypipes refactored a ton of that code so he'll know best | |
| 14:19:15 | jaypipes | jroll, vdrok, mriedem, edleafe, cdent: I have a suspicion on cause of that bug. | |
| 14:19:31 | edleafe | jaypipes: do tell | |
| 14:20:11 | jaypipes | jroll, vdrok, mriedem, edleafe, cdent: Ironic virt driver is probably returning False for node_is_available() and therefore when update_available_resource() runs, we never even get to the call for InstanceList.get_by_host_and_node() and therefore never clean up the now-deleted instance. | |
| 14:20:48 | mriedem | sdague: how do you feel about this? https://review.openstack.org/#/c/426926/ | |
| 14:20:49 | jaypipes | mriedem: remember when you asked me to change RT.disabled(nodename) to account for the Ironic-only node_is_available() call? :) | |
| 14:20:57 | mriedem | sdague: because i'd like to get the dependent nova change in for ocata | |
| 14:21:15 | mriedem | jaypipes: vaguely | |
| 14:21:35 | mriedem | jaypipes: i'm pretty sure i was drunk reviewing most of that refactor series :) | |
| 14:21:45 | jroll | jaypipes: this is about deleted nodes, not deleted instances, to be clear | |
| 14:22:11 | jroll | jaypipes: so it wouldn't even be returned in get_available_nodes() | |
| 14:22:18 | jaypipes | vdrok: I asked for you to do a flash-patch in the bug above. | |
| 14:22:34 | cdent | is running two nova-computes on the same machine normal and/or an important consideration in this bug? | |
| 14:23:01 | jaypipes | jroll: right, but the inventory won't get cleaned up in the placement DB if the node_is_available() returns False here: | |
| 14:23:02 | vdrok | jaypipes: ok, thanks, will do after lunch | |
| 14:23:02 | jaypipes | https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L414 | |
| 14:23:12 | jaypipes | cdent: no. | |