| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-08-29 | |||
| 17:26:00 | jaypipes | mriedem: my bad, sorry. | |
| 17:26:27 | mriedem | help me find the repo | |
| 17:26:55 | sean-k-mooney | mriedem: could you just hard code the nova.conf passing code to return none | |
| 17:27:09 | sean-k-mooney | mriedem: or whatever it returns when its not set | |
| 17:29:52 | sean-k-mooney | mriedem: they are hard coding it in the local.conf http://dd6b71949550285df7dc-dda4e480e005aaa13ec303551d2d8155.r49.cf1.rackcdn.com/53/597553/1/check/dsvm-tempest-neutron-network/971ea88/logs/local.conf.txt.gz | |
| 17:30:18 | mriedem | i didn't think git://git.openstack.org/openstack/os-xenapi was it | |
| 17:30:32 | mriedem | yeah it's not | |
| 17:32:07 | cdent | mriedem: did you say that zigo's thing was changed to depends-on your debuggery? | |
| 17:32:16 | mriedem | yeah but i'm not sure it would help | |
| 17:32:22 | dansmith | mriedem: yeah that's the pack of xenapi plugins I think | |
| 17:33:02 | mriedem | https://review.openstack.org/#/c/597175/ but that doesn't actually get the nova change into the nova package | |
| 17:34:33 | mriedem | sean-k-mooney: oh you meant the allocation ratios - yes i knew that | |
| 17:34:40 | mriedem | and yeah i might have to hack the nova code to ignore the config | |
| 17:34:44 | mriedem | but first lunch | |
| 17:35:53 | sean-k-mooney | mriedem: yes i ment hack the nova code to ignore the configs so that you can work around there hack to hard code them :) | |
| 17:36:48 | jaypipes | mriedem: https://review.openstack.org/#/c/597428/ | |
| 17:38:39 | sean-k-mooney | jaypipes: oh you found it. am i the only on that is bothered by the fact the repo is xenapi-os-testing when the other one is os-xenapi | |
| 17:39:33 | sean-k-mooney | jaypipes: that change is not merged however which implies the ci is not running off of master of that repo? | |
| 17:42:46 | mriedem | they might be patching that into all CI runs | |
| 17:43:44 | jaypipes | what mriedem said. | |
| 17:43:48 | sean-k-mooney | mriedem: you could see if a depends on would override it. e.g. make a noop patch to xenapi-os-testing then depend on it to force the unpatched version? | |
| 17:44:39 | jaypipes | sean-k-mooney: that would assume the xenserver CI is honouring depends-on, no? | |
| 17:45:04 | sean-k-mooney | true | |
| 17:45:17 | mriedem | i'm pretty sure they do, | |
| 17:45:25 | mriedem | i'll just revert that change and depends-on the nova logging patch | |
| 17:46:49 | openstack | Launchpad bug 1789654 in OpenStack Compute (nova) rocky "placement allocation_ratio initialized with 0.0" [High,Confirmed] | |
| 17:46:49 | sean-k-mooney | is https://bugs.launchpad.net/nova/+bug/1789654 only happenign with the xen driver by the way? | |
| 17:47:16 | mriedem | no, | |
| 17:47:23 | mriedem | zigo is hitting it on rocky with libvirt | |
| 17:48:51 | melwitt | mriedem: I'm unsure whether to +1 this rocky final releases patch given the regression you're investigating. I assume we are too late to fix it for GA, but not 100% sure https://review.openstack.org/597529 | |
| 17:49:48 | mriedem | melwitt: i assume the GA ship has sailed | |
| 17:50:15 | mriedem | given we aren't hitting this in the 'normal' gate and we don't have root cause, i don't think we can hold anything up | |
| 17:51:18 | melwitt | ok. I wasn't clear whether the latest find in xenserver CI yielded a root cause or not. thanks | |
| 17:53:02 | sean-k-mooney | mriedem: strange just looking at my devstack install everything looks fine on libvirt. | |
| 17:53:36 | sean-k-mooney | mriedem: do you know how to reproduce or is that what your currently investiaging | |
| 17:53:58 | mriedem | ... | |
| 17:54:04 | melwitt | in the past, the 0.0 was supposed to be a signal for the scheduler to use different default values, which I always found confusing | |
| 17:54:06 | mriedem | what about "don't know root cause" is .... | |
| 17:54:23 | mriedem | yes it should read from hard-coded values in the compute node | |
| 17:54:25 | mriedem | this isn't the scheduler, | |
| 17:54:32 | mriedem | it's what goes into the resource providers in placement, | |
| 17:54:38 | mriedem | via the RT / ComputeNode object | |
| 17:55:18 | mriedem | here is a revert on the xen ci patch https://review.openstack.org/#/c/597613/1 | |
| 17:55:26 | mriedem | plus nova logs | |
| 17:57:01 | melwitt | this is the quote I was thinking of from the config option help, "NOTE: This can be set per-compute, or if set to 0.0, the value set on the scheduler node(s) or compute node(s) will be used and defaulted to 16.0." | |
| 17:57:19 | mriedem | yes | |
| 17:57:24 | mriedem | the 16.0 comes from the compute node object code | |
| 17:57:35 | melwitt | oh, ok | |
| 17:57:54 | mriedem | https://github.com/openstack/nova/blob/master/nova/objects/compute_node.py#L188 | |
| 17:58:54 | mriedem | which is used here https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L106 | |
| 17:59:20 | mriedem | jaypipes: hmm, maybe we aren't reading from a compute node that's come out of the db | |
| 17:59:43 | melwitt | ok, so somehow that is not being effected into the actual ratio being used (the bug) | |
| 18:00:10 | mriedem | jaypipes: nvm, ComputeNode.create calls _from_db_object | |
| 18:00:16 | mriedem | so those allocation ratio fields will be set after create | |
| 18:02:07 | jaypipes | mriedem: yea | |
| 18:02:20 | mriedem | plus if that were the case we'd always fail this in the gate | |
| 18:07:22 | sean-k-mooney | mriedem: if it will help i can try to run the reporduce.sh script without the hard coded ratios in a clean vm and see if it will result in the 0.0 allocations? | |
| 18:08:31 | melwitt | after it pulls the defaults from the compute node object, what is the thing that is supposed to set those default values in placement? | |
| 18:09:06 | melwitt | the RT? makes a call to update the inventory? (reading from the bug) | |
| 18:11:16 | melwitt | ok, yeah here https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L902 | |
| 18:11:18 | sean-k-mooney | i would have assumed the ratios were ultimatly set in the virt driver in update_provider_tree | |
| 18:13:58 | mtreinish | mriedem: ? | |
| 18:15:25 | sean-k-mooney | melwitt: oh so we get tree from placement, and pass it to the driver in the update_provider_tree call then normalise which set the defulats then the report_clinet updates placement? | |
| 18:17:56 | mriedem | mtreinish: "we work hard and we play hard" | |
| 18:18:14 | mriedem | mtreinish: if you're going to be around at all anymore, i need it to be for my quick simpsons references | |
| 18:18:14 | sean-k-mooney | melwitt: the xenapi driver does not impmentd update provider tree yet so we are hittig the excpet block https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L907-L921 | |
| 18:18:52 | mriedem | melwitt: https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L902 doesn't update placement | |
| 18:18:57 | mriedem | it updates a view of the provider tree locally | |
| 18:19:07 | melwitt | sean-k-mooney: was just thinking about that and whether it's related. does that mean it won't update placement? | |
| 18:19:13 | mriedem | reportclient.update_from_provider_tree(context, prov_tree) is the call that is meant to send the changes from local to remote (placement) | |
| 18:19:17 | melwitt | mriedem: I see, ok, I was just about to look for that | |
| 18:19:49 | sean-k-mooney | mriedem: but we dont call that in the except block... | |
| 18:19:52 | melwitt | mriedem: but that will fail because the xen driver has not implemented it, so then there's a fallback that will do it? trying to understand what is here | |
| 18:20:27 | mriedem | i actually thought the xen driver implemented update_provider_tree, but even so, self.scheduler_client.set_inventory_for_provider( should update the thing | |
| 18:20:29 | mriedem | in placement | |
| 18:20:34 | mtreinish | mriedem: heh, I didn't realize you were in the steel industry | |
| 18:20:42 | mriedem | rust belt baby | |
| 18:21:00 | melwitt | ok, hm | |
| 18:23:15 | mriedem | this is likely related since the report client still relies on a provider tree for caching | |
| 18:24:05 | melwitt | yeah, like you mentioned in the bug. but how could the cache think the allocation ratio is already set to 16.0, for example? and think nothing changed | |
| 18:24:41 | melwitt | if it's currently 0.0 | |
| 18:26:22 | sean-k-mooney | mriedem: well we dont call prov_tree.update_inventory(nodename, inv_data) in the except block you mentioned that updated the internal view | |
| 18:26:23 | mriedem | so my logging patch is most likely not going to help the xen case here because it's not going down that alternative route, i'll update in a bit | |
| 18:26:37 | mriedem | sean-k-mooney: no but the report client does internally | |
| 18:26:38 | cdent | mriedem: what are the odds it was related to this change: https://review.openstack.org/#/c/520024/ | |
| 18:26:43 | mriedem | on "it's" version | |
| 18:26:56 | mriedem | cdent: heh jaypipes already brought that up :) | |
| 18:27:02 | mriedem | and it was the first thing i thought of today | |
| 18:27:11 | mriedem | but that isn't on rocky and zigo said he hit this on rocky | |
| 18:27:14 | cdent | oh, sorry, I was out walking and it crossed my mind | |
| 18:27:32 | mriedem | unless of course zigo is actually hitting stein code somehow | |
| 18:27:40 | cdent | I'm not certain that what zigo is experiencing is the same as what the xen stuff is experiencing | |
| 18:27:54 | melwitt | do we see this log message, LOG.warning('Unable to refresh my resource provider record')? says "# NOTE(danms): Either we failed to fetch/create the RP on our first attempt, or a previous attempt had to invalidate the cache, and we were unable to refresh it. Bail and try again next time." | |
| 18:27:56 | cdent | because it's not clear what setup zigo has or hasn't done | |
| 18:28:17 | jaypipes | yeah, cdent, that was SOOOOO one hour ago, geez! :P | |
| 18:28:29 | cdent | heh, I was fetching elderberries | |
| 18:28:30 | cdent | which is like | |
| 18:28:32 | cdent | so british | |
| 18:28:34 | jaypipes | :) | |
| 18:29:13 | cdent | it was the first thing I came across when trying to figure out why the ComputeNode after get_inventory might be wrong | |