Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-08-29
17:34:44 mriedem but first lunch
17:35:53 sean-k-mooney mriedem: yes i ment hack the nova code to ignore the configs so that you can work around there hack to hard code them :)
17:36:48 jaypipes mriedem: https://review.openstack.org/#/c/597428/
17:38:39 sean-k-mooney jaypipes: oh you found it. am i the only on that is bothered by the fact the repo is xenapi-os-testing when the other one is os-xenapi
17:39:33 sean-k-mooney jaypipes: that change is not merged however which implies the ci is not running off of master of that repo?
17:42:46 mriedem they might be patching that into all CI runs
17:43:44 jaypipes what mriedem said.
17:43:48 sean-k-mooney mriedem: you could see if a depends on would override it. e.g. make a noop patch to xenapi-os-testing then depend on it to force the unpatched version?
17:44:39 jaypipes sean-k-mooney: that would assume the xenserver CI is honouring depends-on, no?
17:45:04 sean-k-mooney true
17:45:17 mriedem i'm pretty sure they do,
17:45:25 mriedem i'll just revert that change and depends-on the nova logging patch
17:46:49 openstack Launchpad bug 1789654 in OpenStack Compute (nova) rocky "placement allocation_ratio initialized with 0.0" [High,Confirmed]
17:46:49 sean-k-mooney is https://bugs.launchpad.net/nova/+bug/1789654 only happenign with the xen driver by the way?
17:47:16 mriedem no,
17:47:23 mriedem zigo is hitting it on rocky with libvirt
17:48:51 melwitt mriedem: I'm unsure whether to +1 this rocky final releases patch given the regression you're investigating. I assume we are too late to fix it for GA, but not 100% sure https://review.openstack.org/597529
17:49:48 mriedem melwitt: i assume the GA ship has sailed
17:50:15 mriedem given we aren't hitting this in the 'normal' gate and we don't have root cause, i don't think we can hold anything up
17:51:18 melwitt ok. I wasn't clear whether the latest find in xenserver CI yielded a root cause or not. thanks
17:53:02 sean-k-mooney mriedem: strange just looking at my devstack install everything looks fine on libvirt.
17:53:36 sean-k-mooney mriedem: do you know how to reproduce or is that what your currently investiaging
17:53:58 mriedem ...
17:54:04 melwitt in the past, the 0.0 was supposed to be a signal for the scheduler to use different default values, which I always found confusing
17:54:06 mriedem what about "don't know root cause" is ....
17:54:23 mriedem yes it should read from hard-coded values in the compute node
17:54:25 mriedem this isn't the scheduler,
17:54:32 mriedem it's what goes into the resource providers in placement,
17:54:38 mriedem via the RT / ComputeNode object
17:55:18 mriedem here is a revert on the xen ci patch https://review.openstack.org/#/c/597613/1
17:55:26 mriedem plus nova logs
17:57:01 melwitt this is the quote I was thinking of from the config option help, "NOTE: This can be set per-compute, or if set to 0.0, the value set on the scheduler node(s) or compute node(s) will be used and defaulted to 16.0."
17:57:19 mriedem yes
17:57:24 mriedem the 16.0 comes from the compute node object code
17:57:35 melwitt oh, ok
17:57:54 mriedem https://github.com/openstack/nova/blob/master/nova/objects/compute_node.py#L188
17:58:54 mriedem which is used here https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L106
17:59:20 mriedem jaypipes: hmm, maybe we aren't reading from a compute node that's come out of the db
17:59:43 melwitt ok, so somehow that is not being effected into the actual ratio being used (the bug)
18:00:10 mriedem jaypipes: nvm, ComputeNode.create calls _from_db_object
18:00:16 mriedem so those allocation ratio fields will be set after create
18:02:07 jaypipes mriedem: yea
18:02:20 mriedem plus if that were the case we'd always fail this in the gate
18:07:22 sean-k-mooney mriedem: if it will help i can try to run the reporduce.sh script without the hard coded ratios in a clean vm and see if it will result in the 0.0 allocations?
18:08:31 melwitt after it pulls the defaults from the compute node object, what is the thing that is supposed to set those default values in placement?
18:09:06 melwitt the RT? makes a call to update the inventory? (reading from the bug)
18:11:16 melwitt ok, yeah here https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L902
18:11:18 sean-k-mooney i would have assumed the ratios were ultimatly set in the virt driver in update_provider_tree
18:13:58 mtreinish mriedem: ?
18:15:25 sean-k-mooney melwitt: oh so we get tree from placement, and pass it to the driver in the update_provider_tree call then normalise which set the defulats then the report_clinet updates placement?
18:17:56 mriedem mtreinish: "we work hard and we play hard"
18:18:14 mriedem mtreinish: if you're going to be around at all anymore, i need it to be for my quick simpsons references
18:18:14 sean-k-mooney melwitt: the xenapi driver does not impmentd update provider tree yet so we are hittig the excpet block https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L907-L921
18:18:52 mriedem melwitt: https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L902 doesn't update placement
18:18:57 mriedem it updates a view of the provider tree locally
18:19:07 melwitt sean-k-mooney: was just thinking about that and whether it's related. does that mean it won't update placement?
18:19:13 mriedem reportclient.update_from_provider_tree(context, prov_tree) is the call that is meant to send the changes from local to remote (placement)
18:19:17 melwitt mriedem: I see, ok, I was just about to look for that
18:19:49 sean-k-mooney mriedem: but we dont call that in the except block...
18:19:52 melwitt mriedem: but that will fail because the xen driver has not implemented it, so then there's a fallback that will do it? trying to understand what is here
18:20:27 mriedem i actually thought the xen driver implemented update_provider_tree, but even so, self.scheduler_client.set_inventory_for_provider( should update the thing
18:20:29 mriedem in placement
18:20:34 mtreinish mriedem: heh, I didn't realize you were in the steel industry
18:20:42 mriedem rust belt baby
18:21:00 melwitt ok, hm
18:23:15 mriedem this is likely related since the report client still relies on a provider tree for caching
18:24:05 melwitt yeah, like you mentioned in the bug. but how could the cache think the allocation ratio is already set to 16.0, for example? and think nothing changed
18:24:41 melwitt if it's currently 0.0
18:26:22 sean-k-mooney mriedem: well we dont call prov_tree.update_inventory(nodename, inv_data) in the except block you mentioned that updated the internal view
18:26:23 mriedem so my logging patch is most likely not going to help the xen case here because it's not going down that alternative route, i'll update in a bit
18:26:37 mriedem sean-k-mooney: no but the report client does internally
18:26:38 cdent mriedem: what are the odds it was related to this change: https://review.openstack.org/#/c/520024/
18:26:43 mriedem on "it's" version
18:26:56 mriedem cdent: heh jaypipes already brought that up :)
18:27:02 mriedem and it was the first thing i thought of today
18:27:11 mriedem but that isn't on rocky and zigo said he hit this on rocky
18:27:14 cdent oh, sorry, I was out walking and it crossed my mind
18:27:32 mriedem unless of course zigo is actually hitting stein code somehow
18:27:40 cdent I'm not certain that what zigo is experiencing is the same as what the xen stuff is experiencing
18:27:54 melwitt do we see this log message, LOG.warning('Unable to refresh my resource provider record')? says "# NOTE(danms): Either we failed to fetch/create the RP on our first attempt, or a previous attempt had to invalidate the cache, and we were unable to refresh it. Bail and try again next time."
18:27:56 cdent because it's not clear what setup zigo has or hasn't done
18:28:17 jaypipes yeah, cdent, that was SOOOOO one hour ago, geez! :P
18:28:29 cdent heh, I was fetching elderberries
18:28:30 cdent which is like
18:28:32 cdent so british
18:28:34 jaypipes :)
18:29:13 cdent it was the first thing I came across when trying to figure out why the ComputeNode after get_inventory might be wrong
18:30:00 cdent because tracing the code the ComputeNode at that point has to have a cpu_allocation_ratio of 0 for us to see the results that are happening
18:30:29 cdent (was also prepared to fetch blackberries but they aren't ready)
18:31:54 mriedem are dingleberries in season yet?
18:32:02 jaypipes cdent: yeah, after finding out that zigo was on rocky, I went and reviewed the last 3 months of patches to rocky that could have anything at all do with allocation ratio or inventory setting and couldn't find anything at all. Also note that the patch you mentioned above isn't in Rocky
18:32:09 jaypipes mriedem: always.
18:32:13 mriedem ha
18:33:00 mriedem melwitt: http://logs.openstack.org/41/590041/17/check/tempest-full/b3f9ddd/controller/logs/screen-n-cpu.txt.gz#_Aug_27_14_18_24_078058 is a failed xen run if you want to dig for logs
18:33:11 melwitt thx
18:36:24 openstackgerrit Matt Riedemann proposed openstack/nova master: Add contributor guide for upgrade status checks https://review.openstack.org/596902
18:41:18 melwitt not seeing this message in the log LOG.debug('Updated inventory for %s at generation %i', which should be there if we've ever successfully updated inventory. which supports the theory that self._provider_tree.has_inventory_changed is returning False
18:41:46 melwitt I don't see any of the error log messages associated with a failure to update inventory
18:43:09 melwitt looks like it would be helpful to have a debug message "Inventory has not changed, skipping update" when it skips
18:43:10 sean-k-mooney its kind of dump but im going to hard code a not implmented excpetion here https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L895 and restack and see if i get the same behavior

Earlier   Later