Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-08-29
17:57:54 mriedem https://github.com/openstack/nova/blob/master/nova/objects/compute_node.py#L188
17:58:54 mriedem which is used here https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L106
17:59:20 mriedem jaypipes: hmm, maybe we aren't reading from a compute node that's come out of the db
17:59:43 melwitt ok, so somehow that is not being effected into the actual ratio being used (the bug)
18:00:10 mriedem jaypipes: nvm, ComputeNode.create calls _from_db_object
18:00:16 mriedem so those allocation ratio fields will be set after create
18:02:07 jaypipes mriedem: yea
18:02:20 mriedem plus if that were the case we'd always fail this in the gate
18:07:22 sean-k-mooney mriedem: if it will help i can try to run the reporduce.sh script without the hard coded ratios in a clean vm and see if it will result in the 0.0 allocations?
18:08:31 melwitt after it pulls the defaults from the compute node object, what is the thing that is supposed to set those default values in placement?
18:09:06 melwitt the RT? makes a call to update the inventory? (reading from the bug)
18:11:16 melwitt ok, yeah here https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L902
18:11:18 sean-k-mooney i would have assumed the ratios were ultimatly set in the virt driver in update_provider_tree
18:13:58 mtreinish mriedem: ?
18:15:25 sean-k-mooney melwitt: oh so we get tree from placement, and pass it to the driver in the update_provider_tree call then normalise which set the defulats then the report_clinet updates placement?
18:17:56 mriedem mtreinish: "we work hard and we play hard"
18:18:14 mriedem mtreinish: if you're going to be around at all anymore, i need it to be for my quick simpsons references
18:18:14 sean-k-mooney melwitt: the xenapi driver does not impmentd update provider tree yet so we are hittig the excpet block https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L907-L921
18:18:52 mriedem melwitt: https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L902 doesn't update placement
18:18:57 mriedem it updates a view of the provider tree locally
18:19:07 melwitt sean-k-mooney: was just thinking about that and whether it's related. does that mean it won't update placement?
18:19:13 mriedem reportclient.update_from_provider_tree(context, prov_tree) is the call that is meant to send the changes from local to remote (placement)
18:19:17 melwitt mriedem: I see, ok, I was just about to look for that
18:19:49 sean-k-mooney mriedem: but we dont call that in the except block...
18:19:52 melwitt mriedem: but that will fail because the xen driver has not implemented it, so then there's a fallback that will do it? trying to understand what is here
18:20:27 mriedem i actually thought the xen driver implemented update_provider_tree, but even so, self.scheduler_client.set_inventory_for_provider( should update the thing
18:20:29 mriedem in placement
18:20:34 mtreinish mriedem: heh, I didn't realize you were in the steel industry
18:20:42 mriedem rust belt baby
18:21:00 melwitt ok, hm
18:23:15 mriedem this is likely related since the report client still relies on a provider tree for caching
18:24:05 melwitt yeah, like you mentioned in the bug. but how could the cache think the allocation ratio is already set to 16.0, for example? and think nothing changed
18:24:41 melwitt if it's currently 0.0
18:26:22 sean-k-mooney mriedem: well we dont call prov_tree.update_inventory(nodename, inv_data) in the except block you mentioned that updated the internal view
18:26:23 mriedem so my logging patch is most likely not going to help the xen case here because it's not going down that alternative route, i'll update in a bit
18:26:37 mriedem sean-k-mooney: no but the report client does internally
18:26:38 cdent mriedem: what are the odds it was related to this change: https://review.openstack.org/#/c/520024/
18:26:43 mriedem on "it's" version
18:26:56 mriedem cdent: heh jaypipes already brought that up :)
18:27:02 mriedem and it was the first thing i thought of today
18:27:11 mriedem but that isn't on rocky and zigo said he hit this on rocky
18:27:14 cdent oh, sorry, I was out walking and it crossed my mind
18:27:32 mriedem unless of course zigo is actually hitting stein code somehow
18:27:40 cdent I'm not certain that what zigo is experiencing is the same as what the xen stuff is experiencing
18:27:54 melwitt do we see this log message, LOG.warning('Unable to refresh my resource provider record')? says "# NOTE(danms): Either we failed to fetch/create the RP on our first attempt, or a previous attempt had to invalidate the cache, and we were unable to refresh it. Bail and try again next time."
18:27:56 cdent because it's not clear what setup zigo has or hasn't done
18:28:17 jaypipes yeah, cdent, that was SOOOOO one hour ago, geez! :P
18:28:29 cdent heh, I was fetching elderberries
18:28:30 cdent which is like
18:28:32 cdent so british
18:28:34 jaypipes :)
18:29:13 cdent it was the first thing I came across when trying to figure out why the ComputeNode after get_inventory might be wrong
18:30:00 cdent because tracing the code the ComputeNode at that point has to have a cpu_allocation_ratio of 0 for us to see the results that are happening
18:30:29 cdent (was also prepared to fetch blackberries but they aren't ready)
18:31:54 mriedem are dingleberries in season yet?
18:32:02 jaypipes cdent: yeah, after finding out that zigo was on rocky, I went and reviewed the last 3 months of patches to rocky that could have anything at all do with allocation ratio or inventory setting and couldn't find anything at all. Also note that the patch you mentioned above isn't in Rocky
18:32:09 jaypipes mriedem: always.
18:32:13 mriedem ha
18:33:00 mriedem melwitt: http://logs.openstack.org/41/590041/17/check/tempest-full/b3f9ddd/controller/logs/screen-n-cpu.txt.gz#_Aug_27_14_18_24_078058 is a failed xen run if you want to dig for logs
18:33:11 melwitt thx
18:36:24 openstackgerrit Matt Riedemann proposed openstack/nova master: Add contributor guide for upgrade status checks https://review.openstack.org/596902
18:41:18 melwitt not seeing this message in the log LOG.debug('Updated inventory for %s at generation %i', which should be there if we've ever successfully updated inventory. which supports the theory that self._provider_tree.has_inventory_changed is returning False
18:41:46 melwitt I don't see any of the error log messages associated with a failure to update inventory
18:43:09 melwitt looks like it would be helpful to have a debug message "Inventory has not changed, skipping update" when it skips
18:43:10 sean-k-mooney its kind of dump but im going to hard code a not implmented excpetion here https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L895 and restack and see if i get the same behavior
18:44:46 mriedem melwitt: yeah that's what my debug patch is doing
18:44:51 sean-k-mooney melwitt: i think thi is where we check if things have changed https://github.com/openstack/nova/blob/722d5b477219f0a2435a9f4ad4d54c61b83219f1/nova/scheduler/client/report.py#L865
18:45:18 mriedem but now i'm distracted by fracas in the tc channel
18:45:24 melwitt mriedem: yeah, just saw that and was about to say, that's what you're already doing and are going to update it to also do it for the non-provider tree route
18:45:45 melwitt since that's where we're going for xen anyway
18:45:51 mriedem yes in progress
18:45:56 mriedem despite fracas
18:46:00 melwitt sean-k-mooney: yes, that's it
18:51:46 openstackgerrit Matt Riedemann proposed openstack/nova master: Add debug logs for when provider inventory changes https://review.openstack.org/597560
18:51:48 mriedem updated for alt path ^ note hyperv and vmware etc would all be failing from this as well if that alternate path is the issue
18:54:02 sean-k-mooney mriedem: i wonder if the hyperv ci also hardcordes the allocation ratios in the conf
18:56:31 mriedem we can find out
18:58:14 sean-k-mooney mriedem: the hyperv ones look ok
18:58:25 sean-k-mooney as in not set
18:58:29 melwitt oh, well, if xen is hard-coding values that match the 0.0 defaults, then placement would _not_ be updated right?
18:59:03 sean-k-mooney mlavalle: xen were hardcoding real ratios
18:59:03 melwitt oh, nevermind. reportclient should be comparing with what's in placement
18:59:40 sean-k-mooney * melwitt: ^
18:59:49 sean-k-mooney melwitt: http://dd6b71949550285df7dc-dda4e480e005aaa13ec303551d2d8155.r49.cf1.rackcdn.com/60/597560/1/check/dsvm-tempest-neutron-network/0b2f0a8/logs/local.conf.txt.gz
19:00:10 sean-k-mooney [DEFAULT]
19:00:11 sean-k-mooney disk_allocation_ratio = 2.0
19:00:13 sean-k-mooney ram_allocation_ratio = 1.5
19:00:15 sean-k-mooney cpu_allocation_ratio = 16.0
19:00:38 melwitt yeah. I was trying to think if that would appear to reportclient as "no change" and therefore not update placement
19:00:41 sean-k-mooney all vaild that said i would never advise setting the disk_allocation_ration over 1
19:01:20 melwitt but, reportclient should be comparing what placement has with those new values, so it should see a change. but from what we know so far, it looks like it isn't seeing a change. mriedem's debug logs will confirm
19:01:30 sean-k-mooney melwitt: if you could some how get the value to the report clinet as 0.0 then yes
19:02:34 melwitt right, yeah
19:03:31 mriedem sean-k-mooney: they are *now*
19:03:35 mriedem they weren't when they reported the issue
19:03:42 mriedem they are hard-coding them in config as a workaround for the CI failure
19:03:54 mriedem which is why i've reverted that change to try and actually get a recreate with logging
19:04:03 sean-k-mooney right so before they were not set
19:04:24 cdent mriedem: have you added logs which watch the value of the cn.*_allocation_ratio in some way?
19:05:02 cdent i'm looking at https://review.openstack.org/#/c/597560/2/nova/compute/resource_tracker.py,unified and wonder if we want more info about the state of the compute node along the way

Earlier   Later