| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-08-29 | |||
| 17:47:16 | mriedem | no, | |
| 17:47:23 | mriedem | zigo is hitting it on rocky with libvirt | |
| 17:48:51 | melwitt | mriedem: I'm unsure whether to +1 this rocky final releases patch given the regression you're investigating. I assume we are too late to fix it for GA, but not 100% sure https://review.openstack.org/597529 | |
| 17:49:48 | mriedem | melwitt: i assume the GA ship has sailed | |
| 17:50:15 | mriedem | given we aren't hitting this in the 'normal' gate and we don't have root cause, i don't think we can hold anything up | |
| 17:51:18 | melwitt | ok. I wasn't clear whether the latest find in xenserver CI yielded a root cause or not. thanks | |
| 17:53:02 | sean-k-mooney | mriedem: strange just looking at my devstack install everything looks fine on libvirt. | |
| 17:53:36 | sean-k-mooney | mriedem: do you know how to reproduce or is that what your currently investiaging | |
| 17:53:58 | mriedem | ... | |
| 17:54:04 | melwitt | in the past, the 0.0 was supposed to be a signal for the scheduler to use different default values, which I always found confusing | |
| 17:54:06 | mriedem | what about "don't know root cause" is .... | |
| 17:54:23 | mriedem | yes it should read from hard-coded values in the compute node | |
| 17:54:25 | mriedem | this isn't the scheduler, | |
| 17:54:32 | mriedem | it's what goes into the resource providers in placement, | |
| 17:54:38 | mriedem | via the RT / ComputeNode object | |
| 17:55:18 | mriedem | here is a revert on the xen ci patch https://review.openstack.org/#/c/597613/1 | |
| 17:55:26 | mriedem | plus nova logs | |
| 17:57:01 | melwitt | this is the quote I was thinking of from the config option help, "NOTE: This can be set per-compute, or if set to 0.0, the value set on the scheduler node(s) or compute node(s) will be used and defaulted to 16.0." | |
| 17:57:19 | mriedem | yes | |
| 17:57:24 | mriedem | the 16.0 comes from the compute node object code | |
| 17:57:35 | melwitt | oh, ok | |
| 17:57:54 | mriedem | https://github.com/openstack/nova/blob/master/nova/objects/compute_node.py#L188 | |
| 17:58:54 | mriedem | which is used here https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L106 | |
| 17:59:20 | mriedem | jaypipes: hmm, maybe we aren't reading from a compute node that's come out of the db | |
| 17:59:43 | melwitt | ok, so somehow that is not being effected into the actual ratio being used (the bug) | |
| 18:00:10 | mriedem | jaypipes: nvm, ComputeNode.create calls _from_db_object | |
| 18:00:16 | mriedem | so those allocation ratio fields will be set after create | |
| 18:02:07 | jaypipes | mriedem: yea | |
| 18:02:20 | mriedem | plus if that were the case we'd always fail this in the gate | |
| 18:07:22 | sean-k-mooney | mriedem: if it will help i can try to run the reporduce.sh script without the hard coded ratios in a clean vm and see if it will result in the 0.0 allocations? | |
| 18:08:31 | melwitt | after it pulls the defaults from the compute node object, what is the thing that is supposed to set those default values in placement? | |
| 18:09:06 | melwitt | the RT? makes a call to update the inventory? (reading from the bug) | |
| 18:11:16 | melwitt | ok, yeah here https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L902 | |
| 18:11:18 | sean-k-mooney | i would have assumed the ratios were ultimatly set in the virt driver in update_provider_tree | |
| 18:13:58 | mtreinish | mriedem: ? | |
| 18:15:25 | sean-k-mooney | melwitt: oh so we get tree from placement, and pass it to the driver in the update_provider_tree call then normalise which set the defulats then the report_clinet updates placement? | |
| 18:17:56 | mriedem | mtreinish: "we work hard and we play hard" | |
| 18:18:14 | sean-k-mooney | melwitt: the xenapi driver does not impmentd update provider tree yet so we are hittig the excpet block https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L907-L921 | |
| 18:18:14 | mriedem | mtreinish: if you're going to be around at all anymore, i need it to be for my quick simpsons references | |
| 18:18:52 | mriedem | melwitt: https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L902 doesn't update placement | |
| 18:18:57 | mriedem | it updates a view of the provider tree locally | |
| 18:19:07 | melwitt | sean-k-mooney: was just thinking about that and whether it's related. does that mean it won't update placement? | |
| 18:19:13 | mriedem | reportclient.update_from_provider_tree(context, prov_tree) is the call that is meant to send the changes from local to remote (placement) | |
| 18:19:17 | melwitt | mriedem: I see, ok, I was just about to look for that | |
| 18:19:49 | sean-k-mooney | mriedem: but we dont call that in the except block... | |
| 18:19:52 | melwitt | mriedem: but that will fail because the xen driver has not implemented it, so then there's a fallback that will do it? trying to understand what is here | |
| 18:20:27 | mriedem | i actually thought the xen driver implemented update_provider_tree, but even so, self.scheduler_client.set_inventory_for_provider( should update the thing | |
| 18:20:29 | mriedem | in placement | |
| 18:20:34 | mtreinish | mriedem: heh, I didn't realize you were in the steel industry | |
| 18:20:42 | mriedem | rust belt baby | |
| 18:21:00 | melwitt | ok, hm | |
| 18:23:15 | mriedem | this is likely related since the report client still relies on a provider tree for caching | |
| 18:24:05 | melwitt | yeah, like you mentioned in the bug. but how could the cache think the allocation ratio is already set to 16.0, for example? and think nothing changed | |
| 18:24:41 | melwitt | if it's currently 0.0 | |
| 18:26:22 | sean-k-mooney | mriedem: well we dont call prov_tree.update_inventory(nodename, inv_data) in the except block you mentioned that updated the internal view | |
| 18:26:23 | mriedem | so my logging patch is most likely not going to help the xen case here because it's not going down that alternative route, i'll update in a bit | |
| 18:26:37 | mriedem | sean-k-mooney: no but the report client does internally | |
| 18:26:38 | cdent | mriedem: what are the odds it was related to this change: https://review.openstack.org/#/c/520024/ | |
| 18:26:43 | mriedem | on "it's" version | |
| 18:26:56 | mriedem | cdent: heh jaypipes already brought that up :) | |
| 18:27:02 | mriedem | and it was the first thing i thought of today | |
| 18:27:11 | mriedem | but that isn't on rocky and zigo said he hit this on rocky | |
| 18:27:14 | cdent | oh, sorry, I was out walking and it crossed my mind | |
| 18:27:32 | mriedem | unless of course zigo is actually hitting stein code somehow | |
| 18:27:40 | cdent | I'm not certain that what zigo is experiencing is the same as what the xen stuff is experiencing | |
| 18:27:54 | melwitt | do we see this log message, LOG.warning('Unable to refresh my resource provider record')? says "# NOTE(danms): Either we failed to fetch/create the RP on our first attempt, or a previous attempt had to invalidate the cache, and we were unable to refresh it. Bail and try again next time." | |
| 18:27:56 | cdent | because it's not clear what setup zigo has or hasn't done | |
| 18:28:17 | jaypipes | yeah, cdent, that was SOOOOO one hour ago, geez! :P | |
| 18:28:29 | cdent | heh, I was fetching elderberries | |
| 18:28:30 | cdent | which is like | |
| 18:28:32 | cdent | so british | |
| 18:28:34 | jaypipes | :) | |
| 18:29:13 | cdent | it was the first thing I came across when trying to figure out why the ComputeNode after get_inventory might be wrong | |
| 18:30:00 | cdent | because tracing the code the ComputeNode at that point has to have a cpu_allocation_ratio of 0 for us to see the results that are happening | |
| 18:30:29 | cdent | (was also prepared to fetch blackberries but they aren't ready) | |
| 18:31:54 | mriedem | are dingleberries in season yet? | |
| 18:32:02 | jaypipes | cdent: yeah, after finding out that zigo was on rocky, I went and reviewed the last 3 months of patches to rocky that could have anything at all do with allocation ratio or inventory setting and couldn't find anything at all. Also note that the patch you mentioned above isn't in Rocky | |
| 18:32:09 | jaypipes | mriedem: always. | |
| 18:32:13 | mriedem | ha | |
| 18:33:00 | mriedem | melwitt: http://logs.openstack.org/41/590041/17/check/tempest-full/b3f9ddd/controller/logs/screen-n-cpu.txt.gz#_Aug_27_14_18_24_078058 is a failed xen run if you want to dig for logs | |
| 18:33:11 | melwitt | thx | |
| 18:36:24 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add contributor guide for upgrade status checks https://review.openstack.org/596902 | |
| 18:41:18 | melwitt | not seeing this message in the log LOG.debug('Updated inventory for %s at generation %i', which should be there if we've ever successfully updated inventory. which supports the theory that self._provider_tree.has_inventory_changed is returning False | |
| 18:41:46 | melwitt | I don't see any of the error log messages associated with a failure to update inventory | |
| 18:43:09 | melwitt | looks like it would be helpful to have a debug message "Inventory has not changed, skipping update" when it skips | |
| 18:43:10 | sean-k-mooney | its kind of dump but im going to hard code a not implmented excpetion here https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L895 and restack and see if i get the same behavior | |
| 18:44:46 | mriedem | melwitt: yeah that's what my debug patch is doing | |
| 18:44:51 | sean-k-mooney | melwitt: i think thi is where we check if things have changed https://github.com/openstack/nova/blob/722d5b477219f0a2435a9f4ad4d54c61b83219f1/nova/scheduler/client/report.py#L865 | |
| 18:45:18 | mriedem | but now i'm distracted by fracas in the tc channel | |
| 18:45:24 | melwitt | mriedem: yeah, just saw that and was about to say, that's what you're already doing and are going to update it to also do it for the non-provider tree route | |
| 18:45:45 | melwitt | since that's where we're going for xen anyway | |
| 18:45:51 | mriedem | yes in progress | |
| 18:45:56 | mriedem | despite fracas | |
| 18:46:00 | melwitt | sean-k-mooney: yes, that's it | |
| 18:51:46 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add debug logs for when provider inventory changes https://review.openstack.org/597560 | |
| 18:51:48 | mriedem | updated for alt path ^ note hyperv and vmware etc would all be failing from this as well if that alternate path is the issue | |
| 18:54:02 | sean-k-mooney | mriedem: i wonder if the hyperv ci also hardcordes the allocation ratios in the conf | |
| 18:56:31 | mriedem | we can find out | |
| 18:58:14 | sean-k-mooney | mriedem: the hyperv ones look ok | |
| 18:58:25 | sean-k-mooney | as in not set | |