Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-08-29
17:03:50 gibi mlavalle: but you might be interested still
17:04:38 mlavalle gibi: ack, thanks for the heads up
17:13:56 sean-k-mooney mlavalle: i tend to start late and work late.
17:14:21 mriedem i work hard and i play hard
17:14:24 mriedem mtreinish:
17:15:24 sean-k-mooney gibi: sure if you want to cover that in the placement sessions then that works too. just wanted to make sure it did not slip through the cracks
17:16:06 gibi sean-k-mooney: at the moment the resource mapping does not affect Neutron it either affect placement and nova or nova only.
17:16:47 gibi sean-k-mooney: neutron will get the nework device RP uuid from nova during the port binding anyhow
17:17:44 openstackgerrit Matt Riedemann proposed openstack/nova master: (Re)start caching scheduler after starting computes in tests https://review.openstack.org/597606
17:18:04 sean-k-mooney gibi: previded we modify nova to pass them to you :)
17:18:48 sean-k-mooney gibi: looking at the spec this looks pretty familar to what we discussed back in dublin
17:19:30 gibi sean-k-mooney: code is up that does passes the RP from nova to neutron https://review.openstack.org/#/c/569459/26/nova/network/neutronv2/api.py@3129
17:19:46 gibi sean-k-mooney: it only works for the simple cases
17:20:08 gibi sean-k-mooney: I can make it work for the general case but it won't scale
17:20:22 gibi sean-k-mooney: so I proposed the spec to do the mapping in placement
17:21:17 gibi sean-k-mooney: I have to leave for today I happy to continue the discussion tomorrow, or on the review, and eventually on the PTG
17:21:18 sean-k-mooney gibi: oh cool i had not seen that series. i will have to pull it down and try it out
17:21:53 sean-k-mooney gibi: no worries have a good evening
17:21:59 gibi sean-k-mooney: same to you
17:24:44 mriedem so my debug logging patch didn't fail xenserver ci since they hard-coded the allocation ratios in nova.conf http://dd6b71949550285df7dc-dda4e480e005aaa13ec303551d2d8155.r49.cf1.rackcdn.com/53/597553/1/check/dsvm-tempest-neutron-network/971ea88/logs/etc/nova/nova.conf.txt.gz
17:24:52 mriedem i need to find their repo to revert that change
17:26:00 jaypipes mriedem: my bad, sorry.
17:26:27 mriedem help me find the repo
17:26:55 sean-k-mooney mriedem: could you just hard code the nova.conf passing code to return none
17:27:09 sean-k-mooney mriedem: or whatever it returns when its not set
17:29:52 sean-k-mooney mriedem: they are hard coding it in the local.conf http://dd6b71949550285df7dc-dda4e480e005aaa13ec303551d2d8155.r49.cf1.rackcdn.com/53/597553/1/check/dsvm-tempest-neutron-network/971ea88/logs/local.conf.txt.gz
17:30:18 mriedem i didn't think git://git.openstack.org/openstack/os-xenapi was it
17:30:32 mriedem yeah it's not
17:32:07 cdent mriedem: did you say that zigo's thing was changed to depends-on your debuggery?
17:32:16 mriedem yeah but i'm not sure it would help
17:32:22 dansmith mriedem: yeah that's the pack of xenapi plugins I think
17:33:02 mriedem https://review.openstack.org/#/c/597175/ but that doesn't actually get the nova change into the nova package
17:34:33 mriedem sean-k-mooney: oh you meant the allocation ratios - yes i knew that
17:34:40 mriedem and yeah i might have to hack the nova code to ignore the config
17:34:44 mriedem but first lunch
17:35:53 sean-k-mooney mriedem: yes i ment hack the nova code to ignore the configs so that you can work around there hack to hard code them :)
17:36:48 jaypipes mriedem: https://review.openstack.org/#/c/597428/
17:38:39 sean-k-mooney jaypipes: oh you found it. am i the only on that is bothered by the fact the repo is xenapi-os-testing when the other one is os-xenapi
17:39:33 sean-k-mooney jaypipes: that change is not merged however which implies the ci is not running off of master of that repo?
17:42:46 mriedem they might be patching that into all CI runs
17:43:44 jaypipes what mriedem said.
17:43:48 sean-k-mooney mriedem: you could see if a depends on would override it. e.g. make a noop patch to xenapi-os-testing then depend on it to force the unpatched version?
17:44:39 jaypipes sean-k-mooney: that would assume the xenserver CI is honouring depends-on, no?
17:45:04 sean-k-mooney true
17:45:17 mriedem i'm pretty sure they do,
17:45:25 mriedem i'll just revert that change and depends-on the nova logging patch
17:46:49 openstack Launchpad bug 1789654 in OpenStack Compute (nova) rocky "placement allocation_ratio initialized with 0.0" [High,Confirmed]
17:46:49 sean-k-mooney is https://bugs.launchpad.net/nova/+bug/1789654 only happenign with the xen driver by the way?
17:47:16 mriedem no,
17:47:23 mriedem zigo is hitting it on rocky with libvirt
17:48:51 melwitt mriedem: I'm unsure whether to +1 this rocky final releases patch given the regression you're investigating. I assume we are too late to fix it for GA, but not 100% sure https://review.openstack.org/597529
17:49:48 mriedem melwitt: i assume the GA ship has sailed
17:50:15 mriedem given we aren't hitting this in the 'normal' gate and we don't have root cause, i don't think we can hold anything up
17:51:18 melwitt ok. I wasn't clear whether the latest find in xenserver CI yielded a root cause or not. thanks
17:53:02 sean-k-mooney mriedem: strange just looking at my devstack install everything looks fine on libvirt.
17:53:36 sean-k-mooney mriedem: do you know how to reproduce or is that what your currently investiaging
17:53:58 mriedem ...
17:54:04 melwitt in the past, the 0.0 was supposed to be a signal for the scheduler to use different default values, which I always found confusing
17:54:06 mriedem what about "don't know root cause" is ....
17:54:23 mriedem yes it should read from hard-coded values in the compute node
17:54:25 mriedem this isn't the scheduler,
17:54:32 mriedem it's what goes into the resource providers in placement,
17:54:38 mriedem via the RT / ComputeNode object
17:55:18 mriedem here is a revert on the xen ci patch https://review.openstack.org/#/c/597613/1
17:55:26 mriedem plus nova logs
17:57:01 melwitt this is the quote I was thinking of from the config option help, "NOTE: This can be set per-compute, or if set to 0.0, the value set on the scheduler node(s) or compute node(s) will be used and defaulted to 16.0."
17:57:19 mriedem yes
17:57:24 mriedem the 16.0 comes from the compute node object code
17:57:35 melwitt oh, ok
17:57:54 mriedem https://github.com/openstack/nova/blob/master/nova/objects/compute_node.py#L188
17:58:54 mriedem which is used here https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L106
17:59:20 mriedem jaypipes: hmm, maybe we aren't reading from a compute node that's come out of the db
17:59:43 melwitt ok, so somehow that is not being effected into the actual ratio being used (the bug)
18:00:10 mriedem jaypipes: nvm, ComputeNode.create calls _from_db_object
18:00:16 mriedem so those allocation ratio fields will be set after create
18:02:07 jaypipes mriedem: yea
18:02:20 mriedem plus if that were the case we'd always fail this in the gate
18:07:22 sean-k-mooney mriedem: if it will help i can try to run the reporduce.sh script without the hard coded ratios in a clean vm and see if it will result in the 0.0 allocations?
18:08:31 melwitt after it pulls the defaults from the compute node object, what is the thing that is supposed to set those default values in placement?
18:09:06 melwitt the RT? makes a call to update the inventory? (reading from the bug)
18:11:16 melwitt ok, yeah here https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L902
18:11:18 sean-k-mooney i would have assumed the ratios were ultimatly set in the virt driver in update_provider_tree
18:13:58 mtreinish mriedem: ?
18:15:25 sean-k-mooney melwitt: oh so we get tree from placement, and pass it to the driver in the update_provider_tree call then normalise which set the defulats then the report_clinet updates placement?
18:17:56 mriedem mtreinish: "we work hard and we play hard"
18:18:14 mriedem mtreinish: if you're going to be around at all anymore, i need it to be for my quick simpsons references
18:18:14 sean-k-mooney melwitt: the xenapi driver does not impmentd update provider tree yet so we are hittig the excpet block https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L907-L921
18:18:52 mriedem melwitt: https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L902 doesn't update placement
18:18:57 mriedem it updates a view of the provider tree locally
18:19:07 melwitt sean-k-mooney: was just thinking about that and whether it's related. does that mean it won't update placement?
18:19:13 mriedem reportclient.update_from_provider_tree(context, prov_tree) is the call that is meant to send the changes from local to remote (placement)
18:19:17 melwitt mriedem: I see, ok, I was just about to look for that
18:19:49 sean-k-mooney mriedem: but we dont call that in the except block...
18:19:52 melwitt mriedem: but that will fail because the xen driver has not implemented it, so then there's a fallback that will do it? trying to understand what is here
18:20:27 mriedem i actually thought the xen driver implemented update_provider_tree, but even so, self.scheduler_client.set_inventory_for_provider( should update the thing
18:20:29 mriedem in placement
18:20:34 mtreinish mriedem: heh, I didn't realize you were in the steel industry
18:20:42 mriedem rust belt baby
18:21:00 melwitt ok, hm
18:23:15 mriedem this is likely related since the report client still relies on a provider tree for caching

Earlier   Later