Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-08-21
14:51:34 dtantsur I think it was fixed by mriedem recently
14:52:03 sean-k-mooney proably i just head about it a day or two ago
14:52:39 mnaser sean-k-mooney, dtantsur, TheJulia: https://review.opendev.org/#/c/675704/ is the stack you might want to look at
14:52:41 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove deprecated [neutron]/url option https://review.opendev.org/677755
14:52:41 sean-k-mooney dtantsur: its possible the reciving node check if the RP exists pefore teh souce node deletes it
14:53:27 dtantsur yep, that's my guess
14:53:39 dtantsur relevant log lines: https://etherpad.openstack.org/p/zCQ8vPcGX6
14:53:59 sean-k-mooney the first patch looks relevent https://review.opendev.org/#/c/675704/
14:54:00 jroll ?
14:54:09 mriedem mordred: efried: i'm removing the old deprecated [neutron]/url option in https://review.opendev.org/677755 and i noticed that the neutron install guide still uses it: https://bugs.launchpad.net/neutron/+bug/1840930 - as i mentioned in that neutron bug, we could just replace with endpoint_override but i don't think that's what we want, right?
14:54:10 openstack Launchpad bug 1840930 in neutron "Networking service in neutron - install guide says to configure nova with [neutron]/url which is deprecated" [Undecided,New]
14:54:26 mriedem mordred: efried: we want to avoid endpoint_override and just have config for letting KSA do it's thing with the service catalog
14:54:33 dtantsur jroll: see scrollback, potential race condition on take over
14:54:46 dtantsur you may know this code better than me
14:54:50 TheJulia So an important note from what dtantsur is mentioning is that we're seeing this in our grenade job, which means this is when stein is still running from what we see in the logs
14:55:32 mriedem stein doesn't have the fix yet
14:55:36 mriedem waiting for stable reviews
14:55:49 mriedem https://review.opendev.org/#/q/topic:bug/1839560+branch:stable/stein
14:55:50 sean-k-mooney TheJulia: is this only before we do the upgrade.
14:55:52 jroll there we go
14:56:00 jroll dtantsur: side question: does the new compute service re-add it on the following RT run?
14:56:06 dtantsur jroll: nope, never
14:56:11 TheJulia sean-k-mooney: well, we never actually get to perform the upgrade
14:56:12 jroll oh boy
14:56:13 dtantsur it becomes orphaned essentially
14:56:16 efried mriedem: Right; for people who just gotta specify, they can s/url/endpoint_override/, but it is better to just let the service catalog do its thing.
14:56:38 mriedem efried: yeah, i think the question is - for an install guide, we don't want to document the exceptional path
14:56:45 sean-k-mooney TheJulia: i was wondering if ye tried it manually but ya it would fail tempest and not upgrade so make sense
14:56:51 mriedem so i was just going to remove the url mention from the install guide
14:57:09 mriedem dtantsur: TheJulia: i'm curious, this has been a thing since rocky, so why are you hitting it now?
14:57:12 mriedem new tests?
14:57:13 TheJulia sean-k-mooney: we wouldn't even make it to tempest, there is a resource check where placement is consulted
14:57:32 TheJulia if we don't have the number of "baremetal" nodes, we eventually time out and fail
14:57:37 dtantsur mriedem: I'm curious as well. The simplest explanation is that it's become more likely for some unrelated reason.
14:57:37 mriedem https://review.opendev.org/#/c/571535/ is since rocky i mean
14:57:50 efried mriedem: We should just be able to document "do this the same way you would do any service backed by the catalog". mordred might have a central document somewhere that recommends best practices for that. If we don't have that doc, we should.
14:58:12 sean-k-mooney TheJulia: well if we can recreate it reliably we proably should add a depens on against the stien backport and see if that resolves it
14:58:21 TheJulia We've seen some fun variations in performance so I guess any changes are going to cause us to find races more often....
14:58:22 mriedem jroll: before the fix, the old compute will delete the comptue node and resource provider, but the compute node record is soft deleted; the new compute will try to create the compute node record with the same uuid - which has a unique constraint on it - and fail, so it won't re-create the resource provider
14:59:41 TheJulia sean-k-mooney: could work if it survives a couple rechecks... I think the overall job failure rate has been only 20-30%.
15:00:06 mriedem i don't know if this matters for y'all, but when the old compute deletes the compute node and resource provider, it will also delete any allocations for an instance on that node/provider, and the new compute won't re-create those allocations...
15:00:13 sean-k-mooney that is still high enough to be concerned
15:00:24 mriedem back in ocata/pike the new could RT would re-create the allocations
15:00:28 mriedem but that code was removed...
15:00:34 dtantsur ouch
15:00:42 jroll mriedem: yep, that makes sense. we're seeing something similar in ocata recently, but I can't decide if this is related (as the compute_nodes table is also affected)
15:00:55 jroll um, that allocations thing seems... really bad
15:00:59 mriedem jroll: the "re-use the node uuid for the compute node uuid" thing was only rocky
15:01:14 jroll mriedem: right, that's just why I was asking for more info
15:01:34 sean-k-mooney jroll: deleteing all the allocation is require to be able to delet eh placmeent RP
15:01:59 mriedem jroll: https://review.opendev.org/#/c/602481/
15:02:05 sean-k-mooney if we dont do that then we cant delete the rp wehn we delete the compute service reliably
15:02:07 jroll sean-k-mooney: sure, but in an ironic world where we churn these "compute nodes" all the time, that seems scary
15:02:18 mriedem ^ removed the thing so that the ironic driver would always tell the RT to report the allocations
15:02:19 jroll maybe I'm misunderstanding how this works
15:02:41 jroll and I really don't have time to deep dive this part of nova today :(
15:02:52 mriedem since i never think if ironic node re-balancing stuff, and we don't test it (at least in nova) i guess i didn't think about this
15:03:17 mriedem i would also think that johnthetubaguy (who added this workaround i think) and stackhpc would have noticed a problem by now
15:03:27 mriedem but i don't know what version of openstack they are giving to customers
15:03:38 mriedem https://review.opendev.org/#/c/602481/ was stein
15:04:08 sean-k-mooney when we are rebalcne we are moving the compute nodes between compute service right. and in an ironic case we have mupliple compute nodes (1 per server) each with there one placment RP associates to a singel compute service correct
15:04:18 sean-k-mooney just making sure i have that right in my head
15:04:44 mriedem ironic node : compute node : instance = 1:1:1
15:04:56 jroll correct, until the end - there is a placement RP for each ironic node / compute node
15:04:56 mriedem + : resource provider
15:05:27 sean-k-mooney jroll: yep that is want i ment
15:05:40 sean-k-mooney one placment RP per compute/ironic node
15:05:53 jroll cool, you are correct then
15:06:36 sean-k-mooney durign a reblance we are not actully delet ing he nova compute service we are just moveing the comnpute nodes so we really should not need to update placmeent
15:06:49 sean-k-mooney or rather delete the compute node/placmente RP
15:07:30 mriedem the compute manager doesn't know you're doing a re-balance
15:07:31 sean-k-mooney all we should need to do on the nova side is modify the service to compute node mapping but i dont know how we do that today
15:07:51 jroll except that a rebalance happens when the number of compute services changes. so if you shut down a compute service (which deletes compute nodes and RPs), ironic nodes get moved to another, and those compute nodes and RPs are re-created
15:07:54 mriedem it knows, "oh i was managing node x before, but now the driver says i'm not, so delete"
15:08:29 mriedem jroll: yeah and before rocky, those new compute nodes and RPs would all have a unique uuid
15:08:34 mriedem since rocky, they re-use the node uuid
15:08:48 jroll yeah
15:09:04 mriedem so i think with my recent fix that's all fine,
15:09:16 mriedem the question is what happens with allocations against the re-balanced node resource provider
15:09:20 mriedem b/c i'm guessing since stein those are gone
15:09:23 sean-k-mooney mriedem: right but im wonder that im not "managing because the drive said so, therefor delete is required"
15:09:28 jroll yeah, that's what worries me
15:09:54 mriedem i guess someone needs to do some testing
15:10:00 sean-k-mooney in a libvirt case we would normaly delete the compute service and have it delete the compute nodes under neat it
15:10:26 mriedem sean-k-mooney: the compute service doesn't delete itself,
15:10:28 mriedem that happens in the API
15:10:34 mriedem DELETE /os-services/{service_id}
15:10:35 sean-k-mooney well yes
15:10:41 mriedem which is related to a number of other fun issues
15:10:43 sean-k-mooney i know that but that is what i ment
15:10:44 mriedem which are in the ML
15:11:07 sean-k-mooney we externally trigger the deletion of the compute service when removing a libvir host
15:11:21 sean-k-mooney does vmware do something similar to ironic
15:11:44 sean-k-mooney they have multiple compute nodes in the same service too right
15:11:51 mriedem http://lists.openstack.org/pipermail/openstack-discuss/2019-June/007135.html
15:12:06 mriedem sean-k-mooney: no, vmware driver does not do that since kilo or liberty
15:13:44 mriedem TheJulia: tl;dr, i think you need https://review.opendev.org/#/q/topic:bug/1839560+branch:stable/stein for your grenade failures,
15:14:01 mriedem and there might be a regression due to https://review.opendev.org/#/c/602481/ where allocations in placement are lost during a re-balance
15:14:06 mriedem but someone needs to test that
15:14:19 TheJulia mriedem: sean-k-mooney: I added a depends-on flag to one of my test patches for the patches above, if I see the issue re-appear on that job, I'll let you all know

Earlier   Later