Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-08-21
14:54:50 TheJulia So an important note from what dtantsur is mentioning is that we're seeing this in our grenade job, which means this is when stein is still running from what we see in the logs
14:55:32 mriedem stein doesn't have the fix yet
14:55:36 mriedem waiting for stable reviews
14:55:49 mriedem https://review.opendev.org/#/q/topic:bug/1839560+branch:stable/stein
14:55:50 sean-k-mooney TheJulia: is this only before we do the upgrade.
14:55:52 jroll there we go
14:56:00 jroll dtantsur: side question: does the new compute service re-add it on the following RT run?
14:56:06 dtantsur jroll: nope, never
14:56:11 TheJulia sean-k-mooney: well, we never actually get to perform the upgrade
14:56:12 jroll oh boy
14:56:13 dtantsur it becomes orphaned essentially
14:56:16 efried mriedem: Right; for people who just gotta specify, they can s/url/endpoint_override/, but it is better to just let the service catalog do its thing.
14:56:38 mriedem efried: yeah, i think the question is - for an install guide, we don't want to document the exceptional path
14:56:45 sean-k-mooney TheJulia: i was wondering if ye tried it manually but ya it would fail tempest and not upgrade so make sense
14:56:51 mriedem so i was just going to remove the url mention from the install guide
14:57:09 mriedem dtantsur: TheJulia: i'm curious, this has been a thing since rocky, so why are you hitting it now?
14:57:12 mriedem new tests?
14:57:13 TheJulia sean-k-mooney: we wouldn't even make it to tempest, there is a resource check where placement is consulted
14:57:32 TheJulia if we don't have the number of "baremetal" nodes, we eventually time out and fail
14:57:37 dtantsur mriedem: I'm curious as well. The simplest explanation is that it's become more likely for some unrelated reason.
14:57:37 mriedem https://review.opendev.org/#/c/571535/ is since rocky i mean
14:57:50 efried mriedem: We should just be able to document "do this the same way you would do any service backed by the catalog". mordred might have a central document somewhere that recommends best practices for that. If we don't have that doc, we should.
14:58:12 sean-k-mooney TheJulia: well if we can recreate it reliably we proably should add a depens on against the stien backport and see if that resolves it
14:58:21 TheJulia We've seen some fun variations in performance so I guess any changes are going to cause us to find races more often....
14:58:22 mriedem jroll: before the fix, the old compute will delete the comptue node and resource provider, but the compute node record is soft deleted; the new compute will try to create the compute node record with the same uuid - which has a unique constraint on it - and fail, so it won't re-create the resource provider
14:59:41 TheJulia sean-k-mooney: could work if it survives a couple rechecks... I think the overall job failure rate has been only 20-30%.
15:00:06 mriedem i don't know if this matters for y'all, but when the old compute deletes the compute node and resource provider, it will also delete any allocations for an instance on that node/provider, and the new compute won't re-create those allocations...
15:00:13 sean-k-mooney that is still high enough to be concerned
15:00:24 mriedem back in ocata/pike the new could RT would re-create the allocations
15:00:28 mriedem but that code was removed...
15:00:34 dtantsur ouch
15:00:42 jroll mriedem: yep, that makes sense. we're seeing something similar in ocata recently, but I can't decide if this is related (as the compute_nodes table is also affected)
15:00:55 jroll um, that allocations thing seems... really bad
15:00:59 mriedem jroll: the "re-use the node uuid for the compute node uuid" thing was only rocky
15:01:14 jroll mriedem: right, that's just why I was asking for more info
15:01:34 sean-k-mooney jroll: deleteing all the allocation is require to be able to delet eh placmeent RP
15:01:59 mriedem jroll: https://review.opendev.org/#/c/602481/
15:02:05 sean-k-mooney if we dont do that then we cant delete the rp wehn we delete the compute service reliably
15:02:07 jroll sean-k-mooney: sure, but in an ironic world where we churn these "compute nodes" all the time, that seems scary
15:02:18 mriedem ^ removed the thing so that the ironic driver would always tell the RT to report the allocations
15:02:19 jroll maybe I'm misunderstanding how this works
15:02:41 jroll and I really don't have time to deep dive this part of nova today :(
15:02:52 mriedem since i never think if ironic node re-balancing stuff, and we don't test it (at least in nova) i guess i didn't think about this
15:03:17 mriedem i would also think that johnthetubaguy (who added this workaround i think) and stackhpc would have noticed a problem by now
15:03:27 mriedem but i don't know what version of openstack they are giving to customers
15:03:38 mriedem https://review.opendev.org/#/c/602481/ was stein
15:04:08 sean-k-mooney when we are rebalcne we are moving the compute nodes between compute service right. and in an ironic case we have mupliple compute nodes (1 per server) each with there one placment RP associates to a singel compute service correct
15:04:18 sean-k-mooney just making sure i have that right in my head
15:04:44 mriedem ironic node : compute node : instance = 1:1:1
15:04:56 jroll correct, until the end - there is a placement RP for each ironic node / compute node
15:04:56 mriedem + : resource provider
15:05:27 sean-k-mooney jroll: yep that is want i ment
15:05:40 sean-k-mooney one placment RP per compute/ironic node
15:05:53 jroll cool, you are correct then
15:06:36 sean-k-mooney durign a reblance we are not actully delet ing he nova compute service we are just moveing the comnpute nodes so we really should not need to update placmeent
15:06:49 sean-k-mooney or rather delete the compute node/placmente RP
15:07:30 mriedem the compute manager doesn't know you're doing a re-balance
15:07:31 sean-k-mooney all we should need to do on the nova side is modify the service to compute node mapping but i dont know how we do that today
15:07:51 jroll except that a rebalance happens when the number of compute services changes. so if you shut down a compute service (which deletes compute nodes and RPs), ironic nodes get moved to another, and those compute nodes and RPs are re-created
15:07:54 mriedem it knows, "oh i was managing node x before, but now the driver says i'm not, so delete"
15:08:29 mriedem jroll: yeah and before rocky, those new compute nodes and RPs would all have a unique uuid
15:08:34 mriedem since rocky, they re-use the node uuid
15:08:48 jroll yeah
15:09:04 mriedem so i think with my recent fix that's all fine,
15:09:16 mriedem the question is what happens with allocations against the re-balanced node resource provider
15:09:20 mriedem b/c i'm guessing since stein those are gone
15:09:23 sean-k-mooney mriedem: right but im wonder that im not "managing because the drive said so, therefor delete is required"
15:09:28 jroll yeah, that's what worries me
15:09:54 mriedem i guess someone needs to do some testing
15:10:00 sean-k-mooney in a libvirt case we would normaly delete the compute service and have it delete the compute nodes under neat it
15:10:26 mriedem sean-k-mooney: the compute service doesn't delete itself,
15:10:28 mriedem that happens in the API
15:10:34 mriedem DELETE /os-services/{service_id}
15:10:35 sean-k-mooney well yes
15:10:41 mriedem which is related to a number of other fun issues
15:10:43 sean-k-mooney i know that but that is what i ment
15:10:44 mriedem which are in the ML
15:11:07 sean-k-mooney we externally trigger the deletion of the compute service when removing a libvir host
15:11:21 sean-k-mooney does vmware do something similar to ironic
15:11:44 sean-k-mooney they have multiple compute nodes in the same service too right
15:11:51 mriedem http://lists.openstack.org/pipermail/openstack-discuss/2019-June/007135.html
15:12:06 mriedem sean-k-mooney: no, vmware driver does not do that since kilo or liberty
15:13:44 mriedem TheJulia: tl;dr, i think you need https://review.opendev.org/#/q/topic:bug/1839560+branch:stable/stein for your grenade failures,
15:14:01 mriedem and there might be a regression due to https://review.opendev.org/#/c/602481/ where allocations in placement are lost during a re-balance
15:14:06 mriedem but someone needs to test that
15:14:19 TheJulia mriedem: sean-k-mooney: I added a depends-on flag to one of my test patches for the patches above, if I see the issue re-appear on that job, I'll let you all know
15:18:33 mriedem so if the lost allocations on re-balance is a real thing, i might have a solution, which i wrote down in a comment in https://review.opendev.org/#/c/602481/
15:19:01 mriedem b/c the code that deletes the resource provider and it's allocations pre-dates the removal of the RT allocation healing code, and unsurprisingly we lost track of all those random one off moving parts
15:19:29 mriedem especially since we don't have functional testing of that stuff in tree
15:37:40 openstackgerrit Adam Spiers proposed openstack/nova master: libvirt: Parse the 'os' element from domainCapabilities https://review.opendev.org/673790
15:39:26 gibi mriedem: hi! I have no problem moving the small parts of the cross-cell resize series to the front. I can re-apply my +2 if you rebase the series
15:39:57 mriedem gibi: ok thanks
15:46:42 openstackgerrit Matt Riedemann proposed openstack/nova master: FUP for I5576fa2a67d2771614266022428b4a95487ab6d5 https://review.opendev.org/677769
15:55:45 aspiers anyone object to me splitting libvirt/test_driver.py into smaller pieces? at 23k LoC it's almost *double* the size of the next largest file in nova (compute/test_compute.py in case you're curious)
15:55:58 aspiers my editor really hates such a huge file, and so do I
15:56:09 aspiers efried_afk, sean-k-mooney: ^^^
15:56:40 aspiers of course this will probably cause a few merge conflicts so maybe we need to time it right
15:56:55 sean-k-mooney aspiers: you mean it does not like 10s of 1000s or lines
15:57:41 aspiers sean-k-mooney: yes, syntax highlighting something that big is really expensive
15:57:56 aspiers unless your highlighting algorithm cuts corners

Earlier   Later