| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-08-21 | |||
| 14:45:36 | sean-k-mooney | or fakelibvirt.virDomain but we should be able to cover the behaivor i think | |
| 14:48:08 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove 'hw:cpu_policy', 'hw:mem_page_size' extra specs from API samples https://review.opendev.org/675338 | |
| 14:48:08 | openstackgerrit | Stephen Finucane proposed openstack/nova master: scheduler: Flatten 'ResourceRequest.from_extra_specs', 'from_image_props' https://review.opendev.org/674894 | |
| 14:48:09 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Add support for translating CPU policy extra specs, image meta https://review.opendev.org/671801 | |
| 14:48:09 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Fold in argument to '_update_provider_tree_for_vgpu' https://review.opendev.org/676729 | |
| 14:48:10 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Add reshaper for PCPU https://review.opendev.org/674895 | |
| 14:49:31 | dtantsur | hey folks. TheJulia and I might be seeing a race condition during ironic node take over. | |
| 14:50:00 | dtantsur | it seems that when the hash ring rebalances, the old n-cpu may clean up the node records, including the resource provide in placement | |
| 14:50:10 | sean-k-mooney | node takeover e.g. when a node is moved to a different ironic compute service | |
| 14:50:13 | dtantsur | yep | |
| 14:50:48 | sean-k-mooney | is this for all node or just ones in mantaince mode | |
| 14:50:56 | dtantsur | it was for a normal node | |
| 14:51:11 | dtantsur | apparently under some condition the new (receiving) n-cpu may not re-create the resource provider for the node it took over | |
| 14:51:19 | sean-k-mooney | dtantsur: ok mnaser noted there is an issue with nodes in mainance mode being deleted | |
| 14:51:34 | dtantsur | I think it was fixed by mriedem recently | |
| 14:52:03 | sean-k-mooney | proably i just head about it a day or two ago | |
| 14:52:39 | mnaser | sean-k-mooney, dtantsur, TheJulia: https://review.opendev.org/#/c/675704/ is the stack you might want to look at | |
| 14:52:41 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove deprecated [neutron]/url option https://review.opendev.org/677755 | |
| 14:52:41 | sean-k-mooney | dtantsur: its possible the reciving node check if the RP exists pefore teh souce node deletes it | |
| 14:53:27 | dtantsur | yep, that's my guess | |
| 14:53:39 | dtantsur | relevant log lines: https://etherpad.openstack.org/p/zCQ8vPcGX6 | |
| 14:53:59 | sean-k-mooney | the first patch looks relevent https://review.opendev.org/#/c/675704/ | |
| 14:54:00 | jroll | ? | |
| 14:54:09 | mriedem | mordred: efried: i'm removing the old deprecated [neutron]/url option in https://review.opendev.org/677755 and i noticed that the neutron install guide still uses it: https://bugs.launchpad.net/neutron/+bug/1840930 - as i mentioned in that neutron bug, we could just replace with endpoint_override but i don't think that's what we want, right? | |
| 14:54:10 | openstack | Launchpad bug 1840930 in neutron "Networking service in neutron - install guide says to configure nova with [neutron]/url which is deprecated" [Undecided,New] | |
| 14:54:26 | mriedem | mordred: efried: we want to avoid endpoint_override and just have config for letting KSA do it's thing with the service catalog | |
| 14:54:33 | dtantsur | jroll: see scrollback, potential race condition on take over | |
| 14:54:46 | dtantsur | you may know this code better than me | |
| 14:54:50 | TheJulia | So an important note from what dtantsur is mentioning is that we're seeing this in our grenade job, which means this is when stein is still running from what we see in the logs | |
| 14:55:32 | mriedem | stein doesn't have the fix yet | |
| 14:55:36 | mriedem | waiting for stable reviews | |
| 14:55:49 | mriedem | https://review.opendev.org/#/q/topic:bug/1839560+branch:stable/stein | |
| 14:55:50 | sean-k-mooney | TheJulia: is this only before we do the upgrade. | |
| 14:55:52 | jroll | there we go | |
| 14:56:00 | jroll | dtantsur: side question: does the new compute service re-add it on the following RT run? | |
| 14:56:06 | dtantsur | jroll: nope, never | |
| 14:56:11 | TheJulia | sean-k-mooney: well, we never actually get to perform the upgrade | |
| 14:56:12 | jroll | oh boy | |
| 14:56:13 | dtantsur | it becomes orphaned essentially | |
| 14:56:16 | efried | mriedem: Right; for people who just gotta specify, they can s/url/endpoint_override/, but it is better to just let the service catalog do its thing. | |
| 14:56:38 | mriedem | efried: yeah, i think the question is - for an install guide, we don't want to document the exceptional path | |
| 14:56:45 | sean-k-mooney | TheJulia: i was wondering if ye tried it manually but ya it would fail tempest and not upgrade so make sense | |
| 14:56:51 | mriedem | so i was just going to remove the url mention from the install guide | |
| 14:57:09 | mriedem | dtantsur: TheJulia: i'm curious, this has been a thing since rocky, so why are you hitting it now? | |
| 14:57:12 | mriedem | new tests? | |
| 14:57:13 | TheJulia | sean-k-mooney: we wouldn't even make it to tempest, there is a resource check where placement is consulted | |
| 14:57:32 | TheJulia | if we don't have the number of "baremetal" nodes, we eventually time out and fail | |
| 14:57:37 | dtantsur | mriedem: I'm curious as well. The simplest explanation is that it's become more likely for some unrelated reason. | |
| 14:57:37 | mriedem | https://review.opendev.org/#/c/571535/ is since rocky i mean | |
| 14:57:50 | efried | mriedem: We should just be able to document "do this the same way you would do any service backed by the catalog". mordred might have a central document somewhere that recommends best practices for that. If we don't have that doc, we should. | |
| 14:58:12 | sean-k-mooney | TheJulia: well if we can recreate it reliably we proably should add a depens on against the stien backport and see if that resolves it | |
| 14:58:21 | TheJulia | We've seen some fun variations in performance so I guess any changes are going to cause us to find races more often.... | |
| 14:58:22 | mriedem | jroll: before the fix, the old compute will delete the comptue node and resource provider, but the compute node record is soft deleted; the new compute will try to create the compute node record with the same uuid - which has a unique constraint on it - and fail, so it won't re-create the resource provider | |
| 14:59:41 | TheJulia | sean-k-mooney: could work if it survives a couple rechecks... I think the overall job failure rate has been only 20-30%. | |
| 15:00:06 | mriedem | i don't know if this matters for y'all, but when the old compute deletes the compute node and resource provider, it will also delete any allocations for an instance on that node/provider, and the new compute won't re-create those allocations... | |
| 15:00:13 | sean-k-mooney | that is still high enough to be concerned | |
| 15:00:24 | mriedem | back in ocata/pike the new could RT would re-create the allocations | |
| 15:00:28 | mriedem | but that code was removed... | |
| 15:00:34 | dtantsur | ouch | |
| 15:00:42 | jroll | mriedem: yep, that makes sense. we're seeing something similar in ocata recently, but I can't decide if this is related (as the compute_nodes table is also affected) | |
| 15:00:55 | jroll | um, that allocations thing seems... really bad | |
| 15:00:59 | mriedem | jroll: the "re-use the node uuid for the compute node uuid" thing was only rocky | |
| 15:01:14 | jroll | mriedem: right, that's just why I was asking for more info | |
| 15:01:34 | sean-k-mooney | jroll: deleteing all the allocation is require to be able to delet eh placmeent RP | |
| 15:01:59 | mriedem | jroll: https://review.opendev.org/#/c/602481/ | |
| 15:02:05 | sean-k-mooney | if we dont do that then we cant delete the rp wehn we delete the compute service reliably | |
| 15:02:07 | jroll | sean-k-mooney: sure, but in an ironic world where we churn these "compute nodes" all the time, that seems scary | |
| 15:02:18 | mriedem | ^ removed the thing so that the ironic driver would always tell the RT to report the allocations | |
| 15:02:19 | jroll | maybe I'm misunderstanding how this works | |
| 15:02:41 | jroll | and I really don't have time to deep dive this part of nova today :( | |
| 15:02:52 | mriedem | since i never think if ironic node re-balancing stuff, and we don't test it (at least in nova) i guess i didn't think about this | |
| 15:03:17 | mriedem | i would also think that johnthetubaguy (who added this workaround i think) and stackhpc would have noticed a problem by now | |
| 15:03:27 | mriedem | but i don't know what version of openstack they are giving to customers | |
| 15:03:38 | mriedem | https://review.opendev.org/#/c/602481/ was stein | |
| 15:04:08 | sean-k-mooney | when we are rebalcne we are moving the compute nodes between compute service right. and in an ironic case we have mupliple compute nodes (1 per server) each with there one placment RP associates to a singel compute service correct | |
| 15:04:18 | sean-k-mooney | just making sure i have that right in my head | |
| 15:04:44 | mriedem | ironic node : compute node : instance = 1:1:1 | |
| 15:04:56 | jroll | correct, until the end - there is a placement RP for each ironic node / compute node | |
| 15:04:56 | mriedem | + : resource provider | |
| 15:05:27 | sean-k-mooney | jroll: yep that is want i ment | |
| 15:05:40 | sean-k-mooney | one placment RP per compute/ironic node | |
| 15:05:53 | jroll | cool, you are correct then | |
| 15:06:36 | sean-k-mooney | durign a reblance we are not actully delet ing he nova compute service we are just moveing the comnpute nodes so we really should not need to update placmeent | |
| 15:06:49 | sean-k-mooney | or rather delete the compute node/placmente RP | |
| 15:07:30 | mriedem | the compute manager doesn't know you're doing a re-balance | |
| 15:07:31 | sean-k-mooney | all we should need to do on the nova side is modify the service to compute node mapping but i dont know how we do that today | |
| 15:07:51 | jroll | except that a rebalance happens when the number of compute services changes. so if you shut down a compute service (which deletes compute nodes and RPs), ironic nodes get moved to another, and those compute nodes and RPs are re-created | |
| 15:07:54 | mriedem | it knows, "oh i was managing node x before, but now the driver says i'm not, so delete" | |
| 15:08:29 | mriedem | jroll: yeah and before rocky, those new compute nodes and RPs would all have a unique uuid | |
| 15:08:34 | mriedem | since rocky, they re-use the node uuid | |
| 15:08:48 | jroll | yeah | |
| 15:09:04 | mriedem | so i think with my recent fix that's all fine, | |
| 15:09:16 | mriedem | the question is what happens with allocations against the re-balanced node resource provider | |
| 15:09:20 | mriedem | b/c i'm guessing since stein those are gone | |
| 15:09:23 | sean-k-mooney | mriedem: right but im wonder that im not "managing because the drive said so, therefor delete is required" | |
| 15:09:28 | jroll | yeah, that's what worries me | |
| 15:09:54 | mriedem | i guess someone needs to do some testing | |
| 15:10:00 | sean-k-mooney | in a libvirt case we would normaly delete the compute service and have it delete the compute nodes under neat it | |
| 15:10:26 | mriedem | sean-k-mooney: the compute service doesn't delete itself, | |
| 15:10:28 | mriedem | that happens in the API | |