| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-08-21 | |||
| 14:11:39 | openstackgerrit | Adam Spiers proposed openstack/nova master: Enable booting of libvirt guests with AMD SEV memory encryption https://review.opendev.org/666616 | |
| 14:20:34 | openstackgerrit | Matthew Booth proposed openstack/nova master: libvirt: Fix service-wide pauses caused by un-proxied libvirt calls https://review.opendev.org/677736 | |
| 14:36:54 | openstackgerrit | Adam Spiers proposed openstack/nova master: libvirt: Parse the 'os' element from domainCapabilities https://review.opendev.org/673790 | |
| 14:39:53 | mriedem | mdbooth: can we avoid the libvirt-python test dependency? i think that will mean local dev needs to install a native package, from what i remember years ago anyway | |
| 14:40:16 | mdbooth | mriedem: Yeah, I threw that in there to have a starting point, tbh | |
| 14:40:20 | mriedem | and it was a source of problems before too | |
| 14:40:22 | mriedem | ok | |
| 14:40:31 | mdbooth | mriedem: I mostly wanted to get a full tempest run against it. | |
| 14:40:42 | mriedem | so WIP it? | |
| 14:40:55 | mdbooth | mriedem: Will do. | |
| 14:41:01 | mriedem | thanks | |
| 14:41:28 | mdbooth | mriedem: I really want eyes on it, though :) Also, ideas for useful testing without using libvirt. | |
| 14:42:00 | mdbooth | Might look at something using fakelibvirt | |
| 14:42:07 | mriedem | i assumed fakelibvirt | |
| 14:42:12 | mriedem | since that's what we always use | |
| 14:42:28 | sean-k-mooney | mdbooth: you porbaly could mock the call to inspect | |
| 14:42:48 | sean-k-mooney | and pass in a set of different moduels for it to use and assert they were wrapped | |
| 14:43:13 | mdbooth | sean-k-mooney: The test I really want to do is that Host.list_instance_domains() returns <list of tpool.Proxy(virDomain)> | |
| 14:43:56 | mdbooth | If I can get that, I'll be confident that a bunch of things are lined up | |
| 14:44:08 | sean-k-mooney | i think that would be doable with out the really libvirt module | |
| 14:44:23 | mdbooth | I think so | |
| 14:44:39 | mriedem | were you going to open an issue against eventlet for the not-handling-list thing? | |
| 14:44:51 | sean-k-mooney | we might have to asser it returns <list of tpool.Proxy(mock)> | |
| 14:45:36 | sean-k-mooney | or fakelibvirt.virDomain but we should be able to cover the behaivor i think | |
| 14:48:08 | openstackgerrit | Stephen Finucane proposed openstack/nova master: scheduler: Flatten 'ResourceRequest.from_extra_specs', 'from_image_props' https://review.opendev.org/674894 | |
| 14:48:08 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove 'hw:cpu_policy', 'hw:mem_page_size' extra specs from API samples https://review.opendev.org/675338 | |
| 14:48:09 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Fold in argument to '_update_provider_tree_for_vgpu' https://review.opendev.org/676729 | |
| 14:48:09 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Add support for translating CPU policy extra specs, image meta https://review.opendev.org/671801 | |
| 14:48:10 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Add reshaper for PCPU https://review.opendev.org/674895 | |
| 14:49:31 | dtantsur | hey folks. TheJulia and I might be seeing a race condition during ironic node take over. | |
| 14:50:00 | dtantsur | it seems that when the hash ring rebalances, the old n-cpu may clean up the node records, including the resource provide in placement | |
| 14:50:10 | sean-k-mooney | node takeover e.g. when a node is moved to a different ironic compute service | |
| 14:50:13 | dtantsur | yep | |
| 14:50:48 | sean-k-mooney | is this for all node or just ones in mantaince mode | |
| 14:50:56 | dtantsur | it was for a normal node | |
| 14:51:11 | dtantsur | apparently under some condition the new (receiving) n-cpu may not re-create the resource provider for the node it took over | |
| 14:51:19 | sean-k-mooney | dtantsur: ok mnaser noted there is an issue with nodes in mainance mode being deleted | |
| 14:51:34 | dtantsur | I think it was fixed by mriedem recently | |
| 14:52:03 | sean-k-mooney | proably i just head about it a day or two ago | |
| 14:52:39 | mnaser | sean-k-mooney, dtantsur, TheJulia: https://review.opendev.org/#/c/675704/ is the stack you might want to look at | |
| 14:52:41 | sean-k-mooney | dtantsur: its possible the reciving node check if the RP exists pefore teh souce node deletes it | |
| 14:52:41 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove deprecated [neutron]/url option https://review.opendev.org/677755 | |
| 14:53:27 | dtantsur | yep, that's my guess | |
| 14:53:39 | dtantsur | relevant log lines: https://etherpad.openstack.org/p/zCQ8vPcGX6 | |
| 14:53:59 | sean-k-mooney | the first patch looks relevent https://review.opendev.org/#/c/675704/ | |
| 14:54:00 | jroll | ? | |
| 14:54:09 | mriedem | mordred: efried: i'm removing the old deprecated [neutron]/url option in https://review.opendev.org/677755 and i noticed that the neutron install guide still uses it: https://bugs.launchpad.net/neutron/+bug/1840930 - as i mentioned in that neutron bug, we could just replace with endpoint_override but i don't think that's what we want, right? | |
| 14:54:10 | openstack | Launchpad bug 1840930 in neutron "Networking service in neutron - install guide says to configure nova with [neutron]/url which is deprecated" [Undecided,New] | |
| 14:54:26 | mriedem | mordred: efried: we want to avoid endpoint_override and just have config for letting KSA do it's thing with the service catalog | |
| 14:54:33 | dtantsur | jroll: see scrollback, potential race condition on take over | |
| 14:54:46 | dtantsur | you may know this code better than me | |
| 14:54:50 | TheJulia | So an important note from what dtantsur is mentioning is that we're seeing this in our grenade job, which means this is when stein is still running from what we see in the logs | |
| 14:55:32 | mriedem | stein doesn't have the fix yet | |
| 14:55:36 | mriedem | waiting for stable reviews | |
| 14:55:49 | mriedem | https://review.opendev.org/#/q/topic:bug/1839560+branch:stable/stein | |
| 14:55:50 | sean-k-mooney | TheJulia: is this only before we do the upgrade. | |
| 14:55:52 | jroll | there we go | |
| 14:56:00 | jroll | dtantsur: side question: does the new compute service re-add it on the following RT run? | |
| 14:56:06 | dtantsur | jroll: nope, never | |
| 14:56:11 | TheJulia | sean-k-mooney: well, we never actually get to perform the upgrade | |
| 14:56:12 | jroll | oh boy | |
| 14:56:13 | dtantsur | it becomes orphaned essentially | |
| 14:56:16 | efried | mriedem: Right; for people who just gotta specify, they can s/url/endpoint_override/, but it is better to just let the service catalog do its thing. | |
| 14:56:38 | mriedem | efried: yeah, i think the question is - for an install guide, we don't want to document the exceptional path | |
| 14:56:45 | sean-k-mooney | TheJulia: i was wondering if ye tried it manually but ya it would fail tempest and not upgrade so make sense | |
| 14:56:51 | mriedem | so i was just going to remove the url mention from the install guide | |
| 14:57:09 | mriedem | dtantsur: TheJulia: i'm curious, this has been a thing since rocky, so why are you hitting it now? | |
| 14:57:12 | mriedem | new tests? | |
| 14:57:13 | TheJulia | sean-k-mooney: we wouldn't even make it to tempest, there is a resource check where placement is consulted | |
| 14:57:32 | TheJulia | if we don't have the number of "baremetal" nodes, we eventually time out and fail | |
| 14:57:37 | mriedem | https://review.opendev.org/#/c/571535/ is since rocky i mean | |
| 14:57:37 | dtantsur | mriedem: I'm curious as well. The simplest explanation is that it's become more likely for some unrelated reason. | |
| 14:57:50 | efried | mriedem: We should just be able to document "do this the same way you would do any service backed by the catalog". mordred might have a central document somewhere that recommends best practices for that. If we don't have that doc, we should. | |
| 14:58:12 | sean-k-mooney | TheJulia: well if we can recreate it reliably we proably should add a depens on against the stien backport and see if that resolves it | |
| 14:58:21 | TheJulia | We've seen some fun variations in performance so I guess any changes are going to cause us to find races more often.... | |
| 14:58:22 | mriedem | jroll: before the fix, the old compute will delete the comptue node and resource provider, but the compute node record is soft deleted; the new compute will try to create the compute node record with the same uuid - which has a unique constraint on it - and fail, so it won't re-create the resource provider | |
| 14:59:41 | TheJulia | sean-k-mooney: could work if it survives a couple rechecks... I think the overall job failure rate has been only 20-30%. | |
| 15:00:06 | mriedem | i don't know if this matters for y'all, but when the old compute deletes the compute node and resource provider, it will also delete any allocations for an instance on that node/provider, and the new compute won't re-create those allocations... | |
| 15:00:13 | sean-k-mooney | that is still high enough to be concerned | |
| 15:00:24 | mriedem | back in ocata/pike the new could RT would re-create the allocations | |
| 15:00:28 | mriedem | but that code was removed... | |
| 15:00:34 | dtantsur | ouch | |
| 15:00:42 | jroll | mriedem: yep, that makes sense. we're seeing something similar in ocata recently, but I can't decide if this is related (as the compute_nodes table is also affected) | |
| 15:00:55 | jroll | um, that allocations thing seems... really bad | |
| 15:00:59 | mriedem | jroll: the "re-use the node uuid for the compute node uuid" thing was only rocky | |
| 15:01:14 | jroll | mriedem: right, that's just why I was asking for more info | |
| 15:01:34 | sean-k-mooney | jroll: deleteing all the allocation is require to be able to delet eh placmeent RP | |
| 15:01:59 | mriedem | jroll: https://review.opendev.org/#/c/602481/ | |
| 15:02:05 | sean-k-mooney | if we dont do that then we cant delete the rp wehn we delete the compute service reliably | |
| 15:02:07 | jroll | sean-k-mooney: sure, but in an ironic world where we churn these "compute nodes" all the time, that seems scary | |
| 15:02:18 | mriedem | ^ removed the thing so that the ironic driver would always tell the RT to report the allocations | |
| 15:02:19 | jroll | maybe I'm misunderstanding how this works | |
| 15:02:41 | jroll | and I really don't have time to deep dive this part of nova today :( | |
| 15:02:52 | mriedem | since i never think if ironic node re-balancing stuff, and we don't test it (at least in nova) i guess i didn't think about this | |
| 15:03:17 | mriedem | i would also think that johnthetubaguy (who added this workaround i think) and stackhpc would have noticed a problem by now | |
| 15:03:27 | mriedem | but i don't know what version of openstack they are giving to customers | |
| 15:03:38 | mriedem | https://review.opendev.org/#/c/602481/ was stein | |
| 15:04:08 | sean-k-mooney | when we are rebalcne we are moving the compute nodes between compute service right. and in an ironic case we have mupliple compute nodes (1 per server) each with there one placment RP associates to a singel compute service correct | |
| 15:04:18 | sean-k-mooney | just making sure i have that right in my head | |
| 15:04:44 | mriedem | ironic node : compute node : instance = 1:1:1 | |