Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2026-04-07
16:35:03 melwitt something has to be accumulating I thought, just not sure exactly what
16:35:21 melwitt yeah this should just be for eventlet
16:35:32 sean-k-mooney so we can create fake pci devices in the fucntional tests
16:35:41 sean-k-mooney and we can run the perodics explcitly in those
16:35:58 sean-k-mooney so it might be wroth callign it like 100 times and tiem each oen to see if we can repoduce
16:36:12 sean-k-mooney sort of like in my ironic simulator patch
16:36:17 melwitt gibi said he just got an env set up and will run some experiments
16:36:19 sean-k-mooney if we can we can then debug that
16:36:58 sean-k-mooney ack i have a env live with pci device too although i currently have the pci grouping serice deployed on taht but i can at least check my logs i gess and see what the time looks like
16:37:15 sean-k-mooney melwitt: the other thing of note was the fact they said more instance make it degrade faster
16:37:28 sean-k-mooney the number of isntance dont affect the livbirt retrival path
16:37:43 sean-k-mooney so i wonder if its related to the isntace objects instead or the instance part fo this
16:38:06 melwitt I guess I assumed more instance = more PCI devices but I guess you're saying not?
16:38:23 sean-k-mooney well if its the same hosts in both cases
16:38:27 sean-k-mooney or same server sku
16:38:31 dansmith sean-k-mooney: in eventlet with tpool.proxy you're creating worker threads for every property invocation on a wrapped obejct
16:38:39 dansmith nothing to do with our threaded implementation
16:39:05 sean-k-mooney dansmith: ack yes you are hookign get_atribute or simialr and dispatch the callables to a trhead pool
16:39:10 melwitt or yeah I guess I see. so if more instances = more other unrelated tpool.Proxy things maybe
16:39:36 sean-k-mooney melwitt: i think we pull all fo the domains for example in this periodic
16:39:59 sean-k-mooney so im wondifing if its related to the number of vms with passthough devices in some way
16:40:34 sean-k-mooney i.e. if on an empy host its fixed/stable but on a host with isntance it degrades over time
16:41:39 sean-k-mooney it is odd however that it reported against _get_pci_passthrough_devices if that was the case
16:41:43 melwitt I feel like it must be something less common ... like otherwise this complaint would have come up a long time ago I would think. and more people affected
16:41:59 dansmith if we iterate all the host pci devices once (or more) per instance, then that could be the increase that scales with instances
16:42:26 sean-k-mooney https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L8921-L8925
16:42:50 dansmith melwitt: this change happened just a year ago right? so real production people are probably just starting to run it
16:42:57 sean-k-mooney we are iterating over raw nodedev defivce form libvirt here
16:43:15 melwitt dansmith: yeah a little over a year ago
16:43:28 sean-k-mooney list_all_device here https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L8913
16:43:32 melwitt but it got backported everywhere also
16:43:57 dansmith ack about the backports, that's fair
16:44:04 sean-k-mooney well no that wraped into a proxy in list_all_devices
16:44:13 sean-k-mooney https://github.com/openstack/nova/blob/master/nova/virt/libvirt/host.py#L1744
16:44:15 melwitt I had seen a lot of positive feedback about the change (since before that people were getting nova-compute lockups) but I hadn't seen negative until today
16:44:51 dansmith yeah, I mean.. it improves one thing at the expense of another, so not surprising I think
16:45:13 dansmith and if you have fewer PCI devices or slower periodic config you probably appreciate the change :)
16:45:22 melwitt yeah. I just meant as far as, the conditions under which it happens being less common or more common
16:45:28 MaxLamprecht[m] I remember about a similar problem. See my patch and comment https://review.opendev.org/c/openstack/nova/+/922855/comment/a0361266_9c58663c/. There was a bug/performance regression in libvirt < 10.5.
16:46:35 melwitt oh interesting
16:46:39 sean-k-mooney dansmith: so we are only calling it once
16:46:47 sean-k-mooney not once per isntance as we are callign it here https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L10506-L10560
16:47:20 dansmith sean-k-mooney: ack, I hadn't looked I was just saying _if_ it scales with instances and _if_ we are calling once per instance...
16:47:39 sean-k-mooney yep i was just doublecking that we were not
16:48:25 melwitt MaxLamprecht[m]: I guess though, the bug reporter did say they tested with and without tpool.Proxy wrap (assuming same libvirt version) and said they didn't get the slowdown as long as tpool.Proxy wrap was not used
16:49:27 melwitt I'm not sure if that means it could not be related. might be worth asking them on the bug what their libvirt version is regardless
16:49:45 dansmith that's expected because of the overhead it introduces.. but MaxLamprecht[m]'s change just makes us never call it again which is also an improvement regardless of the proxy
16:50:44 sean-k-mooney yes we coudl also jsut impvoe this to convert the data to dicts earlier
16:51:23 MaxLamprecht[m] ack, for us upgrading libvirt to 10.5 (which includes https://gitlab.com/libvirt/libvirt/-/commit/98f1cf88fa7e0f992d93f376418fbfb3996a9690). I would start to check this :)
16:51:44 sean-k-mooney list_all_devices is only uses in 2 places
16:51:44 MaxLamprecht[m] *solved it
16:52:26 sean-k-mooney it would not be hard to replace it with 2 function one that return the lsit of mdev names and a second that returns the capablites as a dict and just remove the tpool wrapper entirly
16:53:01 dansmith you had me at "remove the tpool wrapper entirely"
16:54:05 sean-k-mooney hum well not quite https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1225-L1242 but that call can be moved to host.py and it does not return the obejct so there is no long lived usage
16:54:34 sean-k-mooney _start_inactive_mediated_devices is only called once in init_host
16:55:04 dansmith just converting to dicts when we do it, in a single worker thread and returning (probably cached if that's appropriate) would solve all the things.. then callers can iterate over the dicts as they iterate over objects today without the penality
16:55:19 sean-k-mooney +1
16:55:45 melwitt yeah, let's not remove the wrapper entirely please and return to the lockups
16:56:07 sean-k-mooney well we can do it in phases
16:56:29 dansmith we want to remove the wrappER, just not remove the wrappING( of the initial call)
16:56:56 melwitt O_O
16:57:01 melwitt ok
16:57:36 dansmith I can cook up the start of a change if it's still not clear
16:58:01 melwitt let him cook
17:00:11 melwitt I think I understand the basic idea, I was not thinking of the distinction between wrappER and wrappING so I'm good :)
17:00:29 dansmith ack
17:00:37 melwitt regardless if anyone wants to cook a change, please do. I didn't come up with any of the ideas
17:02:51 opendevreview Takashi Kajinami proposed openstack/nova-specs master: libvirt: AMD SEV-SNP support https://review.opendev.org/c/openstack/nova-specs/+/983376
17:07:22 opendevreview Takashi Kajinami proposed openstack/nova-specs master: libvirt: AMD SEV-SNP support https://review.opendev.org/c/openstack/nova-specs/+/983376
17:07:24 melwitt just to keep things rolling with problems, I noticed it seems like the test_resize_reschedule_uses_host_lists_3_fails unit test is still nondeterministically failing somehow https://f1460b24809dc27f2808-70ad550016ca78bab2d4ae8445294bcc.ssl.cf2.rackcdn.com/openstack/243619dc5a054b6cbca4265b38ce41ed/testr_results.html
17:07:37 melwitt I don't understand why yet
17:10:07 sean-k-mooney hum i have not seen that but have nto looked latesly is it always the 3 varian or is it also failing on test_resize_reschedule_uses_host_lists_1_fail
17:10:33 sean-k-mooney all the logic is common bar the parmters os the instablity must be in https://github.com/openstack/nova/blob/b84864d9781ece8387daa967be5208b859e20d97/nova/tests/functional/test_servers.py#L3453-L3490
17:11:28 sean-k-mooney oh that keeps going https://github.com/openstack/nova/blob/b84864d9781ece8387daa967be5208b859e20d97/nova/tests/functional/test_servers.py#L3453-L3558
17:11:35 melwitt I have only seen the 3 variant. and I think gmaan did something to improve it but I don't remember. I need to look at the git blame
17:14:19 gibi initial results: 142 devs returned from libvirt (10.0), no instances on the host. Over 10 periodic runs I don't see any trends. But I confirm that eventlet mode is slower overall.
17:14:20 sean-k-mooney is that on master
17:14:43 gibi no proxy: _get_pci_passthrough_devices took 0.137605510999947 secs for 142 libvirt devices
17:14:54 gibi with proxy: periodic update_available_resource took 3.2481384640000215 secs
17:14:57 gibi sorry
17:15:07 gibi with proxy: _get_pci_passthrough_devices took 0.5433623499998248 secs for 142 libvirt devices
17:15:18 sean-k-mooney i asume you just seting the env flag on master?
17:15:22 gibi yepp
17:15:31 sean-k-mooney its nice we can do that :)
17:15:35 gibi :)
17:15:50 gibi I will start adding VMs
17:16:02 gibi and taking a longer measurement window
17:16:51 gibi this is the current data I have https://paste.openstack.org/show/bNCNkXgjaGe1EXOjAxNf/
17:16:59 sean-k-mooney they said 60 repocues it but if we could get 4-8 it should still be visible it just might take longer to notice but runniing the perodic like ever 10 second should offset that
17:17:41 gibi yepp
17:18:04 sean-k-mooney i guess those are new logs
17:18:22 sean-k-mooney let me check my sriov dev env for the overall perod time quickkly
17:18:55 gibi yeah I needed new logs otherwise it is not clear how long things take
17:19:14 sean-k-mooney i think in debug mode we print the total perodic time
17:21:41 sean-k-mooney Lock "compute_resources" "released" by "nova.compute.resource_tracker.ResourceTracker._update_available_resource" :: held 2.483s
17:21:49 sean-k-mooney wo we have thsoe logs
17:22:08 sean-k-mooney but im not sure if the call is in that loc
17:22:21 sean-k-mooney it ideally is not
17:24:27 sean-k-mooney ok no we are computing the resouce view well before that
17:25:08 gibi I'm wondering that the "scaling with the number of VMs" is an artifact of the compute node being more and more under load by those VMs. It is definitely the case for my compute by default. I will measure this with VMs in shutoff state to limit that effect

Earlier   Later