Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2026-04-07
16:44:15 melwitt I had seen a lot of positive feedback about the change (since before that people were getting nova-compute lockups) but I hadn't seen negative until today
16:44:51 dansmith yeah, I mean.. it improves one thing at the expense of another, so not surprising I think
16:45:13 dansmith and if you have fewer PCI devices or slower periodic config you probably appreciate the change :)
16:45:22 melwitt yeah. I just meant as far as, the conditions under which it happens being less common or more common
16:45:28 MaxLamprecht[m] I remember about a similar problem. See my patch and comment https://review.opendev.org/c/openstack/nova/+/922855/comment/a0361266_9c58663c/. There was a bug/performance regression in libvirt < 10.5.
16:46:35 melwitt oh interesting
16:46:39 sean-k-mooney dansmith: so we are only calling it once
16:46:47 sean-k-mooney not once per isntance as we are callign it here https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L10506-L10560
16:47:20 dansmith sean-k-mooney: ack, I hadn't looked I was just saying _if_ it scales with instances and _if_ we are calling once per instance...
16:47:39 sean-k-mooney yep i was just doublecking that we were not
16:48:25 melwitt MaxLamprecht[m]: I guess though, the bug reporter did say they tested with and without tpool.Proxy wrap (assuming same libvirt version) and said they didn't get the slowdown as long as tpool.Proxy wrap was not used
16:49:27 melwitt I'm not sure if that means it could not be related. might be worth asking them on the bug what their libvirt version is regardless
16:49:45 dansmith that's expected because of the overhead it introduces.. but MaxLamprecht[m]'s change just makes us never call it again which is also an improvement regardless of the proxy
16:50:44 sean-k-mooney yes we coudl also jsut impvoe this to convert the data to dicts earlier
16:51:23 MaxLamprecht[m] ack, for us upgrading libvirt to 10.5 (which includes https://gitlab.com/libvirt/libvirt/-/commit/98f1cf88fa7e0f992d93f376418fbfb3996a9690). I would start to check this :)
16:51:44 sean-k-mooney list_all_devices is only uses in 2 places
16:51:44 MaxLamprecht[m] *solved it
16:52:26 sean-k-mooney it would not be hard to replace it with 2 function one that return the lsit of mdev names and a second that returns the capablites as a dict and just remove the tpool wrapper entirly
16:53:01 dansmith you had me at "remove the tpool wrapper entirely"
16:54:05 sean-k-mooney hum well not quite https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1225-L1242 but that call can be moved to host.py and it does not return the obejct so there is no long lived usage
16:54:34 sean-k-mooney _start_inactive_mediated_devices is only called once in init_host
16:55:04 dansmith just converting to dicts when we do it, in a single worker thread and returning (probably cached if that's appropriate) would solve all the things.. then callers can iterate over the dicts as they iterate over objects today without the penality
16:55:19 sean-k-mooney +1
16:55:45 melwitt yeah, let's not remove the wrapper entirely please and return to the lockups
16:56:07 sean-k-mooney well we can do it in phases
16:56:29 dansmith we want to remove the wrappER, just not remove the wrappING( of the initial call)
16:56:56 melwitt O_O
16:57:01 melwitt ok
16:57:36 dansmith I can cook up the start of a change if it's still not clear
16:58:01 melwitt let him cook
17:00:11 melwitt I think I understand the basic idea, I was not thinking of the distinction between wrappER and wrappING so I'm good :)
17:00:29 dansmith ack
17:00:37 melwitt regardless if anyone wants to cook a change, please do. I didn't come up with any of the ideas
17:02:51 opendevreview Takashi Kajinami proposed openstack/nova-specs master: libvirt: AMD SEV-SNP support https://review.opendev.org/c/openstack/nova-specs/+/983376
17:07:22 opendevreview Takashi Kajinami proposed openstack/nova-specs master: libvirt: AMD SEV-SNP support https://review.opendev.org/c/openstack/nova-specs/+/983376
17:07:24 melwitt just to keep things rolling with problems, I noticed it seems like the test_resize_reschedule_uses_host_lists_3_fails unit test is still nondeterministically failing somehow https://f1460b24809dc27f2808-70ad550016ca78bab2d4ae8445294bcc.ssl.cf2.rackcdn.com/openstack/243619dc5a054b6cbca4265b38ce41ed/testr_results.html
17:07:37 melwitt I don't understand why yet
17:10:07 sean-k-mooney hum i have not seen that but have nto looked latesly is it always the 3 varian or is it also failing on test_resize_reschedule_uses_host_lists_1_fail
17:10:33 sean-k-mooney all the logic is common bar the parmters os the instablity must be in https://github.com/openstack/nova/blob/b84864d9781ece8387daa967be5208b859e20d97/nova/tests/functional/test_servers.py#L3453-L3490
17:11:28 sean-k-mooney oh that keeps going https://github.com/openstack/nova/blob/b84864d9781ece8387daa967be5208b859e20d97/nova/tests/functional/test_servers.py#L3453-L3558
17:11:35 melwitt I have only seen the 3 variant. and I think gmaan did something to improve it but I don't remember. I need to look at the git blame
17:14:19 gibi initial results: 142 devs returned from libvirt (10.0), no instances on the host. Over 10 periodic runs I don't see any trends. But I confirm that eventlet mode is slower overall.
17:14:20 sean-k-mooney is that on master
17:14:43 gibi no proxy: _get_pci_passthrough_devices took 0.137605510999947 secs for 142 libvirt devices
17:14:54 gibi with proxy: periodic update_available_resource took 3.2481384640000215 secs
17:14:57 gibi sorry
17:15:07 gibi with proxy: _get_pci_passthrough_devices took 0.5433623499998248 secs for 142 libvirt devices
17:15:18 sean-k-mooney i asume you just seting the env flag on master?
17:15:22 gibi yepp
17:15:31 sean-k-mooney its nice we can do that :)
17:15:35 gibi :)
17:15:50 gibi I will start adding VMs
17:16:02 gibi and taking a longer measurement window
17:16:51 gibi this is the current data I have https://paste.openstack.org/show/bNCNkXgjaGe1EXOjAxNf/
17:16:59 sean-k-mooney they said 60 repocues it but if we could get 4-8 it should still be visible it just might take longer to notice but runniing the perodic like ever 10 second should offset that
17:17:41 gibi yepp
17:18:04 sean-k-mooney i guess those are new logs
17:18:22 sean-k-mooney let me check my sriov dev env for the overall perod time quickkly
17:18:55 gibi yeah I needed new logs otherwise it is not clear how long things take
17:19:14 sean-k-mooney i think in debug mode we print the total perodic time
17:21:41 sean-k-mooney Lock "compute_resources" "released" by "nova.compute.resource_tracker.ResourceTracker._update_available_resource" :: held 2.483s
17:21:49 sean-k-mooney wo we have thsoe logs
17:22:08 sean-k-mooney but im not sure if the call is in that loc
17:22:21 sean-k-mooney it ideally is not
17:24:27 sean-k-mooney ok no we are computing the resouce view well before that
17:25:08 gibi I'm wondering that the "scaling with the number of VMs" is an artifact of the compute node being more and more under load by those VMs. It is definitely the case for my compute by default. I will measure this with VMs in shutoff state to limit that effect
17:25:35 sean-k-mooney even so the pull perodic time for me locally is 12 seconds
17:25:38 sean-k-mooney *full
17:26:01 sean-k-mooney well there will be more contention for the compute_resouces lock
17:26:09 sean-k-mooney in teh resouce tracker formother perodics
17:27:10 sean-k-mooney we do print the wait time ock "compute_resources" acquired by "nova.compute.resource_tracker.ResourceTracker._update_available_resource" :: waited 0.001s so mabe that a factor
17:28:40 sean-k-mooney although locally the other thing that uses it is Lock "compute_resources" acquired by "nova.compute.resource_tracker.ResourceTracker.clean_compute_node_cache"
17:29:03 sean-k-mooney i think we aquire it when bootign vms as well
17:29:19 sean-k-mooney but in terms of perodic that contenion does nto seam very high
17:31:49 sean-k-mooney Active: active (running) since Wed 2026-03-11 17:16:44 UTC; 3 weeks 6 days ago
17:31:58 sean-k-mooney so i do not currently have instnace on this env
17:32:26 sean-k-mooney but i ahve had them runign with pci device in the past so it does seam to return to a sane baseline at least on an empty host
17:34:04 sean-k-mooney gibi i guess the next best thign to do is leave it run over night in your env and see if it repoduces when there are isntances
17:34:59 gibi 10 VM that are not using PCI devs on the host and are in shutoff state does not increase the time of the _get_pci_passthrough_devices call. It does increase the time of th whole periodic obviously. I don't yet see any treands in either of the times over time
17:35:23 gibi sean-k-mooney: yeah I will, I just try to see what setup make sense to use
17:35:54 gibi next I will try with PCI user VMs
17:37:46 gibi I have 6+6 VFs (2 PFs) so each VM can take a VF
17:37:48 sean-k-mooney ack im currnetly comparing in my cybrog/pci groupign env. ill leave it alone for not but if you dong see anygint i can maybe run ~50-80 vms
17:39:00 sean-k-mooney i belive i have 2 sriov capable nic on this host and i shoudl barely have enough disk/ram but i havent really pushed this host to find its limits
17:44:42 gibi I don't see any timing data difference between 10VMs without PCI usage and 10 VMs with one VF usage each (both cases the VMs are in shutoff)
17:45:31 gibi so what I will do I will keep the 10 VMs with one VF each,take a GMR and then let the periodic run every 10 seconds during the night
17:47:59 sean-k-mooney hum im not sure what runing vs not would cause in this context
17:48:34 sean-k-mooney techinaly that does change the retivial of mac adresses for VFs
17:49:01 sean-k-mooney i.e. if livbirt is restart after the vf is passed to the vm it may not see the mac
17:49:34 sean-k-mooney but did the bug say it was realted to runign/active vms or are you just leaving them stop to reduces load
17:49:37 gibi I simply want to avoid the cpu usage noise of the 10 VMs running
17:49:44 sean-k-mooney ah ok
17:49:55 sean-k-mooney you could start them and then pause them
17:49:57 gibi it is non 0 even with cirros
17:50:01 gibi sure
17:50:03 gibi that is better
17:50:25 sean-k-mooney but i suspect we wont see much of a delta on the perodic althoug hif we do it would be an interesting datapoint i guess
18:17:33 gibi a quick look in two GMRs ~5 minutes distance I don't see increasing number of Green Threads or Threads reported.
18:25:59 gibi OK. This is the starting state of my measurement setup for the night https://paste.openstack.org/show/b96mgBQpguw6KUseBAgZ/
18:34:26 melwitt awesome gibi

Earlier   Later