Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2025-10-31
10:49:29 sean-k-mooney oh virtual file systems
10:49:34 sean-k-mooney not Virutal fucntions
10:49:42 sean-k-mooney i was like how does sriov come into this
10:50:12 gibi :)
10:50:23 gibi anyhow there are more work to do
10:50:46 gibi but I'm happy with this for now :)
10:51:18 sean-k-mooney have you considerd putting a dnm with an eventlet poison on calls to monkey patch
10:51:40 sean-k-mooney basiclly to prove that no one calls it in the job via a lib ectra
10:52:04 sean-k-mooney importing eventlet is kind of ok provide its not monkey patch but tha twoudl be a nice addtion even if we dont watn to ship that
10:52:34 gibi yeah I want to do that. I cannot do it right now as we still import it. But we are close to get rid of those imports.
10:52:48 sean-k-mooney well im not saying poison the import
10:53:01 sean-k-mooney im saying poison the eventlet.monkey_patch function
10:53:06 gibi ahh OK
10:53:15 gibi didn't we do that already?
10:53:33 opendevreview Rajesh Tailor proposed openstack/nova master: Update the api-ref for unshelve https://review.opendev.org/c/openstack/nova/+/938054
10:53:56 sean-k-mooney we coudl also https://opendev.org/openstack/nova/src/commit/7712e9193a2fc02454d73a43949a056b977076dd/nova/monkey_patch.py
10:54:09 gibi https://github.com/openstack/nova/blob/30bf8c1025ea24823ebde3eacede4ea34ec6be8c/nova/monkey_patch.py#L115-L131
10:54:12 sean-k-mooney so that you first patch
10:54:30 sean-k-mooney ya so your right
10:54:32 sean-k-mooney we did
10:54:38 sean-k-mooney so ok that good
10:54:56 sean-k-mooney so it possibe a lib is doing eventlet.spawn
10:55:28 sean-k-mooney we know that if you dont monkey patch and you call the restul object that cause the eventlet to run
10:55:42 sean-k-mooney blockign the caller until complete
10:55:45 gibi yeah
10:55:56 sean-k-mooney i wonder if we could also poison eventlet.spawn?
10:56:03 gibi so if the libvirt event handling works it works bec;se of that
10:56:04 sean-k-mooney honestly this is probaly overkill
10:56:36 sean-k-mooney lets see how ci fairs but this is great progress
10:57:36 sean-k-mooney if we do what dan suggested for spliting the venvs into one with eventlet and one without
10:57:48 sean-k-mooney we could slithgly update our code to do condtional imports
10:58:06 sean-k-mooney and when eventlet is not there use the threaded mode even if no env var is set
10:58:38 nicolairuckel Can anyone help me explaining the issue here? https://gitlab.com/libvirt/libvirt/-/issues/828 i was under the assumption that libvirt should take care of any permissions but either that's not the case or I'm not able to explain the problem well enough.
10:59:24 nicolairuckel maybe jkulik?
10:59:56 sean-k-mooney nicolairuckel: so honestly i need to trace the code for your orgianl patch and make sure we are not changing anything along the way
11:00:21 sean-k-mooney i parlly wonder if this is an edge case with libvirt dynmaic ownership feature
11:00:48 sean-k-mooney when you start a domain that is manage by libvirt libvier changes the ownwership of the root disk for example to be owned by qemu
11:00:54 sean-k-mooney and then reverts it back when the domain is stopped
11:01:05 sean-k-mooney i dont knwo if tis is doing something similar fo ht nvram file
11:01:24 sean-k-mooney the nvram file as far as i am aware shoudl nto be owned by nova
11:01:46 sean-k-mooney i shoudl be owned by either swtpm user or qemu
11:01:47 nicolairuckel You mean those could still be issues caused by the first version of my patch?
11:02:14 sean-k-mooney im unsure but that or soemthign else nova is doing that predates your patch
11:02:40 nicolairuckel I understand.
11:03:16 nicolairuckel The owner and group of the file are the same as for the disk (libvirt-qemu and kvm respectively) if that helps.
11:04:16 nicolairuckel So at least that looks correct to me.
11:13:10 sean-k-mooney if you dont change this how does it break
11:13:31 sean-k-mooney i.e. if you dont update form 600 to 644
11:13:52 sean-k-mooney i assume the swtpm cant acess it
11:14:04 sean-k-mooney what i think we need to do is test this on a clean env
11:14:29 sean-k-mooney ill see if i can set up a devstack today on a clean vm and look at the owner ship after the inial spawn in some detail
11:14:58 sean-k-mooney i think 640 woudl make sense assuming the swtpm user and qemu user and in the same group
11:17:08 jkulik but we got some information that we were looking for, right?
11:17:10 jkulik > When shutting down the VM libvirt also ought to return the permissions the file had before the VM was started.
11:17:43 jkulik so trying to figure out if we change the permissions somewhere, would be a good next step
11:20:40 sean-k-mooney jkulik: nicolairuckel i have been talkign to the libvirt maintainer interlly and they dont see a clear way that this could be a libvirt bug but also 600 should be ok
11:20:50 sean-k-mooney provide the libvirt dynmaic ownership feature is enabeld
11:20:57 sean-k-mooney nicolairuckel: do you have that disabeld in your env
11:21:41 sean-k-mooney jkulik: 600 shoudl be valid provided the file si chown'd to be owned by qemu before qemu starts
11:26:12 jkulik sean-k-mooney: I get that. I thought the point was, that it doesn't get re-chowned after the VM is shut down.
11:26:58 sean-k-mooney jkulik: we need to confirm the behvior and who is creatign the nvram file
11:27:05 sean-k-mooney and with what permisson
11:27:06 jkulik yes. I agree with that.
11:29:29 jkulik > If the NVRAM image is created by libvirt on first start of the VM it will have 600 permissions and will (originally) be owned by root; which will be restored after the VM is turned off.
11:29:45 jkulik ^ from the ticket. sounds plausible to me. when are we creating the NVRAM - do we rely on libvirt?
11:30:50 nicolairuckel I have a stacktrace of the error message in the libvirt issue, sean-k-mooney
11:31:45 nicolairuckel I need to look into the dynamic ownership since my colleague set up the Dev Stack.
11:33:15 sean-k-mooney nicolairuckel: lookign at the env its not supproted anymore
11:33:33 sean-k-mooney python 3.10 is
11:33:39 sean-k-mooney but your using ubuntu 22.04
11:33:49 sean-k-mooney which is not part of the testign runtimes
11:34:34 sean-k-mooney you are using libvirt 8.0.0 which is our current min version https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L220
11:35:07 sean-k-mooney but its possible that there is some odd interaction here happeing with the version
11:35:27 sean-k-mooney im goign to try and replciate this on ubuntu 24.04 and see what happens
11:36:25 sean-k-mooney we will or likely shoudl move to libvirt 10.0.0 as our min version this cycle
11:36:49 sean-k-mooney taht or current NEXT_MIN_LIBVIRT_VERSION
11:38:11 opendevreview OpenStack Release Bot proposed openstack/nova master: reno: Update master for unmaintained/2024.1 https://review.opendev.org/c/openstack/nova/+/965491
11:38:33 opendevreview OpenStack Release Bot proposed openstack/os-vif master: reno: Update master for unmaintained/2024.1 https://review.opendev.org/c/openstack/os-vif/+/965493
11:38:49 opendevreview OpenStack Release Bot proposed openstack/osc-placement master: reno: Update master for unmaintained/2024.1 https://review.opendev.org/c/openstack/osc-placement/+/965495
11:39:07 opendevreview OpenStack Release Bot proposed openstack/placement master: reno: Update master for unmaintained/2024.1 https://review.opendev.org/c/openstack/placement/+/965497
11:39:12 nicolairuckel sean-k-mooney, thank you!
11:39:13 sean-k-mooney that raises the minum to the minium we test with i.e. what ship in the disto with the oldest livbirt cover by the testing runtimes which siubuntu 24.04
11:39:25 opendevreview OpenStack Release Bot proposed openstack/python-novaclient master: reno: Update master for unmaintained/2024.1 https://review.opendev.org/c/openstack/python-novaclient/+/965500
11:45:18 opendevreview Masanori Kuroha proposed openstack/nova master: Copy applied provider config https://review.opendev.org/c/openstack/nova/+/948304
12:15:09 gibi bauzas: dansmith: can I get a second set of eyes on this to readd +2 https://review.opendev.org/c/openstack/nova/+/937275 The devstack dependency is merged
12:16:45 gibi this is also a pretty simple test only change around the same topic https://review.opendev.org/c/openstack/nova/+/963036
14:01:18 bauzas Uggla: fwiw the slot is not booked for nova
14:01:28 bauzas grizzly-FriB1 is empty
14:01:47 dansmith looks like it is to me
14:02:26 Uggla bauzas sounds good to me
14:02:50 bauzas oh doh, my maths
14:03:01 bauzas this is 2pm UTC now
14:03:07 Uggla I have not booked grizzly-FriB5 because I have the secret expectation we will manage to finish early.
14:03:45 bauzas Uggla: I'll need to skip the last hour then for us due to the tc meeting
14:04:18 Uggla ok
14:19:57 gibi sean-k-mooney: we have some tempest results with threaded compute. It is not good but not terrible https://review.opendev.org/c/openstack/nova/+/965467/1#message-518b76afc6b252d2f42c48d65f69f6570d9ecb8e
14:20:08 gibi e.g. we have passing complex tests
14:20:09 gibi 2025-10-31 13:07:09.576696 | controller | {2} tempest.scenario.test_instances_with_cinder_volumes.TestInstancesWithCinderVolumes.test_instances_with_cinder_volumes_on_all_compute_nodes [348.575935s] ... ok
14:30:34 nicolairuckel sean-k-mooney, does https://gitlab.com/libvirt/libvirt/-/issues/828#note_2858916854 mean it works with a more recent Ubuntu version?
14:33:09 sean-k-mooney nicolairuckel: i have not tried your patch yet

Earlier   Later