Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2025-10-31
10:46:01 gibi but it boots and deletes VMs for me locally
10:46:40 sean-k-mooney yep im reviweing th echange now i pushed qution to https://review.opendev.org/c/openstack/nova/+/965466
10:47:02 gibi I will dig a bit into the libvirt direct greenthread usage as that might only work just by luck
10:47:23 gibi but the tpools are neutered
10:47:32 gibi when running with native threading
10:47:47 sean-k-mooney i dont see wehre you did that by the way
10:47:54 sean-k-mooney oh
10:48:01 sean-k-mooney its in earlier pathces
10:48:08 gibi yepp earlier patches
10:48:15 sean-k-mooney https://review.opendev.org/c/openstack/nova/+/956089/12 and https://review.opendev.org/c/openstack/nova/+/956090/12
10:48:18 sean-k-mooney ya ok
10:48:22 gibi one for the vfs and one for the libvirt tpool
10:48:39 sean-k-mooney VFs?
10:49:01 gibi hm
10:49:03 sean-k-mooney do you mean rbd?
10:49:04 gibi then I missed one
10:49:07 gibi I did the rbd
10:49:17 gibi but there is on in nova/virt/disk/vfs/guestfs.py:from eventlet import tpool
10:49:29 sean-k-mooney oh virtual file systems
10:49:34 sean-k-mooney not Virutal fucntions
10:49:42 sean-k-mooney i was like how does sriov come into this
10:50:12 gibi :)
10:50:23 gibi anyhow there are more work to do
10:50:46 gibi but I'm happy with this for now :)
10:51:18 sean-k-mooney have you considerd putting a dnm with an eventlet poison on calls to monkey patch
10:51:40 sean-k-mooney basiclly to prove that no one calls it in the job via a lib ectra
10:52:04 sean-k-mooney importing eventlet is kind of ok provide its not monkey patch but tha twoudl be a nice addtion even if we dont watn to ship that
10:52:34 gibi yeah I want to do that. I cannot do it right now as we still import it. But we are close to get rid of those imports.
10:52:48 sean-k-mooney well im not saying poison the import
10:53:01 sean-k-mooney im saying poison the eventlet.monkey_patch function
10:53:06 gibi ahh OK
10:53:15 gibi didn't we do that already?
10:53:33 opendevreview Rajesh Tailor proposed openstack/nova master: Update the api-ref for unshelve https://review.opendev.org/c/openstack/nova/+/938054
10:53:56 sean-k-mooney we coudl also https://opendev.org/openstack/nova/src/commit/7712e9193a2fc02454d73a43949a056b977076dd/nova/monkey_patch.py
10:54:09 gibi https://github.com/openstack/nova/blob/30bf8c1025ea24823ebde3eacede4ea34ec6be8c/nova/monkey_patch.py#L115-L131
10:54:12 sean-k-mooney so that you first patch
10:54:30 sean-k-mooney ya so your right
10:54:32 sean-k-mooney we did
10:54:38 sean-k-mooney so ok that good
10:54:56 sean-k-mooney so it possibe a lib is doing eventlet.spawn
10:55:28 sean-k-mooney we know that if you dont monkey patch and you call the restul object that cause the eventlet to run
10:55:42 sean-k-mooney blockign the caller until complete
10:55:45 gibi yeah
10:55:56 sean-k-mooney i wonder if we could also poison eventlet.spawn?
10:56:03 gibi so if the libvirt event handling works it works bec;se of that
10:56:04 sean-k-mooney honestly this is probaly overkill
10:56:36 sean-k-mooney lets see how ci fairs but this is great progress
10:57:36 sean-k-mooney if we do what dan suggested for spliting the venvs into one with eventlet and one without
10:57:48 sean-k-mooney we could slithgly update our code to do condtional imports
10:58:06 sean-k-mooney and when eventlet is not there use the threaded mode even if no env var is set
10:58:38 nicolairuckel Can anyone help me explaining the issue here? https://gitlab.com/libvirt/libvirt/-/issues/828 i was under the assumption that libvirt should take care of any permissions but either that's not the case or I'm not able to explain the problem well enough.
10:59:24 nicolairuckel maybe jkulik?
10:59:56 sean-k-mooney nicolairuckel: so honestly i need to trace the code for your orgianl patch and make sure we are not changing anything along the way
11:00:21 sean-k-mooney i parlly wonder if this is an edge case with libvirt dynmaic ownership feature
11:00:48 sean-k-mooney when you start a domain that is manage by libvirt libvier changes the ownwership of the root disk for example to be owned by qemu
11:00:54 sean-k-mooney and then reverts it back when the domain is stopped
11:01:05 sean-k-mooney i dont knwo if tis is doing something similar fo ht nvram file
11:01:24 sean-k-mooney the nvram file as far as i am aware shoudl nto be owned by nova
11:01:46 sean-k-mooney i shoudl be owned by either swtpm user or qemu
11:01:47 nicolairuckel You mean those could still be issues caused by the first version of my patch?
11:02:14 sean-k-mooney im unsure but that or soemthign else nova is doing that predates your patch
11:02:40 nicolairuckel I understand.
11:03:16 nicolairuckel The owner and group of the file are the same as for the disk (libvirt-qemu and kvm respectively) if that helps.
11:04:16 nicolairuckel So at least that looks correct to me.
11:13:10 sean-k-mooney if you dont change this how does it break
11:13:31 sean-k-mooney i.e. if you dont update form 600 to 644
11:13:52 sean-k-mooney i assume the swtpm cant acess it
11:14:04 sean-k-mooney what i think we need to do is test this on a clean env
11:14:29 sean-k-mooney ill see if i can set up a devstack today on a clean vm and look at the owner ship after the inial spawn in some detail
11:14:58 sean-k-mooney i think 640 woudl make sense assuming the swtpm user and qemu user and in the same group
11:17:08 jkulik but we got some information that we were looking for, right?
11:17:10 jkulik > When shutting down the VM libvirt also ought to return the permissions the file had before the VM was started.
11:17:43 jkulik so trying to figure out if we change the permissions somewhere, would be a good next step
11:20:40 sean-k-mooney jkulik: nicolairuckel i have been talkign to the libvirt maintainer interlly and they dont see a clear way that this could be a libvirt bug but also 600 should be ok
11:20:50 sean-k-mooney provide the libvirt dynmaic ownership feature is enabeld
11:20:57 sean-k-mooney nicolairuckel: do you have that disabeld in your env
11:21:41 sean-k-mooney jkulik: 600 shoudl be valid provided the file si chown'd to be owned by qemu before qemu starts
11:26:12 jkulik sean-k-mooney: I get that. I thought the point was, that it doesn't get re-chowned after the VM is shut down.
11:26:58 sean-k-mooney jkulik: we need to confirm the behvior and who is creatign the nvram file
11:27:05 sean-k-mooney and with what permisson
11:27:06 jkulik yes. I agree with that.
11:29:29 jkulik > If the NVRAM image is created by libvirt on first start of the VM it will have 600 permissions and will (originally) be owned by root; which will be restored after the VM is turned off.
11:29:45 jkulik ^ from the ticket. sounds plausible to me. when are we creating the NVRAM - do we rely on libvirt?
11:30:50 nicolairuckel I have a stacktrace of the error message in the libvirt issue, sean-k-mooney
11:31:45 nicolairuckel I need to look into the dynamic ownership since my colleague set up the Dev Stack.
11:33:15 sean-k-mooney nicolairuckel: lookign at the env its not supproted anymore
11:33:33 sean-k-mooney python 3.10 is
11:33:39 sean-k-mooney but your using ubuntu 22.04
11:33:49 sean-k-mooney which is not part of the testign runtimes
11:34:34 sean-k-mooney you are using libvirt 8.0.0 which is our current min version https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L220
11:35:07 sean-k-mooney but its possible that there is some odd interaction here happeing with the version
11:35:27 sean-k-mooney im goign to try and replciate this on ubuntu 24.04 and see what happens
11:36:25 sean-k-mooney we will or likely shoudl move to libvirt 10.0.0 as our min version this cycle
11:36:49 sean-k-mooney taht or current NEXT_MIN_LIBVIRT_VERSION
11:38:11 opendevreview OpenStack Release Bot proposed openstack/nova master: reno: Update master for unmaintained/2024.1 https://review.opendev.org/c/openstack/nova/+/965491
11:38:33 opendevreview OpenStack Release Bot proposed openstack/os-vif master: reno: Update master for unmaintained/2024.1 https://review.opendev.org/c/openstack/os-vif/+/965493
11:38:49 opendevreview OpenStack Release Bot proposed openstack/osc-placement master: reno: Update master for unmaintained/2024.1 https://review.opendev.org/c/openstack/osc-placement/+/965495
11:39:07 opendevreview OpenStack Release Bot proposed openstack/placement master: reno: Update master for unmaintained/2024.1 https://review.opendev.org/c/openstack/placement/+/965497
11:39:12 nicolairuckel sean-k-mooney, thank you!
11:39:13 sean-k-mooney that raises the minum to the minium we test with i.e. what ship in the disto with the oldest livbirt cover by the testing runtimes which siubuntu 24.04

Earlier   Later