Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2025-10-31
10:45:33 gibi I made the Tpools running the calls on the caller thread, and basically swiched all our executors to ThreadPools (already in tree) with the env variable. I needed to convince oslo.service not to fork a worker process for compute. And that is it. Compute just started working :)
10:45:54 gibi probably fails in edge cases
10:46:01 gibi but it boots and deletes VMs for me locally
10:46:40 sean-k-mooney yep im reviweing th echange now i pushed qution to https://review.opendev.org/c/openstack/nova/+/965466
10:47:02 gibi I will dig a bit into the libvirt direct greenthread usage as that might only work just by luck
10:47:23 gibi but the tpools are neutered
10:47:32 gibi when running with native threading
10:47:47 sean-k-mooney i dont see wehre you did that by the way
10:47:54 sean-k-mooney oh
10:48:01 sean-k-mooney its in earlier pathces
10:48:08 gibi yepp earlier patches
10:48:15 sean-k-mooney https://review.opendev.org/c/openstack/nova/+/956089/12 and https://review.opendev.org/c/openstack/nova/+/956090/12
10:48:18 sean-k-mooney ya ok
10:48:22 gibi one for the vfs and one for the libvirt tpool
10:48:39 sean-k-mooney VFs?
10:49:01 gibi hm
10:49:03 sean-k-mooney do you mean rbd?
10:49:04 gibi then I missed one
10:49:07 gibi I did the rbd
10:49:17 gibi but there is on in nova/virt/disk/vfs/guestfs.py:from eventlet import tpool
10:49:29 sean-k-mooney oh virtual file systems
10:49:34 sean-k-mooney not Virutal fucntions
10:49:42 sean-k-mooney i was like how does sriov come into this
10:50:12 gibi :)
10:50:23 gibi anyhow there are more work to do
10:50:46 gibi but I'm happy with this for now :)
10:51:18 sean-k-mooney have you considerd putting a dnm with an eventlet poison on calls to monkey patch
10:51:40 sean-k-mooney basiclly to prove that no one calls it in the job via a lib ectra
10:52:04 sean-k-mooney importing eventlet is kind of ok provide its not monkey patch but tha twoudl be a nice addtion even if we dont watn to ship that
10:52:34 gibi yeah I want to do that. I cannot do it right now as we still import it. But we are close to get rid of those imports.
10:52:48 sean-k-mooney well im not saying poison the import
10:53:01 sean-k-mooney im saying poison the eventlet.monkey_patch function
10:53:06 gibi ahh OK
10:53:15 gibi didn't we do that already?
10:53:33 opendevreview Rajesh Tailor proposed openstack/nova master: Update the api-ref for unshelve https://review.opendev.org/c/openstack/nova/+/938054
10:53:56 sean-k-mooney we coudl also https://opendev.org/openstack/nova/src/commit/7712e9193a2fc02454d73a43949a056b977076dd/nova/monkey_patch.py
10:54:09 gibi https://github.com/openstack/nova/blob/30bf8c1025ea24823ebde3eacede4ea34ec6be8c/nova/monkey_patch.py#L115-L131
10:54:12 sean-k-mooney so that you first patch
10:54:30 sean-k-mooney ya so your right
10:54:32 sean-k-mooney we did
10:54:38 sean-k-mooney so ok that good
10:54:56 sean-k-mooney so it possibe a lib is doing eventlet.spawn
10:55:28 sean-k-mooney we know that if you dont monkey patch and you call the restul object that cause the eventlet to run
10:55:42 sean-k-mooney blockign the caller until complete
10:55:45 gibi yeah
10:55:56 sean-k-mooney i wonder if we could also poison eventlet.spawn?
10:56:03 gibi so if the libvirt event handling works it works bec;se of that
10:56:04 sean-k-mooney honestly this is probaly overkill
10:56:36 sean-k-mooney lets see how ci fairs but this is great progress
10:57:36 sean-k-mooney if we do what dan suggested for spliting the venvs into one with eventlet and one without
10:57:48 sean-k-mooney we could slithgly update our code to do condtional imports
10:58:06 sean-k-mooney and when eventlet is not there use the threaded mode even if no env var is set
10:58:38 nicolairuckel Can anyone help me explaining the issue here? https://gitlab.com/libvirt/libvirt/-/issues/828 i was under the assumption that libvirt should take care of any permissions but either that's not the case or I'm not able to explain the problem well enough.
10:59:24 nicolairuckel maybe jkulik?
10:59:56 sean-k-mooney nicolairuckel: so honestly i need to trace the code for your orgianl patch and make sure we are not changing anything along the way
11:00:21 sean-k-mooney i parlly wonder if this is an edge case with libvirt dynmaic ownership feature
11:00:48 sean-k-mooney when you start a domain that is manage by libvirt libvier changes the ownwership of the root disk for example to be owned by qemu
11:00:54 sean-k-mooney and then reverts it back when the domain is stopped
11:01:05 sean-k-mooney i dont knwo if tis is doing something similar fo ht nvram file
11:01:24 sean-k-mooney the nvram file as far as i am aware shoudl nto be owned by nova
11:01:46 sean-k-mooney i shoudl be owned by either swtpm user or qemu
11:01:47 nicolairuckel You mean those could still be issues caused by the first version of my patch?
11:02:14 sean-k-mooney im unsure but that or soemthign else nova is doing that predates your patch
11:02:40 nicolairuckel I understand.
11:03:16 nicolairuckel The owner and group of the file are the same as for the disk (libvirt-qemu and kvm respectively) if that helps.
11:04:16 nicolairuckel So at least that looks correct to me.
11:13:10 sean-k-mooney if you dont change this how does it break
11:13:31 sean-k-mooney i.e. if you dont update form 600 to 644
11:13:52 sean-k-mooney i assume the swtpm cant acess it
11:14:04 sean-k-mooney what i think we need to do is test this on a clean env
11:14:29 sean-k-mooney ill see if i can set up a devstack today on a clean vm and look at the owner ship after the inial spawn in some detail
11:14:58 sean-k-mooney i think 640 woudl make sense assuming the swtpm user and qemu user and in the same group
11:17:08 jkulik but we got some information that we were looking for, right?
11:17:10 jkulik > When shutting down the VM libvirt also ought to return the permissions the file had before the VM was started.
11:17:43 jkulik so trying to figure out if we change the permissions somewhere, would be a good next step
11:20:40 sean-k-mooney jkulik: nicolairuckel i have been talkign to the libvirt maintainer interlly and they dont see a clear way that this could be a libvirt bug but also 600 should be ok
11:20:50 sean-k-mooney provide the libvirt dynmaic ownership feature is enabeld
11:20:57 sean-k-mooney nicolairuckel: do you have that disabeld in your env
11:21:41 sean-k-mooney jkulik: 600 shoudl be valid provided the file si chown'd to be owned by qemu before qemu starts
11:26:12 jkulik sean-k-mooney: I get that. I thought the point was, that it doesn't get re-chowned after the VM is shut down.
11:26:58 sean-k-mooney jkulik: we need to confirm the behvior and who is creatign the nvram file
11:27:05 sean-k-mooney and with what permisson
11:27:06 jkulik yes. I agree with that.
11:29:29 jkulik > If the NVRAM image is created by libvirt on first start of the VM it will have 600 permissions and will (originally) be owned by root; which will be restored after the VM is turned off.
11:29:45 jkulik ^ from the ticket. sounds plausible to me. when are we creating the NVRAM - do we rely on libvirt?
11:30:50 nicolairuckel I have a stacktrace of the error message in the libvirt issue, sean-k-mooney
11:31:45 nicolairuckel I need to look into the dynamic ownership since my colleague set up the Dev Stack.
11:33:15 sean-k-mooney nicolairuckel: lookign at the env its not supproted anymore
11:33:33 sean-k-mooney python 3.10 is
11:33:39 sean-k-mooney but your using ubuntu 22.04
11:33:49 sean-k-mooney which is not part of the testign runtimes
11:34:34 sean-k-mooney you are using libvirt 8.0.0 which is our current min version https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L220
11:35:07 sean-k-mooney but its possible that there is some odd interaction here happeing with the version
11:35:27 sean-k-mooney im goign to try and replciate this on ubuntu 24.04 and see what happens
11:36:25 sean-k-mooney we will or likely shoudl move to libvirt 10.0.0 as our min version this cycle
11:36:49 sean-k-mooney taht or current NEXT_MIN_LIBVIRT_VERSION
11:38:11 opendevreview OpenStack Release Bot proposed openstack/nova master: reno: Update master for unmaintained/2024.1 https://review.opendev.org/c/openstack/nova/+/965491
11:38:33 opendevreview OpenStack Release Bot proposed openstack/os-vif master: reno: Update master for unmaintained/2024.1 https://review.opendev.org/c/openstack/os-vif/+/965493
11:38:49 opendevreview OpenStack Release Bot proposed openstack/osc-placement master: reno: Update master for unmaintained/2024.1 https://review.opendev.org/c/openstack/osc-placement/+/965495
11:39:07 opendevreview OpenStack Release Bot proposed openstack/placement master: reno: Update master for unmaintained/2024.1 https://review.opendev.org/c/openstack/placement/+/965497

Earlier   Later