Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2026-07-08
15:41:19 opendevreview Takashi Kajinami proposed openstack/nova master: libvirt: Remove redundant version check for virtio-fs https://review.opendev.org/c/openstack/nova/+/996496
15:41:35 tkajinam ^^^ wondering if we should check the virt type instead
15:42:14 sean-k-mooney we woudl need to check for both qemu and kvm
15:42:15 tkajinam (this also makes me wonder if we should require kvm for sev support. it may not work for the other virt type really.
15:42:55 sean-k-mooney i.e virtio fs shoudl wor for qemu and kvm
15:43:10 sean-k-mooney sev maybe
15:43:25 tkajinam sean-k-mooney, yeah and I don't think it may work for lxc for example
15:43:35 sean-k-mooney i dont know if that works for qemu or just with kvm
15:44:19 sean-k-mooney well for lxc we are not actully using a vm so i dont think sev works for contianer in general
15:44:37 sean-k-mooney you can enabeld sev for all memoy at the host level but if you do that it used to break sriov
15:45:10 tkajinam sev definitely requires kvm because it relies on the implementations in kvm_amd module. I'm not too sure about virtiofs on the other hand
15:45:48 sean-k-mooney i dont think that requires kvm
15:46:05 sean-k-mooney but ya for sev you could add checkign teh vrit type to the supprot crtiria
15:46:10 sean-k-mooney if its not already there
15:46:20 tkajinam it's not, yet
15:46:40 tkajinam that's has never been added since sev support was first introduced.
15:47:02 tkajinam because that's an existing problem and may be trivial I'll look into that later, separately from sev-snp work
15:47:03 sean-k-mooney well if it does not work it would just fail to boot the vm
15:47:08 tkajinam yeah
15:47:26 sean-k-mooney so its not really a regression to not report sev supprot when virt type is qemu
15:47:57 sean-k-mooney tkajinam: if you can confim it i would just file a bug for it
15:48:48 sean-k-mooney https://review.opendev.org/c/openstack/nova/+/996496 looks ok ot me although im serpised we dont have any unit test coverage of the functions your removing
15:49:25 tkajinam sean-k-mooney, I'll file one by my end
15:49:43 tkajinam we might want to change that check to virtio-fs to account virt_type, instead of just removing it.
15:49:53 tkajinam (I'll record that as a separate bug, too
16:07:42 opendevreview Takashi Kajinami proposed openstack/nova master: Do not report SEV capability for non kvm virt_type https://review.opendev.org/c/openstack/nova/+/996498
16:16:43 gibi sean-k-mooney: I think futurist is just slow to shut down an exector https://bugs.launchpad.net/futurist/+bug/2160159
16:17:19 sean-k-mooney ah ok
16:17:50 sean-k-mooney i mean we could actully swap to concurrent.futures.ThreadPoolExecutor(max_workers=100) if needed
16:18:15 sean-k-mooney we have centralised the executor interaction in nova.utils
16:19:04 sean-k-mooney but cool that sound like a real bug
16:19:32 gibi yeah I will check if I can spot why
16:21:22 sean-k-mooney https://github.com/openstack/futurist/blob/master/futurist/_thread.py#L110-L147
16:22:24 sean-k-mooney so its polling ever second https://github.com/openstack/futurist/blob/master/futurist/_thread.py#L98-L108
16:22:29 sean-k-mooney when its waitign for the work to complete
16:23:35 sean-k-mooney but its also joining the work thread https://github.com/openstack/futurist/blob/master/futurist/_thread.py#L142
16:24:01 sean-k-mooney but in your lamda case that should not be costly
16:24:16 gibi is it due to https://github.com/openstack/futurist/blob/master/futurist/_thread.py#L103
16:24:19 gibi ?
16:24:27 gibi there is a clear 1 sec timeout passed there
16:25:00 sean-k-mooney that what i was wondering too
16:25:13 sean-k-mooney but that get is in blockign mode with a max of 1 second
16:25:26 sean-k-mooney if the queue has a value it wont wait for one second
16:26:06 sean-k-mooney i was wondering if you coudl jsut monkeypatch https://github.com/openstack/futurist/blob/master/futurist/_thread.py#L142 to 0.1
16:26:14 sean-k-mooney if that woudl have any effect on the timeings
16:26:19 sean-k-mooney but i dont think it should
16:26:50 gibi yeah I'm about to try it :)
16:27:35 sean-k-mooney jsut looking at the code im not seeing nayting obviouly dumb like a hardcoded sleep or similar
16:29:28 sean-k-mooney so your tesitn calling shudown but do we call stop before we do that. i assume shutdown sill do that interenally
16:29:59 sean-k-mooney oh well stop is on the threadworker and shutdown is on the executor
16:30:04 gibi it is the problem
16:30:15 gibi if I set it to 0.1 then the shutdown is 10 times faster
16:30:33 gibi added a comment to the bug
16:31:30 gibi I will propose a patch top of Ashish's functional series to monkey patch that value in the test
16:31:32 sean-k-mooney oh
16:31:39 sean-k-mooney i know what happenign i think
16:31:51 sean-k-mooney so each tradwroker is waiting up to a secodn but we have 10 workers
16:32:05 sean-k-mooney and i bet when we call shutdown it loops over all of them serially
16:32:23 sean-k-mooney i.e. the executor is calling stop on each threqad worker one at a time
16:32:46 sean-k-mooney gibi: i bet if you double the worker count you will see it doubel coorect?
16:33:07 gibi nope I tried that adding more worker does not change the timing
16:34:01 sean-k-mooney huh weird
16:34:05 sean-k-mooney https://docs.python.org/3/library/queue.html#queue.Queue.get
16:35:41 sean-k-mooney it really is the upper bound
16:36:30 gibi but the queue is empty in this case as there is no work left. So it does wait 1 sec before it raises Empty
16:37:02 gibi and the shutdown logic only checked in the empty exception handler
16:37:02 sean-k-mooney yes
16:37:59 gibi so we need better logic. We need to wake up the queue during shutdown with a poison added to the queue
16:38:30 gibi or ignore this and just monkey patch the 1 sec to a small number from our test :) In production that 1 sec is totally OK during shutdown
16:39:33 sean-k-mooney https://github.com/openstack/futurist/blob/6f870896819197ca323b3dd8af8f3b0389aff69c/futurist/_futures.py#L224-L240
16:40:23 sean-k-mooney gibi: so yes in the tests we can set ti to somethign smal for now
16:40:33 sean-k-mooney but that for over the woekers
16:40:53 sean-k-mooney where we call join is why i tought it would scale with teh worksers
16:41:45 gibi I think we wait that 1 sec per worker in parallel for each worker
16:41:50 opendevreview Merged openstack/nova master: Fix intermittent 409 in TestGracefulShutdown tests https://review.opendev.org/c/openstack/nova/+/996381
16:42:30 sean-k-mooney sortof each worker will do that
16:42:33 sean-k-mooney but https://github.com/openstack/futurist/blob/6f870896819197ca323b3dd8af8f3b0389aff69c/futurist/_futures.py#L237
16:42:45 sean-k-mooney when we call join there is seriailised
16:44:00 sean-k-mooney so we are taking the shutdwon lock, setting studwo to ture and waiting for the queu to drain,
16:44:01 gibi yes but join is not the one that triggers the wait for 1 sec. Join is just checks if the given thread is finished or not. At the first join we wait for 1 sec, then we see that the first thread finished, at the second join we see that the second thread already finished. etc
16:44:08 sean-k-mooney then notifying all workser to stop
16:44:13 sean-k-mooney and finally joining them all
16:44:46 gibi we join serially but the thread wait for their own 1 sec in parallel
16:44:54 gibi *threads
16:46:32 sean-k-mooney i think your correct
16:46:48 sean-k-mooney im just not sure why we see the wait
16:47:29 gibi anyhow I will dig deeper tomorrow. The last thing I do today is put a patch up to nova to nuke this 1 sec in the test
16:57:11 opendevreview Balazs Gibizer proposed openstack/nova master: HACK:Speed up functional test in threading mode https://review.opendev.org/c/openstack/nova/+/996500
16:57:12 sean-k-mooney gibi: i might quickly hack on somehting, if i push anything ill add you to it and you can free free to udpate it or push your own fix
16:57:20 gibi OK thanks
16:57:40 sean-k-mooney im basiclly goign to ask ai to reporduce your finding in a test case and then see if it can rootcause it
16:57:42 gibi https://review.opendev.org/c/openstack/nova/+/996500 for me locally re-gains all the lost time in the threading mode
16:57:50 gibi sean-k-mooney: OK cool
16:58:22 opendevreview Balazs Gibizer proposed openstack/nova master: HACK:Speed up functional test in threading mode https://review.opendev.org/c/openstack/nova/+/996500
16:58:31 sean-k-mooney also cool lets see how that ^ goes in ci over night
16:58:38 gibi yepp
16:58:42 gibi dropping now
16:58:44 gibi see you tomorrow
17:42:56 opendevreview Thibaut Démaret proposed openstack/nova master: libvirt: add disk rotation_rate support for local disks https://review.opendev.org/c/openstack/nova/+/979693
18:45:38 melwitt I have a patch that has been up for a while with one +2, to make QEMU_IMG_LIMITS configurable if anyone might be interested https://review.opendev.org/c/openstack/nova/+/969538 this came from an old upstream CI issue where the image limit was too low for an encrypted disk with ceph

Earlier   Later