Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2026-07-08
15:46:40 tkajinam that's has never been added since sev support was first introduced.
15:47:02 tkajinam because that's an existing problem and may be trivial I'll look into that later, separately from sev-snp work
15:47:03 sean-k-mooney well if it does not work it would just fail to boot the vm
15:47:08 tkajinam yeah
15:47:26 sean-k-mooney so its not really a regression to not report sev supprot when virt type is qemu
15:47:57 sean-k-mooney tkajinam: if you can confim it i would just file a bug for it
15:48:48 sean-k-mooney https://review.opendev.org/c/openstack/nova/+/996496 looks ok ot me although im serpised we dont have any unit test coverage of the functions your removing
15:49:25 tkajinam sean-k-mooney, I'll file one by my end
15:49:43 tkajinam we might want to change that check to virtio-fs to account virt_type, instead of just removing it.
15:49:53 tkajinam (I'll record that as a separate bug, too
16:07:42 opendevreview Takashi Kajinami proposed openstack/nova master: Do not report SEV capability for non kvm virt_type https://review.opendev.org/c/openstack/nova/+/996498
16:16:43 gibi sean-k-mooney: I think futurist is just slow to shut down an exector https://bugs.launchpad.net/futurist/+bug/2160159
16:17:19 sean-k-mooney ah ok
16:17:50 sean-k-mooney i mean we could actully swap to concurrent.futures.ThreadPoolExecutor(max_workers=100) if needed
16:18:15 sean-k-mooney we have centralised the executor interaction in nova.utils
16:19:04 sean-k-mooney but cool that sound like a real bug
16:19:32 gibi yeah I will check if I can spot why
16:21:22 sean-k-mooney https://github.com/openstack/futurist/blob/master/futurist/_thread.py#L110-L147
16:22:24 sean-k-mooney so its polling ever second https://github.com/openstack/futurist/blob/master/futurist/_thread.py#L98-L108
16:22:29 sean-k-mooney when its waitign for the work to complete
16:23:35 sean-k-mooney but its also joining the work thread https://github.com/openstack/futurist/blob/master/futurist/_thread.py#L142
16:24:01 sean-k-mooney but in your lamda case that should not be costly
16:24:16 gibi is it due to https://github.com/openstack/futurist/blob/master/futurist/_thread.py#L103
16:24:19 gibi ?
16:24:27 gibi there is a clear 1 sec timeout passed there
16:25:00 sean-k-mooney that what i was wondering too
16:25:13 sean-k-mooney but that get is in blockign mode with a max of 1 second
16:25:26 sean-k-mooney if the queue has a value it wont wait for one second
16:26:06 sean-k-mooney i was wondering if you coudl jsut monkeypatch https://github.com/openstack/futurist/blob/master/futurist/_thread.py#L142 to 0.1
16:26:14 sean-k-mooney if that woudl have any effect on the timeings
16:26:19 sean-k-mooney but i dont think it should
16:26:50 gibi yeah I'm about to try it :)
16:27:35 sean-k-mooney jsut looking at the code im not seeing nayting obviouly dumb like a hardcoded sleep or similar
16:29:28 sean-k-mooney so your tesitn calling shudown but do we call stop before we do that. i assume shutdown sill do that interenally
16:29:59 sean-k-mooney oh well stop is on the threadworker and shutdown is on the executor
16:30:04 gibi it is the problem
16:30:15 gibi if I set it to 0.1 then the shutdown is 10 times faster
16:30:33 gibi added a comment to the bug
16:31:30 gibi I will propose a patch top of Ashish's functional series to monkey patch that value in the test
16:31:32 sean-k-mooney oh
16:31:39 sean-k-mooney i know what happenign i think
16:31:51 sean-k-mooney so each tradwroker is waiting up to a secodn but we have 10 workers
16:32:05 sean-k-mooney and i bet when we call shutdown it loops over all of them serially
16:32:23 sean-k-mooney i.e. the executor is calling stop on each threqad worker one at a time
16:32:46 sean-k-mooney gibi: i bet if you double the worker count you will see it doubel coorect?
16:33:07 gibi nope I tried that adding more worker does not change the timing
16:34:01 sean-k-mooney huh weird
16:34:05 sean-k-mooney https://docs.python.org/3/library/queue.html#queue.Queue.get
16:35:41 sean-k-mooney it really is the upper bound
16:36:30 gibi but the queue is empty in this case as there is no work left. So it does wait 1 sec before it raises Empty
16:37:02 gibi and the shutdown logic only checked in the empty exception handler
16:37:02 sean-k-mooney yes
16:37:59 gibi so we need better logic. We need to wake up the queue during shutdown with a poison added to the queue
16:38:30 gibi or ignore this and just monkey patch the 1 sec to a small number from our test :) In production that 1 sec is totally OK during shutdown
16:39:33 sean-k-mooney https://github.com/openstack/futurist/blob/6f870896819197ca323b3dd8af8f3b0389aff69c/futurist/_futures.py#L224-L240
16:40:23 sean-k-mooney gibi: so yes in the tests we can set ti to somethign smal for now
16:40:33 sean-k-mooney but that for over the woekers
16:40:53 sean-k-mooney where we call join is why i tought it would scale with teh worksers
16:41:45 gibi I think we wait that 1 sec per worker in parallel for each worker
16:41:50 opendevreview Merged openstack/nova master: Fix intermittent 409 in TestGracefulShutdown tests https://review.opendev.org/c/openstack/nova/+/996381
16:42:30 sean-k-mooney sortof each worker will do that
16:42:33 sean-k-mooney but https://github.com/openstack/futurist/blob/6f870896819197ca323b3dd8af8f3b0389aff69c/futurist/_futures.py#L237
16:42:45 sean-k-mooney when we call join there is seriailised
16:44:00 sean-k-mooney so we are taking the shutdwon lock, setting studwo to ture and waiting for the queu to drain,
16:44:01 gibi yes but join is not the one that triggers the wait for 1 sec. Join is just checks if the given thread is finished or not. At the first join we wait for 1 sec, then we see that the first thread finished, at the second join we see that the second thread already finished. etc
16:44:08 sean-k-mooney then notifying all workser to stop
16:44:13 sean-k-mooney and finally joining them all
16:44:46 gibi we join serially but the thread wait for their own 1 sec in parallel
16:44:54 gibi *threads
16:46:32 sean-k-mooney i think your correct
16:46:48 sean-k-mooney im just not sure why we see the wait
16:47:29 gibi anyhow I will dig deeper tomorrow. The last thing I do today is put a patch up to nova to nuke this 1 sec in the test
16:57:11 opendevreview Balazs Gibizer proposed openstack/nova master: HACK:Speed up functional test in threading mode https://review.opendev.org/c/openstack/nova/+/996500
16:57:12 sean-k-mooney gibi: i might quickly hack on somehting, if i push anything ill add you to it and you can free free to udpate it or push your own fix
16:57:20 gibi OK thanks
16:57:40 sean-k-mooney im basiclly goign to ask ai to reporduce your finding in a test case and then see if it can rootcause it
16:57:42 gibi https://review.opendev.org/c/openstack/nova/+/996500 for me locally re-gains all the lost time in the threading mode
16:57:50 gibi sean-k-mooney: OK cool
16:58:22 opendevreview Balazs Gibizer proposed openstack/nova master: HACK:Speed up functional test in threading mode https://review.opendev.org/c/openstack/nova/+/996500
16:58:31 sean-k-mooney also cool lets see how that ^ goes in ci over night
16:58:38 gibi yepp
16:58:42 gibi dropping now
16:58:44 gibi see you tomorrow
17:42:56 opendevreview Thibaut Démaret proposed openstack/nova master: libvirt: add disk rotation_rate support for local disks https://review.opendev.org/c/openstack/nova/+/979693
18:45:38 melwitt I have a patch that has been up for a while with one +2, to make QEMU_IMG_LIMITS configurable if anyone might be interested https://review.opendev.org/c/openstack/nova/+/969538 this came from an old upstream CI issue where the image limit was too low for an encrypted disk with ceph
20:22:24 sean-k-mooney melwitt: so while that helps
20:22:39 sean-k-mooney we proably shoudl fix the memory limit if we are doign this
20:22:56 sean-k-mooney melwitt: currently we are limiting on adress space but we shoudl be limiting on RSS
20:23:39 melwitt hm ok .. I am not yet familiar with that
20:24:06 sean-k-mooney address space is how much well adrsss space that is mapped into the process with or without actual memory usage
20:24:15 sean-k-mooney RSS is residnet set size
20:24:24 sean-k-mooney i.e. how much actual memory is being used
20:25:05 sean-k-mooney in ci with some config we were seing newver version fo ceph/rbd map alot of memory without actully usign a lot of memroy
20:25:40 melwitt oh I see
20:25:56 sean-k-mooney https://github.com/openstack/oslo.concurrency/blob/master/oslo_concurrency/processutils.py#L162
20:26:19 melwitt a-ha
20:27:15 melwitt so you think if we just s/address_space/resident_set_size/
20:27:43 melwitt looks like rss has been a choice as long as address space ... so it is not a newly available choice. yet it has not been used
20:27:51 melwitt I wonder if that was deliberate
20:28:26 sean-k-mooney i would just sed it yes

Earlier   Later