| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-07-08 | |||
| 16:34:05 | sean-k-mooney | https://docs.python.org/3/library/queue.html#queue.Queue.get | |
| 16:35:41 | sean-k-mooney | it really is the upper bound | |
| 16:36:30 | gibi | but the queue is empty in this case as there is no work left. So it does wait 1 sec before it raises Empty | |
| 16:37:02 | gibi | and the shutdown logic only checked in the empty exception handler | |
| 16:37:02 | sean-k-mooney | yes | |
| 16:37:59 | gibi | so we need better logic. We need to wake up the queue during shutdown with a poison added to the queue | |
| 16:38:30 | gibi | or ignore this and just monkey patch the 1 sec to a small number from our test :) In production that 1 sec is totally OK during shutdown | |
| 16:39:33 | sean-k-mooney | https://github.com/openstack/futurist/blob/6f870896819197ca323b3dd8af8f3b0389aff69c/futurist/_futures.py#L224-L240 | |
| 16:40:23 | sean-k-mooney | gibi: so yes in the tests we can set ti to somethign smal for now | |
| 16:40:33 | sean-k-mooney | but that for over the woekers | |
| 16:40:53 | sean-k-mooney | where we call join is why i tought it would scale with teh worksers | |
| 16:41:45 | gibi | I think we wait that 1 sec per worker in parallel for each worker | |
| 16:41:50 | opendevreview | Merged openstack/nova master: Fix intermittent 409 in TestGracefulShutdown tests https://review.opendev.org/c/openstack/nova/+/996381 | |
| 16:42:30 | sean-k-mooney | sortof each worker will do that | |
| 16:42:33 | sean-k-mooney | but https://github.com/openstack/futurist/blob/6f870896819197ca323b3dd8af8f3b0389aff69c/futurist/_futures.py#L237 | |
| 16:42:45 | sean-k-mooney | when we call join there is seriailised | |
| 16:44:00 | sean-k-mooney | so we are taking the shutdwon lock, setting studwo to ture and waiting for the queu to drain, | |
| 16:44:01 | gibi | yes but join is not the one that triggers the wait for 1 sec. Join is just checks if the given thread is finished or not. At the first join we wait for 1 sec, then we see that the first thread finished, at the second join we see that the second thread already finished. etc | |
| 16:44:08 | sean-k-mooney | then notifying all workser to stop | |
| 16:44:13 | sean-k-mooney | and finally joining them all | |
| 16:44:46 | gibi | we join serially but the thread wait for their own 1 sec in parallel | |
| 16:44:54 | gibi | *threads | |
| 16:46:32 | sean-k-mooney | i think your correct | |
| 16:46:48 | sean-k-mooney | im just not sure why we see the wait | |
| 16:47:29 | gibi | anyhow I will dig deeper tomorrow. The last thing I do today is put a patch up to nova to nuke this 1 sec in the test | |
| 16:57:11 | opendevreview | Balazs Gibizer proposed openstack/nova master: HACK:Speed up functional test in threading mode https://review.opendev.org/c/openstack/nova/+/996500 | |
| 16:57:12 | sean-k-mooney | gibi: i might quickly hack on somehting, if i push anything ill add you to it and you can free free to udpate it or push your own fix | |
| 16:57:20 | gibi | OK thanks | |
| 16:57:40 | sean-k-mooney | im basiclly goign to ask ai to reporduce your finding in a test case and then see if it can rootcause it | |
| 16:57:42 | gibi | https://review.opendev.org/c/openstack/nova/+/996500 for me locally re-gains all the lost time in the threading mode | |
| 16:57:50 | gibi | sean-k-mooney: OK cool | |
| 16:58:22 | opendevreview | Balazs Gibizer proposed openstack/nova master: HACK:Speed up functional test in threading mode https://review.opendev.org/c/openstack/nova/+/996500 | |
| 16:58:31 | sean-k-mooney | also cool lets see how that ^ goes in ci over night | |
| 16:58:38 | gibi | yepp | |
| 16:58:42 | gibi | dropping now | |
| 16:58:44 | gibi | see you tomorrow | |
| 17:42:56 | opendevreview | Thibaut Démaret proposed openstack/nova master: libvirt: add disk rotation_rate support for local disks https://review.opendev.org/c/openstack/nova/+/979693 | |
| 18:45:38 | melwitt | I have a patch that has been up for a while with one +2, to make QEMU_IMG_LIMITS configurable if anyone might be interested https://review.opendev.org/c/openstack/nova/+/969538 this came from an old upstream CI issue where the image limit was too low for an encrypted disk with ceph | |
| 20:22:24 | sean-k-mooney | melwitt: so while that helps | |
| 20:22:39 | sean-k-mooney | we proably shoudl fix the memory limit if we are doign this | |
| 20:22:56 | sean-k-mooney | melwitt: currently we are limiting on adress space but we shoudl be limiting on RSS | |
| 20:23:39 | melwitt | hm ok .. I am not yet familiar with that | |
| 20:24:06 | sean-k-mooney | address space is how much well adrsss space that is mapped into the process with or without actual memory usage | |
| 20:24:15 | sean-k-mooney | RSS is residnet set size | |
| 20:24:24 | sean-k-mooney | i.e. how much actual memory is being used | |
| 20:25:05 | sean-k-mooney | in ci with some config we were seing newver version fo ceph/rbd map alot of memory without actully usign a lot of memroy | |
| 20:25:40 | melwitt | oh I see | |
| 20:25:56 | sean-k-mooney | https://github.com/openstack/oslo.concurrency/blob/master/oslo_concurrency/processutils.py#L162 | |
| 20:26:19 | melwitt | a-ha | |
| 20:27:15 | melwitt | so you think if we just s/address_space/resident_set_size/ | |
| 20:27:43 | melwitt | looks like rss has been a choice as long as address space ... so it is not a newly available choice. yet it has not been used | |
| 20:27:51 | melwitt | I wonder if that was deliberate | |
| 20:28:26 | sean-k-mooney | i would just sed it yes | |
| 20:29:54 | melwitt | looks like no one is using it, huh. https://codesearch.openstack.org/?q=resident_set_size&i=nope&literal=nope&files=&excludeFiles=&repos= | |
| 20:31:25 | sean-k-mooney | so i moved the ceph jobs to debian to work around it https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/955714 and proposed a reviert of the tempest skip https://review.opendev.org/c/openstack/tempest/+/955177 | |
| 20:31:54 | sean-k-mooney | but we never actully merge the tempest skip reviert | |
| 20:32:11 | melwitt | yeah, I remembered that. I had wondered what the "real fix" should be | |
| 20:33:48 | sean-k-mooney | i +1 your patch becasue we coudl proceed with it, i think movign to rss instead fo adresss space woudl make sense btu let see if gibi has or somoen else has an opion | |
| 20:34:20 | sean-k-mooney | melwitt: im not agaisnt makign it configurabl i just dont think that the correct limit to use in this case in general | |
| 20:34:31 | sean-k-mooney | but we coudl change that in a followup | |
| 20:35:14 | melwitt | sure. I'm also happy to just change it to resident_set_size if there is no potential bad thing about that haha | |
| 20:35:31 | melwitt | I am a bit wondering why no one has made use of it before | |
| 20:36:36 | sean-k-mooney | well almost nothign uses this in general https://codesearch.openstack.org/?q=ProcessLimits&i=nope&literal=nope&files=&excludeFiles=&repos= | |
| 20:36:45 | sean-k-mooney | and i think the code was cargo culted form nova to the rest | |
| 20:37:31 | sean-k-mooney | tweaking the adress space limit will allow more meory to be used like rss would | |
| 20:37:44 | melwitt | nova trend setter | |
| 20:37:47 | sean-k-mooney | but you may have to set it higher then otherwise | |
| 20:38:26 | melwitt | yeah. I mostly saw how cinder and ironic both doing it | |
| 20:38:44 | sean-k-mooney | you know what ill appove your current patch but lets chat about this more | |
| 20:38:53 | sean-k-mooney | your currrent patch matches cinder and ironics approch | |
| 20:39:06 | sean-k-mooney | so if it works for them it will proably be fine | |
| 20:39:11 | melwitt | it's ok, I don't mind chatting for rss | |
| 20:39:32 | sean-k-mooney | ok but lets check back tomorrow so we dont forget :) | |
| 20:40:12 | melwitt | I feel like dan would have a good input for this but I think he's away until next week | |
| 20:40:20 | sean-k-mooney | gmaan: related to ^ i rebased https://review.opendev.org/c/openstack/tempest/+/955177 to get new results | |
| 20:41:06 | sean-k-mooney | ok im goign to go eat dinner o/ | |
| 20:41:16 | melwitt | ok seeya o/ | |
| 20:48:49 | gmaan | sean-k-mooney: ack | |
| 21:38:07 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Task tracking mechanism for graceful shutdown https://review.opendev.org/c/openstack/nova/+/996299 | |
| 21:57:03 | opendevreview | Merged openstack/nova master: Rename cyborg-tempest to cyborg-tempest-py3 https://review.opendev.org/c/openstack/nova/+/996297 | |
| 23:18:48 | opendevreview | Ashish Gupta proposed openstack/nova master: tests: file-backed SQLite with WAL in threading mode for Database and CellDatabases Fixtures https://review.opendev.org/c/openstack/nova/+/988583 | |
| 23:24:43 | opendevreview | Ashish Gupta proposed openstack/nova master: tests: use file-backed Placement SQLite in functional threading https://review.opendev.org/c/openstack/nova/+/992581 | |
| 23:27:11 | opendevreview | Ashish Gupta proposed openstack/nova master: tests: Use per-database write locks instead of global lock https://review.opendev.org/c/openstack/nova/+/992862 | |
| #openstack-nova - 2026-07-09 | |||
| 00:11:18 | opendevreview | Ashish Gupta proposed openstack/nova master: tests: file-backed SQLite with WAL in threading mode for Database and CellDatabases Fixtures https://review.opendev.org/c/openstack/nova/+/988583 | |
| 00:13:14 | opendevreview | Ashish Gupta proposed openstack/nova master: tests: use file-backed Placement SQLite in functional threading https://review.opendev.org/c/openstack/nova/+/992581 | |
| 00:15:14 | opendevreview | Ashish Gupta proposed openstack/nova master: tests: Use per-database write locks instead of global lock https://review.opendev.org/c/openstack/nova/+/992862 | |
| 00:27:58 | opendevreview | Takashi Kajinami proposed openstack/nova master: Use domain capabilities to detect SEV-SNP support https://review.opendev.org/c/openstack/nova/+/996316 | |
| 00:52:40 | sean-k-mooney | gibi: https://review.opendev.org/c/openstack/futurist/+/996567 and the followups may be of interst. | |
| 02:24:28 | opendevreview | Takashi Kajinami proposed openstack/nova master: libvirt: Remove redundant version check for virtio-fs https://review.opendev.org/c/openstack/nova/+/996496 | |
| 02:41:05 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Call manager graceful_shutdown while RPC server waits https://review.opendev.org/c/openstack/nova/+/996583 | |
| 07:15:43 | frickler | anyone around for stable reviews of the OSSN-0101 fix? https://review.opendev.org/c/openstack/nova/+/995956 etc. | |
| 07:25:33 | opendevreview | Bhavana proposed openstack/nova master: Add Minimal Displacement Solver for MIG GPU Repartitioning https://review.opendev.org/c/openstack/nova/+/996058 | |
| 07:54:35 | gibi | frickler: +2 | |
| 07:54:47 | gibi | elodilles: could you look at these backports? https://review.opendev.org/c/openstack/nova/+/995956 | |
| 07:57:31 | opendevreview | Balazs Gibizer proposed openstack/nova master: HACK:Speed up functional test in threading mode https://review.opendev.org/c/openstack/nova/+/996500 | |
| 08:14:26 | opendevreview | Thibaut Démaret proposed openstack/nova master: libvirt: add disk rotation_rate support for local disks https://review.opendev.org/c/openstack/nova/+/979693 | |
| 08:39:04 | elodilles | gibi: +2'd it. will follow up with the backports to older branches when this merges | |
| 09:30:16 | gibi | elodilles: thanks | |
| 10:43:54 | ralonsoh | hi folks, a quick review for https://review.opendev.org/c/openstack/os-vif/+/995933 will be appreciated | |
| 10:43:55 | ralonsoh | thanks! | |