| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-08-10 | |||
| 16:29:36 | Leo[m] | Would appreciate any questions, comments, or concerns! | |
| 16:29:56 | Uggla | thx Leo[m] | |
| 16:30:01 | gibi | sorry folks I have to drop for today. o/ | |
| 16:30:37 | gmaan | gibi: sean-k-mooney[m] sambork unifying the threadpool executors is ready, please check https://review.opendev.org/c/openstack/nova/+/998571 | |
| 16:30:37 | Uggla | anything else ? If not I think we can close and keep this meeting short. | |
| 16:31:08 | gmaan | oh, we are still in meeting. did not realize it | |
| 16:31:13 | sambork | gmaan, ack! | |
| 16:31:40 | Uggla | ok closing | |
| 16:31:42 | Uggla | Thanks for joining this meeting. Have a nice day/evening. | |
| 16:31:48 | Uggla | #endmeeting | |
| 16:31:48 | opendevmeet | Meeting ended Mon Aug 10 16:31:48 2026 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | |
| 16:31:48 | opendevmeet | Minutes: https://meetings.opendev.org/meetings/nova/2026/nova.2026-08-10-16.00.html | |
| 16:31:48 | opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/nova/2026/nova.2026-08-10-16.00.txt | |
| 16:31:48 | opendevmeet | Log: https://meetings.opendev.org/meetings/nova/2026/nova.2026-08-10-16.00.log.html | |
| 16:31:56 | lajoskatona | o/ | |
| 16:32:28 | Leo[m] | o/ | |
| 16:32:43 | gmaan | gibi: sambork sean-k-mooney[m] and i separated the graceful shutdown change, the next one on top of it. This is purely to unify the executors things we discussed in the Eventlet meeting. that will help to know if anything it is changing/impacting. | |
| 16:32:59 | Uggla | bauzas, enjoy your PTOs | |
| 16:33:15 | gmaan | bauzas: o/ | |
| 16:34:10 | tkajinam | o/ | |
| 16:35:27 | bauzas | ta | |
| 16:56:38 | opendevreview | sean mooney proposed openstack/nova master: libvirt: Support Cyborg PCI ARQ managed mode https://review.opendev.org/c/openstack/nova/+/994579 | |
| 16:56:39 | opendevreview | sean mooney proposed openstack/nova master: doc: Add guide for NVIDIA vGPU with SR-IOV variant drivers https://review.opendev.org/c/openstack/nova/+/994160 | |
| 16:57:04 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Task tracking mechanism for graceful shutdown https://review.opendev.org/c/openstack/nova/+/996299 | |
| 16:58:25 | gmaan | dansmith: ^^ updated the doc string for fixture. for whitebox testing. i replied about the graceful shutdown job. I think that still make sure graceful shutdown works as an end-to-end scenario. Please check if we need to extend/modify those tests | |
| 17:01:02 | dansmith | ack | |
| 17:14:17 | opendevreview | Merged openstack/python-novaclient master: bindep: Remove stale marker https://review.opendev.org/c/openstack/python-novaclient/+/999529 | |
| 18:34:05 | dansmith | gmaan: what are you doing to test long-running tasks? I could do a migration and maybe try to slow it down, but do you have a better obvious thing to try/ | |
| 18:36:14 | gmaan | dansmith: no other things, job start the migration and make sure SIGTERM signal is raised before migration is completed and make sure service does hot shutdown before migration is completed | |
| 18:37:00 | dansmith | okay.. maybe I can just put a sleep(300) in some action | |
| 18:40:14 | gmaan | yeah, that should work. make sure to change the graceful shutdown timeouts if you want shutdown to wait or timeout before action completed | |
| 18:40:35 | gmaan | devstack set graceful_shutdown_timeout to 5 sec for each service | |
| 18:40:54 | gmaan | manager_shutdown_timeout is 160 defaults | |
| 19:04:19 | dansmith | gmaan: yeah, so I put a long sleep at the beginning of stop_instance() and I see it doing the delay.. but when I SIGTERM compute I see "no in progress tasks" and it exits immediately | |
| 19:04:24 | dansmith | timeout is set to 180s | |
| 19:04:46 | dansmith | stop_instance should be waiting because it's not @skip'd right? | |
| 19:05:38 | gmaan | yeah, it should be. even both @skip'd or non @skip'd should be same in-progress tasks list | |
| 19:06:17 | gmaan | did you put sleep in manager stop_instance right? i mean manager picked up stop operation form RPC | |
| 19:06:56 | dansmith | correct | |
| 19:07:25 | dansmith | I will make the record_task_start log always to see if it's getting recorded | |
| 19:08:00 | gmaan | k | |
| 19:09:22 | gmaan | dansmith: wait, I think it is completed in rpcserver.wait() before manager log it, I am changing the order in this, can you pick this https://review.opendev.org/c/openstack/nova/+/996583/8 | |
| 19:09:43 | gmaan | so it is more of missing logging and ^^ fixes that | |
| 19:10:11 | dansmith | okay but I just added logs and I never see it get recorded | |
| 19:11:51 | dansmith | gmaan: https://termbin.com/22tn I never see these | |
| 19:12:34 | dansmith | gmaan: I've been sitting too long so I gotta get up, but I'll keep poking at this a bit later | |
| 19:16:02 | gmaan | ok, it should record at the time RPC method is called. i also need to step out for a while. will check it after coming back | |
| 22:55:55 | opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Rewrite share admin guide and add user guide https://review.opendev.org/c/openstack/nova/+/1000166 | |
| #openstack-nova - 2026-08-11 | |||
| 02:02:01 | gmaan | dansmith: i checked/tested it, it is same reason why you are not seeing in-progress task in log because of the order of RPC server wait and manager shutdown. rpcserver.wait |
|
| 02:03:56 | gmaan | and with https://review.opendev.org/c/openstack/nova/+/996583, the order is rpcserver.stop |
|
| 02:04:12 | gmaan | Aug 11 01:28:44 ubuntu-resolute nova-compute[289050]: INFO nova.manager [-] Graceful shutdown initiated with 1 in-progress tasks: stop_instance(instance=c4b95ab0-1098-45a9-b03e-7139e1ce5bfc, request_id=req-9746ce32-bcb0-4913-8861-b1499d6abfd6, elapsed=8.216098441998838) | |
| 02:04:51 | opendevreview | Takashi Kajinami proposed openstack/nova master: Stub kernel support check of AMD-SEV in tests https://review.opendev.org/c/openstack/nova/+/999780 | |
| 02:06:42 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Task tracking mechanism for graceful shutdown https://review.opendev.org/c/openstack/nova/+/996299 | |
| 02:07:07 | gmaan | dansmith: ^^ i added the exact logs checks in the tests | |
| 02:47:39 | gmaan | dansmith: This is a better example where I started 1 start, 2 stop instance operations. It logs all in-progress at start, logs as soon as any task is completed, and keeps logging overall progress every 10 sec; at the end, it summarizes the number of completed and failed tasks. https://pastebin.com/3fLsiMwG | |
| 03:50:14 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Task tracking mechanism for graceful shutdown https://review.opendev.org/c/openstack/nova/+/996299 | |
| 04:14:47 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Track and log the main RPC server in-progress task https://review.opendev.org/c/openstack/nova/+/996583 | |
| 04:14:53 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Unify the thread poool executors into a central factory https://review.opendev.org/c/openstack/nova/+/998571 | |
| 04:14:57 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Shutdown thread pool executors during service shutdown https://review.opendev.org/c/openstack/nova/+/1000211 | |
| 04:15:01 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Add doc and release notes for the graceful shutdown task tracking https://review.opendev.org/c/openstack/nova/+/997141 | |
| 04:27:01 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Task tracking mechanism for graceful shutdown https://review.opendev.org/c/openstack/nova/+/996299 | |
| 04:27:16 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Track and log the main RPC server in-progress task https://review.opendev.org/c/openstack/nova/+/996583 | |
| 04:27:21 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Unify the thread poool executors into a central factory https://review.opendev.org/c/openstack/nova/+/998571 | |
| 04:27:25 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Shutdown thread pool executors during service shutdown https://review.opendev.org/c/openstack/nova/+/1000211 | |
| 04:27:28 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Add doc and release notes for the graceful shutdown task tracking https://review.opendev.org/c/openstack/nova/+/997141 | |
| 08:09:51 | opendevreview | Kubilay Kaptanoglu proposed openstack/nova master: Fix simple-tenant-usage state schema for deleted instances https://review.opendev.org/c/openstack/nova/+/1000415 | |
| 12:02:13 | opendevreview | Rocky proposed openstack/nova master: Add workaround to skip cross-cell service version check https://review.opendev.org/c/openstack/nova/+/1000442 | |
| 13:02:52 | opendevreview | Kubilay Kaptanoglu proposed openstack/nova master: Fix simple-tenant-usage state schema for deleted instances https://review.opendev.org/c/openstack/nova/+/1000415 | |
| 13:42:30 | opendevreview | Daniel Marques Caires proposed openstack/nova master: nova: cast set_host_enabled on disable https://review.opendev.org/c/openstack/nova/+/998464 | |
| 13:43:15 | dansmith | gmaan: hmm, I'm not sure I understand, but let me apply that on top while I let the caffeine soak in and see if it helps | |
| 13:46:04 | dansmith | gmaan: nova-compute still exits immediately | |
| 14:11:23 | opendevreview | Anton Iacobaeus proposed openstack/nova master: doc: Add documentation for Intel TDX https://review.opendev.org/c/openstack/nova/+/1000081 | |
| 14:11:23 | opendevreview | Anton Iacobaeus proposed openstack/nova master: libvirt: reject_mem_enc_instances https://review.opendev.org/c/openstack/nova/+/1000082 | |
| 14:11:24 | opendevreview | Anton Iacobaeus proposed openstack/nova master: libvirt: Add os_firmware_secure https://review.opendev.org/c/openstack/nova/+/1000472 | |
| 14:58:56 | gmaan | dansmith: ohk, what is graceful_shutdown_timeout value | |
| 15:38:54 | dansmith | gmaan: 180 | |
| 15:39:02 | dansmith | I gotta run soon, back later | |
| 15:57:27 | gmaan | then it should work. and i hope manager_shutdown_timeout is not set so default 160 applies | |
| 16:02:07 | gmaan | dansmith: just to make sure, 'nova-compute still exits immediately' you mean instance stay active right and stop instance is being interrupted by shutdown immediately and instance did not go to shutoff state | |
| 16:24:32 | bryanfraschetti | Hi folks, I hope you're doing well. Would someone be able to take a look at this patch series https://review.opendev.org/q/topic:%22improve_attachment_resiliance_against_timeout%22? It prevents situations where cinder and nova become out of sync due to volume attachment timeouts (launchpad.net/bugs/2141355) and adds a nova-manage command that could be used to remedy such situations | |
| 18:33:34 | opendevreview | sean mooney proposed openstack/nova master: libvirt: Support Cyborg PCI ARQ managed mode https://review.opendev.org/c/openstack/nova/+/994579 | |
| 18:33:44 | opendevreview | sean mooney proposed openstack/nova master: doc: Add guide for NVIDIA vGPU with SR-IOV variant drivers https://review.opendev.org/c/openstack/nova/+/994160 | |
| 18:37:55 | opendevreview | Daniel Marques Caires proposed openstack/nova master: nova: cast set_host_enabled on disable https://review.opendev.org/c/openstack/nova/+/998464 | |
| 18:38:00 | opendevreview | Daniel Marques Caires proposed openstack/nova master: Add functional regression test for bug #2156927 https://review.opendev.org/c/openstack/nova/+/1000548 | |
| 19:37:26 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Unify the thread poool executors into a central factory https://review.opendev.org/c/openstack/nova/+/998571 | |
| 20:08:02 | dansmith | gmaan: correct.. | |
| 20:11:00 | gmaan | dansmith: other thing to check is systemd timeout TimeoutStopSec, i think that is set as 300 sec in devstack but if that is less than that will stop service immediately | |
| 20:12:42 | dansmith | gmaan: I'm doing a manual kill, not using systemd | |
| 20:25:29 | gmaan | not sure why its not working, I tried with manual kill also and it works fine for me | |
| 20:26:00 | gmaan | can you please paste the log when stop is started and ended | |
| 20:28:53 | dansmith | gmaan: yeah, I'll do some debug myself, I've just had a bunch of other things going on here | |
| 20:29:03 | gmaan | k | |
| 21:50:29 | dansmith | gmaan: hmm, I dunno what was going on but my devstack VM OOMed and I had to restart it ... after that it all seems to be running properly | |
| 21:50:43 | dansmith | I dunno, must have been something not picking up the code changes I was making or something | |
| 21:51:17 | dansmith | however, while I was in graceful shutdown, I sent a stop command for a second instance and it looks like it was received/ignored by nova-compute which AFAIK means it won't be left on the rabbit queue | |
| 21:51:34 | dansmith | when I restarted, it indeed didn't seem to pick up the queued stop request | |
| 21:51:54 | dansmith | I was hoping we'd be able to close the connection so that rabbit would hold inbound requests like that for playback after the shutdown and restart but it appears not.. what is the expectation/ | |
| 21:51:56 | dansmith | ? | |
| 21:55:53 | cardoe | Any chance https://review.opendev.org/c/openstack/nova/+/989002 can land? | |
| 22:18:38 | gmaan | dansmith: yes, it should have stayed in rabbit Q and picked up if compute is coming up again | |