| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-08-10 | |||
| 16:57:04 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Task tracking mechanism for graceful shutdown https://review.opendev.org/c/openstack/nova/+/996299 | |
| 16:58:25 | gmaan | dansmith: ^^ updated the doc string for fixture. for whitebox testing. i replied about the graceful shutdown job. I think that still make sure graceful shutdown works as an end-to-end scenario. Please check if we need to extend/modify those tests | |
| 17:01:02 | dansmith | ack | |
| 17:14:17 | opendevreview | Merged openstack/python-novaclient master: bindep: Remove stale marker https://review.opendev.org/c/openstack/python-novaclient/+/999529 | |
| 18:34:05 | dansmith | gmaan: what are you doing to test long-running tasks? I could do a migration and maybe try to slow it down, but do you have a better obvious thing to try/ | |
| 18:36:14 | gmaan | dansmith: no other things, job start the migration and make sure SIGTERM signal is raised before migration is completed and make sure service does hot shutdown before migration is completed | |
| 18:37:00 | dansmith | okay.. maybe I can just put a sleep(300) in some action | |
| 18:40:14 | gmaan | yeah, that should work. make sure to change the graceful shutdown timeouts if you want shutdown to wait or timeout before action completed | |
| 18:40:35 | gmaan | devstack set graceful_shutdown_timeout to 5 sec for each service | |
| 18:40:54 | gmaan | manager_shutdown_timeout is 160 defaults | |
| 19:04:19 | dansmith | gmaan: yeah, so I put a long sleep at the beginning of stop_instance() and I see it doing the delay.. but when I SIGTERM compute I see "no in progress tasks" and it exits immediately | |
| 19:04:24 | dansmith | timeout is set to 180s | |
| 19:04:46 | dansmith | stop_instance should be waiting because it's not @skip'd right? | |
| 19:05:38 | gmaan | yeah, it should be. even both @skip'd or non @skip'd should be same in-progress tasks list | |
| 19:06:17 | gmaan | did you put sleep in manager stop_instance right? i mean manager picked up stop operation form RPC | |
| 19:06:56 | dansmith | correct | |
| 19:07:25 | dansmith | I will make the record_task_start log always to see if it's getting recorded | |
| 19:08:00 | gmaan | k | |
| 19:09:22 | gmaan | dansmith: wait, I think it is completed in rpcserver.wait() before manager log it, I am changing the order in this, can you pick this https://review.opendev.org/c/openstack/nova/+/996583/8 | |
| 19:09:43 | gmaan | so it is more of missing logging and ^^ fixes that | |
| 19:10:11 | dansmith | okay but I just added logs and I never see it get recorded | |
| 19:11:51 | dansmith | gmaan: https://termbin.com/22tn I never see these | |
| 19:12:34 | dansmith | gmaan: I've been sitting too long so I gotta get up, but I'll keep poking at this a bit later | |
| 19:16:02 | gmaan | ok, it should record at the time RPC method is called. i also need to step out for a while. will check it after coming back | |
| 22:55:55 | opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Rewrite share admin guide and add user guide https://review.opendev.org/c/openstack/nova/+/1000166 | |
| #openstack-nova - 2026-08-11 | |||
| 02:02:01 | gmaan | dansmith: i checked/tested it, it is same reason why you are not seeing in-progress task in log because of the order of RPC server wait and manager shutdown. rpcserver.wait |
|
| 02:03:56 | gmaan | and with https://review.opendev.org/c/openstack/nova/+/996583, the order is rpcserver.stop |
|
| 02:04:12 | gmaan | Aug 11 01:28:44 ubuntu-resolute nova-compute[289050]: INFO nova.manager [-] Graceful shutdown initiated with 1 in-progress tasks: stop_instance(instance=c4b95ab0-1098-45a9-b03e-7139e1ce5bfc, request_id=req-9746ce32-bcb0-4913-8861-b1499d6abfd6, elapsed=8.216098441998838) | |
| 02:04:51 | opendevreview | Takashi Kajinami proposed openstack/nova master: Stub kernel support check of AMD-SEV in tests https://review.opendev.org/c/openstack/nova/+/999780 | |
| 02:06:42 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Task tracking mechanism for graceful shutdown https://review.opendev.org/c/openstack/nova/+/996299 | |
| 02:07:07 | gmaan | dansmith: ^^ i added the exact logs checks in the tests | |
| 02:47:39 | gmaan | dansmith: This is a better example where I started 1 start, 2 stop instance operations. It logs all in-progress at start, logs as soon as any task is completed, and keeps logging overall progress every 10 sec; at the end, it summarizes the number of completed and failed tasks. https://pastebin.com/3fLsiMwG | |
| 03:50:14 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Task tracking mechanism for graceful shutdown https://review.opendev.org/c/openstack/nova/+/996299 | |
| 04:14:47 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Track and log the main RPC server in-progress task https://review.opendev.org/c/openstack/nova/+/996583 | |
| 04:14:53 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Unify the thread poool executors into a central factory https://review.opendev.org/c/openstack/nova/+/998571 | |
| 04:14:57 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Shutdown thread pool executors during service shutdown https://review.opendev.org/c/openstack/nova/+/1000211 | |
| 04:15:01 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Add doc and release notes for the graceful shutdown task tracking https://review.opendev.org/c/openstack/nova/+/997141 | |
| 04:27:01 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Task tracking mechanism for graceful shutdown https://review.opendev.org/c/openstack/nova/+/996299 | |
| 04:27:16 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Track and log the main RPC server in-progress task https://review.opendev.org/c/openstack/nova/+/996583 | |
| 04:27:21 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Unify the thread poool executors into a central factory https://review.opendev.org/c/openstack/nova/+/998571 | |
| 04:27:25 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Shutdown thread pool executors during service shutdown https://review.opendev.org/c/openstack/nova/+/1000211 | |
| 04:27:28 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Add doc and release notes for the graceful shutdown task tracking https://review.opendev.org/c/openstack/nova/+/997141 | |
| 08:09:51 | opendevreview | Kubilay Kaptanoglu proposed openstack/nova master: Fix simple-tenant-usage state schema for deleted instances https://review.opendev.org/c/openstack/nova/+/1000415 | |
| 12:02:13 | opendevreview | Rocky proposed openstack/nova master: Add workaround to skip cross-cell service version check https://review.opendev.org/c/openstack/nova/+/1000442 | |
| 13:02:52 | opendevreview | Kubilay Kaptanoglu proposed openstack/nova master: Fix simple-tenant-usage state schema for deleted instances https://review.opendev.org/c/openstack/nova/+/1000415 | |
| 13:42:30 | opendevreview | Daniel Marques Caires proposed openstack/nova master: nova: cast set_host_enabled on disable https://review.opendev.org/c/openstack/nova/+/998464 | |
| 13:43:15 | dansmith | gmaan: hmm, I'm not sure I understand, but let me apply that on top while I let the caffeine soak in and see if it helps | |
| 13:46:04 | dansmith | gmaan: nova-compute still exits immediately | |
| 14:11:23 | opendevreview | Anton Iacobaeus proposed openstack/nova master: doc: Add documentation for Intel TDX https://review.opendev.org/c/openstack/nova/+/1000081 | |
| 14:11:23 | opendevreview | Anton Iacobaeus proposed openstack/nova master: libvirt: reject_mem_enc_instances https://review.opendev.org/c/openstack/nova/+/1000082 | |
| 14:11:24 | opendevreview | Anton Iacobaeus proposed openstack/nova master: libvirt: Add os_firmware_secure https://review.opendev.org/c/openstack/nova/+/1000472 | |
| 14:58:56 | gmaan | dansmith: ohk, what is graceful_shutdown_timeout value | |
| 15:38:54 | dansmith | gmaan: 180 | |
| 15:39:02 | dansmith | I gotta run soon, back later | |
| 15:57:27 | gmaan | then it should work. and i hope manager_shutdown_timeout is not set so default 160 applies | |
| 16:02:07 | gmaan | dansmith: just to make sure, 'nova-compute still exits immediately' you mean instance stay active right and stop instance is being interrupted by shutdown immediately and instance did not go to shutoff state | |
| 16:24:32 | bryanfraschetti | Hi folks, I hope you're doing well. Would someone be able to take a look at this patch series https://review.opendev.org/q/topic:%22improve_attachment_resiliance_against_timeout%22? It prevents situations where cinder and nova become out of sync due to volume attachment timeouts (launchpad.net/bugs/2141355) and adds a nova-manage command that could be used to remedy such situations | |
| 18:33:34 | opendevreview | sean mooney proposed openstack/nova master: libvirt: Support Cyborg PCI ARQ managed mode https://review.opendev.org/c/openstack/nova/+/994579 | |
| 18:33:44 | opendevreview | sean mooney proposed openstack/nova master: doc: Add guide for NVIDIA vGPU with SR-IOV variant drivers https://review.opendev.org/c/openstack/nova/+/994160 | |
| 18:37:55 | opendevreview | Daniel Marques Caires proposed openstack/nova master: nova: cast set_host_enabled on disable https://review.opendev.org/c/openstack/nova/+/998464 | |
| 18:38:00 | opendevreview | Daniel Marques Caires proposed openstack/nova master: Add functional regression test for bug #2156927 https://review.opendev.org/c/openstack/nova/+/1000548 | |
| 19:37:26 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Unify the thread poool executors into a central factory https://review.opendev.org/c/openstack/nova/+/998571 | |
| 20:08:02 | dansmith | gmaan: correct.. | |
| 20:11:00 | gmaan | dansmith: other thing to check is systemd timeout TimeoutStopSec, i think that is set as 300 sec in devstack but if that is less than that will stop service immediately | |
| 20:12:42 | dansmith | gmaan: I'm doing a manual kill, not using systemd | |
| 20:25:29 | gmaan | not sure why its not working, I tried with manual kill also and it works fine for me | |
| 20:26:00 | gmaan | can you please paste the log when stop is started and ended | |
| 20:28:53 | dansmith | gmaan: yeah, I'll do some debug myself, I've just had a bunch of other things going on here | |
| 20:29:03 | gmaan | k | |
| 21:50:29 | dansmith | gmaan: hmm, I dunno what was going on but my devstack VM OOMed and I had to restart it ... after that it all seems to be running properly | |
| 21:50:43 | dansmith | I dunno, must have been something not picking up the code changes I was making or something | |
| 21:51:17 | dansmith | however, while I was in graceful shutdown, I sent a stop command for a second instance and it looks like it was received/ignored by nova-compute which AFAIK means it won't be left on the rabbit queue | |
| 21:51:34 | dansmith | when I restarted, it indeed didn't seem to pick up the queued stop request | |
| 21:51:54 | dansmith | I was hoping we'd be able to close the connection so that rabbit would hold inbound requests like that for playback after the shutdown and restart but it appears not.. what is the expectation/ | |
| 21:51:56 | dansmith | ? | |
| 21:55:53 | cardoe | Any chance https://review.opendev.org/c/openstack/nova/+/989002 can land? | |
| 22:18:38 | gmaan | dansmith: yes, it should have stayed in rabbit Q and picked up if compute is coming up again | |
| 22:19:33 | gmaan | compute main rpc server is stopped so it should not have picked by compute | |
| 22:19:38 | gmaan | let me try | |
| 22:24:51 | gmaan | dansmith: humm it worked for me. did you send 2nd stop immediately after service stop? I am wondering if rpcserver stop was racing with that | |
| 22:31:37 | gmaan | i stop instance1 then stopped service, waited for few sec and then stop on instance2. shutdown waited for instance1 stop to complete and then instance1 was in shutoff state and instance2 as active | |
| 22:32:58 | gmaan | when i started the service again (after 60 sec as i added sleep of 35 sec in stop_instance so want to make sure 2nd stop is not picked by compute), instance2 went to shutoff state | |
| 22:41:34 | dansmith | gmaan: but the message from nova-compute that it received and is ignoring it means that we received it (i.e. dequeued from rabbit, no?) | |
| 22:42:09 | gmaan | dansmith: yes, once received it is gone from rabiit q | |
| 22:43:15 | gmaan | but if 2nd stop was received by compute then it should have completed as we do wait on main RPC server so that already picked up things will be completed before shutdiown | |
| 22:43:26 | dansmith | Received method (60, 30) during closing channel 1. This method will be ignored | |
| 22:43:34 | dansmith | right^ ? | |
| 22:44:39 | gmaan | ignored but whom? compute | |
| 22:45:25 | dansmith | that's in the n-cpu log.. my point is, the fact that it's being even noticed by n-cpu means the message is being dequeued from rabbit right? | |
| 22:45:57 | gmaan | yes, it should be dequeued | |
| 22:46:10 | dansmith | I'm confused | |
| 22:47:03 | dansmith | After I started the shutdown and manager is waiting for tasks to complete (the first stop) I issued the second stop.. looks like to me after the first task is completed and we go to do the actual shutdown is when the above message is logged, right before we actually terminate | |
| 22:47:58 | dansmith | gmaan: https://paste.opendev.org/show/boU5DwvAGi3WLacQcn1r/ | |
| 22:48:19 | dansmith | those first few lines show the end of waiting for the first stop, second stop was already sent at that point | |
| 22:49:06 | dansmith | then actual shutdown starts "graceful shutdown finished", then "Received method during closing channel" and then stopping RPC server, then again for some reason | |
| 22:49:15 | dansmith | but those were like a minute or two after I sent the second stop | |
| 22:49:24 | gmaan | can you paste a few more log above that. whn shutdown started | |
| 22:49:38 | gmaan | because that time we do stop main rpcserver where 2nd stop_instance should not be picked | |
| 22:50:28 | dansmith | it waited for minutes, so nore than a few lines but.. | |
| 22:50:56 | dansmith | https://paste.opendev.org/show/bGlDyiz2mraL9dQ3DSfd/ | |