| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-08-10 | |||
| 19:06:56 | dansmith | correct | |
| 19:07:25 | dansmith | I will make the record_task_start log always to see if it's getting recorded | |
| 19:08:00 | gmaan | k | |
| 19:09:22 | gmaan | dansmith: wait, I think it is completed in rpcserver.wait() before manager log it, I am changing the order in this, can you pick this https://review.opendev.org/c/openstack/nova/+/996583/8 | |
| 19:09:43 | gmaan | so it is more of missing logging and ^^ fixes that | |
| 19:10:11 | dansmith | okay but I just added logs and I never see it get recorded | |
| 19:11:51 | dansmith | gmaan: https://termbin.com/22tn I never see these | |
| 19:12:34 | dansmith | gmaan: I've been sitting too long so I gotta get up, but I'll keep poking at this a bit later | |
| 19:16:02 | gmaan | ok, it should record at the time RPC method is called. i also need to step out for a while. will check it after coming back | |
| 22:55:55 | opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Rewrite share admin guide and add user guide https://review.opendev.org/c/openstack/nova/+/1000166 | |
| #openstack-nova - 2026-08-11 | |||
| 02:02:01 | gmaan | dansmith: i checked/tested it, it is same reason why you are not seeing in-progress task in log because of the order of RPC server wait and manager shutdown. rpcserver.wait |
|
| 02:03:56 | gmaan | and with https://review.opendev.org/c/openstack/nova/+/996583, the order is rpcserver.stop |
|
| 02:04:12 | gmaan | Aug 11 01:28:44 ubuntu-resolute nova-compute[289050]: INFO nova.manager [-] Graceful shutdown initiated with 1 in-progress tasks: stop_instance(instance=c4b95ab0-1098-45a9-b03e-7139e1ce5bfc, request_id=req-9746ce32-bcb0-4913-8861-b1499d6abfd6, elapsed=8.216098441998838) | |
| 02:04:51 | opendevreview | Takashi Kajinami proposed openstack/nova master: Stub kernel support check of AMD-SEV in tests https://review.opendev.org/c/openstack/nova/+/999780 | |
| 02:06:42 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Task tracking mechanism for graceful shutdown https://review.opendev.org/c/openstack/nova/+/996299 | |
| 02:07:07 | gmaan | dansmith: ^^ i added the exact logs checks in the tests | |
| 02:47:39 | gmaan | dansmith: This is a better example where I started 1 start, 2 stop instance operations. It logs all in-progress at start, logs as soon as any task is completed, and keeps logging overall progress every 10 sec; at the end, it summarizes the number of completed and failed tasks. https://pastebin.com/3fLsiMwG | |
| 03:50:14 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Task tracking mechanism for graceful shutdown https://review.opendev.org/c/openstack/nova/+/996299 | |
| 04:14:47 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Track and log the main RPC server in-progress task https://review.opendev.org/c/openstack/nova/+/996583 | |
| 04:14:53 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Unify the thread poool executors into a central factory https://review.opendev.org/c/openstack/nova/+/998571 | |
| 04:14:57 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Shutdown thread pool executors during service shutdown https://review.opendev.org/c/openstack/nova/+/1000211 | |
| 04:15:01 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Add doc and release notes for the graceful shutdown task tracking https://review.opendev.org/c/openstack/nova/+/997141 | |
| 04:27:01 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Task tracking mechanism for graceful shutdown https://review.opendev.org/c/openstack/nova/+/996299 | |
| 04:27:16 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Track and log the main RPC server in-progress task https://review.opendev.org/c/openstack/nova/+/996583 | |
| 04:27:21 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Unify the thread poool executors into a central factory https://review.opendev.org/c/openstack/nova/+/998571 | |
| 04:27:25 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Shutdown thread pool executors during service shutdown https://review.opendev.org/c/openstack/nova/+/1000211 | |
| 04:27:28 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Add doc and release notes for the graceful shutdown task tracking https://review.opendev.org/c/openstack/nova/+/997141 | |
| 08:09:51 | opendevreview | Kubilay Kaptanoglu proposed openstack/nova master: Fix simple-tenant-usage state schema for deleted instances https://review.opendev.org/c/openstack/nova/+/1000415 | |
| 12:02:13 | opendevreview | Rocky proposed openstack/nova master: Add workaround to skip cross-cell service version check https://review.opendev.org/c/openstack/nova/+/1000442 | |
| 13:02:52 | opendevreview | Kubilay Kaptanoglu proposed openstack/nova master: Fix simple-tenant-usage state schema for deleted instances https://review.opendev.org/c/openstack/nova/+/1000415 | |
| 13:42:30 | opendevreview | Daniel Marques Caires proposed openstack/nova master: nova: cast set_host_enabled on disable https://review.opendev.org/c/openstack/nova/+/998464 | |
| 13:43:15 | dansmith | gmaan: hmm, I'm not sure I understand, but let me apply that on top while I let the caffeine soak in and see if it helps | |
| 13:46:04 | dansmith | gmaan: nova-compute still exits immediately | |
| 14:11:23 | opendevreview | Anton Iacobaeus proposed openstack/nova master: doc: Add documentation for Intel TDX https://review.opendev.org/c/openstack/nova/+/1000081 | |
| 14:11:23 | opendevreview | Anton Iacobaeus proposed openstack/nova master: libvirt: reject_mem_enc_instances https://review.opendev.org/c/openstack/nova/+/1000082 | |
| 14:11:24 | opendevreview | Anton Iacobaeus proposed openstack/nova master: libvirt: Add os_firmware_secure https://review.opendev.org/c/openstack/nova/+/1000472 | |
| 14:58:56 | gmaan | dansmith: ohk, what is graceful_shutdown_timeout value | |
| 15:38:54 | dansmith | gmaan: 180 | |
| 15:39:02 | dansmith | I gotta run soon, back later | |
| 15:57:27 | gmaan | then it should work. and i hope manager_shutdown_timeout is not set so default 160 applies | |
| 16:02:07 | gmaan | dansmith: just to make sure, 'nova-compute still exits immediately' you mean instance stay active right and stop instance is being interrupted by shutdown immediately and instance did not go to shutoff state | |
| 16:24:32 | bryanfraschetti | Hi folks, I hope you're doing well. Would someone be able to take a look at this patch series https://review.opendev.org/q/topic:%22improve_attachment_resiliance_against_timeout%22? It prevents situations where cinder and nova become out of sync due to volume attachment timeouts (launchpad.net/bugs/2141355) and adds a nova-manage command that could be used to remedy such situations | |
| 18:33:34 | opendevreview | sean mooney proposed openstack/nova master: libvirt: Support Cyborg PCI ARQ managed mode https://review.opendev.org/c/openstack/nova/+/994579 | |
| 18:33:44 | opendevreview | sean mooney proposed openstack/nova master: doc: Add guide for NVIDIA vGPU with SR-IOV variant drivers https://review.opendev.org/c/openstack/nova/+/994160 | |
| 18:37:55 | opendevreview | Daniel Marques Caires proposed openstack/nova master: nova: cast set_host_enabled on disable https://review.opendev.org/c/openstack/nova/+/998464 | |
| 18:38:00 | opendevreview | Daniel Marques Caires proposed openstack/nova master: Add functional regression test for bug #2156927 https://review.opendev.org/c/openstack/nova/+/1000548 | |
| 19:37:26 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Unify the thread poool executors into a central factory https://review.opendev.org/c/openstack/nova/+/998571 | |
| 20:08:02 | dansmith | gmaan: correct.. | |
| 20:11:00 | gmaan | dansmith: other thing to check is systemd timeout TimeoutStopSec, i think that is set as 300 sec in devstack but if that is less than that will stop service immediately | |
| 20:12:42 | dansmith | gmaan: I'm doing a manual kill, not using systemd | |
| 20:25:29 | gmaan | not sure why its not working, I tried with manual kill also and it works fine for me | |
| 20:26:00 | gmaan | can you please paste the log when stop is started and ended | |
| 20:28:53 | dansmith | gmaan: yeah, I'll do some debug myself, I've just had a bunch of other things going on here | |
| 20:29:03 | gmaan | k | |
| 21:50:29 | dansmith | gmaan: hmm, I dunno what was going on but my devstack VM OOMed and I had to restart it ... after that it all seems to be running properly | |
| 21:50:43 | dansmith | I dunno, must have been something not picking up the code changes I was making or something | |
| 21:51:17 | dansmith | however, while I was in graceful shutdown, I sent a stop command for a second instance and it looks like it was received/ignored by nova-compute which AFAIK means it won't be left on the rabbit queue | |
| 21:51:34 | dansmith | when I restarted, it indeed didn't seem to pick up the queued stop request | |
| 21:51:54 | dansmith | I was hoping we'd be able to close the connection so that rabbit would hold inbound requests like that for playback after the shutdown and restart but it appears not.. what is the expectation/ | |
| 21:51:56 | dansmith | ? | |
| 21:55:53 | cardoe | Any chance https://review.opendev.org/c/openstack/nova/+/989002 can land? | |
| 22:18:38 | gmaan | dansmith: yes, it should have stayed in rabbit Q and picked up if compute is coming up again | |
| 22:19:33 | gmaan | compute main rpc server is stopped so it should not have picked by compute | |
| 22:19:38 | gmaan | let me try | |
| 22:24:51 | gmaan | dansmith: humm it worked for me. did you send 2nd stop immediately after service stop? I am wondering if rpcserver stop was racing with that | |
| 22:31:37 | gmaan | i stop instance1 then stopped service, waited for few sec and then stop on instance2. shutdown waited for instance1 stop to complete and then instance1 was in shutoff state and instance2 as active | |
| 22:32:58 | gmaan | when i started the service again (after 60 sec as i added sleep of 35 sec in stop_instance so want to make sure 2nd stop is not picked by compute), instance2 went to shutoff state | |
| 22:41:34 | dansmith | gmaan: but the message from nova-compute that it received and is ignoring it means that we received it (i.e. dequeued from rabbit, no?) | |
| 22:42:09 | gmaan | dansmith: yes, once received it is gone from rabiit q | |
| 22:43:15 | gmaan | but if 2nd stop was received by compute then it should have completed as we do wait on main RPC server so that already picked up things will be completed before shutdiown | |
| 22:43:26 | dansmith | Received method (60, 30) during closing channel 1. This method will be ignored | |
| 22:43:34 | dansmith | right^ ? | |
| 22:44:39 | gmaan | ignored but whom? compute | |
| 22:45:25 | dansmith | that's in the n-cpu log.. my point is, the fact that it's being even noticed by n-cpu means the message is being dequeued from rabbit right? | |
| 22:45:57 | gmaan | yes, it should be dequeued | |
| 22:46:10 | dansmith | I'm confused | |
| 22:47:03 | dansmith | After I started the shutdown and manager is waiting for tasks to complete (the first stop) I issued the second stop.. looks like to me after the first task is completed and we go to do the actual shutdown is when the above message is logged, right before we actually terminate | |
| 22:47:58 | dansmith | gmaan: https://paste.opendev.org/show/boU5DwvAGi3WLacQcn1r/ | |
| 22:48:19 | dansmith | those first few lines show the end of waiting for the first stop, second stop was already sent at that point | |
| 22:49:06 | dansmith | then actual shutdown starts "graceful shutdown finished", then "Received method during closing channel" and then stopping RPC server, then again for some reason | |
| 22:49:15 | dansmith | but those were like a minute or two after I sent the second stop | |
| 22:49:24 | gmaan | can you paste a few more log above that. whn shutdown started | |
| 22:49:38 | gmaan | because that time we do stop main rpcserver where 2nd stop_instance should not be picked | |
| 22:50:28 | dansmith | it waited for minutes, so nore than a few lines but.. | |
| 22:50:56 | dansmith | https://paste.opendev.org/show/bGlDyiz2mraL9dQ3DSfd/ | |
| 22:51:21 | dansmith | first line is graceful start, then waiting for 1 in-progress, somewhere in there is when I issued the second stop | |
| 22:51:54 | dansmith | then L72 is when the first stop finishes its artificial sleep delay and actually completes | |
| 22:53:34 | gmaan | so L5 is where we stop rpc server and i cannot see where is 2ns stop_instance is receveid | |
| 22:54:13 | dansmith | right (on L5).. isn't this us receiving something on the should-be-closed channel? > Received method (60, 30) during closing chan | |
| 22:54:13 | dansmith | nel 1. This method will be ignored | |
| 22:57:14 | dansmith | after restart I don't see my logs that stop gets called again, which is why I'm assuming it is ^ | |
| 22:57:51 | dansmith | one of my instances is wedged because I was testing the manila share stuff before this.. perhaps I'll try getting a fully clean environment and try to reproduce this in case I've got something stuck that is getting in the way | |
| 22:58:09 | gmaan | need to check on oslo.messaging side what is this exactly because in rpcserver.stop() listener stopped listening anything from queue | |
| 22:58:22 | dansmith | ack, that was my expectation of course | |
| 22:58:38 | dansmith | either way, that aside, it's cool to see the graceful shutdown waiting for the stop task as expected now | |
| 22:59:21 | gmaan | it worked for me but yeah please try on clean env. let me check those warning in my cpu log again. i did not gave attension to those as 2nd stop was picked up and successful after compute came up | |
| 23:06:51 | gmaan | dansmith: i can see those amqp warnings even my scenario is passing means 2nd stop was picked up once compute is started again | |
| 23:07:18 | gmaan | i will check these warnings if those are from loopingcall; or so | |
| 23:07:31 | gmaan | later, need to pick kids from school | |
| #openstack-nova - 2026-08-12 | |||
| 00:45:54 | opendevreview | Merged openstack/nova stable/2026.1: Implementing get_num_instances for ironic virt driver https://review.opendev.org/c/openstack/nova/+/989002 | |