| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-08-11 | |||
| 20:11:00 | gmaan | dansmith: other thing to check is systemd timeout TimeoutStopSec, i think that is set as 300 sec in devstack but if that is less than that will stop service immediately | |
| 20:12:42 | dansmith | gmaan: I'm doing a manual kill, not using systemd | |
| 20:25:29 | gmaan | not sure why its not working, I tried with manual kill also and it works fine for me | |
| 20:26:00 | gmaan | can you please paste the log when stop is started and ended | |
| 20:28:53 | dansmith | gmaan: yeah, I'll do some debug myself, I've just had a bunch of other things going on here | |
| 20:29:03 | gmaan | k | |
| 21:50:29 | dansmith | gmaan: hmm, I dunno what was going on but my devstack VM OOMed and I had to restart it ... after that it all seems to be running properly | |
| 21:50:43 | dansmith | I dunno, must have been something not picking up the code changes I was making or something | |
| 21:51:17 | dansmith | however, while I was in graceful shutdown, I sent a stop command for a second instance and it looks like it was received/ignored by nova-compute which AFAIK means it won't be left on the rabbit queue | |
| 21:51:34 | dansmith | when I restarted, it indeed didn't seem to pick up the queued stop request | |
| 21:51:54 | dansmith | I was hoping we'd be able to close the connection so that rabbit would hold inbound requests like that for playback after the shutdown and restart but it appears not.. what is the expectation/ | |
| 21:51:56 | dansmith | ? | |
| 21:55:53 | cardoe | Any chance https://review.opendev.org/c/openstack/nova/+/989002 can land? | |
| 22:18:38 | gmaan | dansmith: yes, it should have stayed in rabbit Q and picked up if compute is coming up again | |
| 22:19:33 | gmaan | compute main rpc server is stopped so it should not have picked by compute | |
| 22:19:38 | gmaan | let me try | |
| 22:24:51 | gmaan | dansmith: humm it worked for me. did you send 2nd stop immediately after service stop? I am wondering if rpcserver stop was racing with that | |
| 22:31:37 | gmaan | i stop instance1 then stopped service, waited for few sec and then stop on instance2. shutdown waited for instance1 stop to complete and then instance1 was in shutoff state and instance2 as active | |
| 22:32:58 | gmaan | when i started the service again (after 60 sec as i added sleep of 35 sec in stop_instance so want to make sure 2nd stop is not picked by compute), instance2 went to shutoff state | |
| 22:41:34 | dansmith | gmaan: but the message from nova-compute that it received and is ignoring it means that we received it (i.e. dequeued from rabbit, no?) | |
| 22:42:09 | gmaan | dansmith: yes, once received it is gone from rabiit q | |
| 22:43:15 | gmaan | but if 2nd stop was received by compute then it should have completed as we do wait on main RPC server so that already picked up things will be completed before shutdiown | |
| 22:43:26 | dansmith | Received method (60, 30) during closing channel 1. This method will be ignored | |
| 22:43:34 | dansmith | right^ ? | |
| 22:44:39 | gmaan | ignored but whom? compute | |
| 22:45:25 | dansmith | that's in the n-cpu log.. my point is, the fact that it's being even noticed by n-cpu means the message is being dequeued from rabbit right? | |
| 22:45:57 | gmaan | yes, it should be dequeued | |
| 22:46:10 | dansmith | I'm confused | |
| 22:47:03 | dansmith | After I started the shutdown and manager is waiting for tasks to complete (the first stop) I issued the second stop.. looks like to me after the first task is completed and we go to do the actual shutdown is when the above message is logged, right before we actually terminate | |
| 22:47:58 | dansmith | gmaan: https://paste.opendev.org/show/boU5DwvAGi3WLacQcn1r/ | |
| 22:48:19 | dansmith | those first few lines show the end of waiting for the first stop, second stop was already sent at that point | |
| 22:49:06 | dansmith | then actual shutdown starts "graceful shutdown finished", then "Received method during closing channel" and then stopping RPC server, then again for some reason | |
| 22:49:15 | dansmith | but those were like a minute or two after I sent the second stop | |
| 22:49:24 | gmaan | can you paste a few more log above that. whn shutdown started | |
| 22:49:38 | gmaan | because that time we do stop main rpcserver where 2nd stop_instance should not be picked | |
| 22:50:28 | dansmith | it waited for minutes, so nore than a few lines but.. | |
| 22:50:56 | dansmith | https://paste.opendev.org/show/bGlDyiz2mraL9dQ3DSfd/ | |
| 22:51:21 | dansmith | first line is graceful start, then waiting for 1 in-progress, somewhere in there is when I issued the second stop | |
| 22:51:54 | dansmith | then L72 is when the first stop finishes its artificial sleep delay and actually completes | |
| 22:53:34 | gmaan | so L5 is where we stop rpc server and i cannot see where is 2ns stop_instance is receveid | |
| 22:54:13 | dansmith | right (on L5).. isn't this us receiving something on the should-be-closed channel? > Received method (60, 30) during closing chan | |
| 22:54:13 | dansmith | nel 1. This method will be ignored | |
| 22:57:14 | dansmith | after restart I don't see my logs that stop gets called again, which is why I'm assuming it is ^ | |
| 22:57:51 | dansmith | one of my instances is wedged because I was testing the manila share stuff before this.. perhaps I'll try getting a fully clean environment and try to reproduce this in case I've got something stuck that is getting in the way | |
| 22:58:09 | gmaan | need to check on oslo.messaging side what is this exactly because in rpcserver.stop() listener stopped listening anything from queue | |
| 22:58:22 | dansmith | ack, that was my expectation of course | |
| 22:58:38 | dansmith | either way, that aside, it's cool to see the graceful shutdown waiting for the stop task as expected now | |
| 22:59:21 | gmaan | it worked for me but yeah please try on clean env. let me check those warning in my cpu log again. i did not gave attension to those as 2nd stop was picked up and successful after compute came up | |
| 23:06:51 | gmaan | dansmith: i can see those amqp warnings even my scenario is passing means 2nd stop was picked up once compute is started again | |
| 23:07:18 | gmaan | i will check these warnings if those are from loopingcall; or so | |
| 23:07:31 | gmaan | later, need to pick kids from school | |
| #openstack-nova - 2026-08-12 | |||
| 00:45:54 | opendevreview | Merged openstack/nova stable/2026.1: Implementing get_num_instances for ironic virt driver https://review.opendev.org/c/openstack/nova/+/989002 | |
| 05:01:25 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Unify the thread pool executors into a central factory https://review.opendev.org/c/openstack/nova/+/998571 | |
| 05:20:47 | opendevreview | Bhavana proposed openstack/nova-specs master: Add spec for MIG-aware PCI Host Weigher https://review.opendev.org/c/openstack/nova-specs/+/998753 | |
| 06:36:03 | opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Extract share management from ComputeManager https://review.opendev.org/c/openstack/nova/+/999349 | |
| 06:36:04 | opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Support cold migration with virtiofs shares https://review.opendev.org/c/openstack/nova/+/989633 | |
| 09:47:27 | opendevreview | Stephen Finucane proposed openstack/nova master: Replace use of datetime.timezone.UTC https://review.opendev.org/c/openstack/nova/+/989515 | |
| 09:47:27 | opendevreview | Stephen Finucane proposed openstack/nova master: trivial: Fix outstanding flake8 issues https://review.opendev.org/c/openstack/nova/+/999714 | |
| 09:47:28 | opendevreview | Stephen Finucane proposed openstack/nova master: Add ruff-check https://review.opendev.org/c/openstack/nova/+/974441 | |
| 09:47:28 | opendevreview | Stephen Finucane proposed openstack/nova master: Enable G (flake8-logging-format) rules https://review.opendev.org/c/openstack/nova/+/998557 | |
| 13:12:25 | gibi | tkajinam: antia: I'm not 100% clear on the status of the TDX discussion. Is there still an open question or antia just needs to adjust the series according to the latest discussion? | |
| 13:16:12 | antia | gibi: I think all is clear now, just doing the final adjustments | |
| 13:18:27 | gibi | antia: cool. thanks | |
| 13:46:52 | opendevreview | Kamil Sambor proposed openstack/nova master: Add regression test for BFV evacuate + Cinder timeout https://review.opendev.org/c/openstack/nova/+/999261 | |
| 13:52:44 | dansmith | gmaan: okay clean install this morning and I'm able to see the full thing working.. I do still get the weird "received, ignored" message before the compute shuts down | |
| 13:53:16 | dansmith | gmaan: however, I can see the second stop queued in rabbit after shutdown and can see it replayed after restart and have it actually do the thing we want, so that's cool | |
| 13:53:41 | dansmith | I dunno why not before, but like I say I was pivoting from the manila stuff so something must not have been right | |
| 15:35:29 | gmaan | dansmith: thanks for testing and review. i did not time to check that warning things yesterday but I will check if that is something valid one because amqp listener stopped listening but queue was up or so | |
| 17:17:10 | gmaan | melwitt: these two changes for graceful shutdown are ready for 2nd core, first one you already reviewed earlier with +1, please check https://review.opendev.org/c/openstack/nova/+/996299 https://review.opendev.org/c/openstack/nova/+/996583 | |
| 17:17:40 | melwitt | gmaan: nice, will look | |
| 17:17:53 | gmaan | thanks | |
| 17:54:38 | opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Support cold migration with virtiofs shares https://review.opendev.org/c/openstack/nova/+/989633 | |
| 20:58:37 | opendevreview | Ashish Gupta proposed openstack/nova master: Make functional-py313-threading voting https://review.opendev.org/c/openstack/nova/+/997216 | |
| 20:58:38 | opendevreview | Ashish Gupta proposed openstack/nova master: Fix test_fill_vifs_migration under native threading https://review.opendev.org/c/openstack/nova/+/1000780 | |
| #openstack-nova - 2026-08-13 | |||
| 04:01:41 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Unify the thread pool executors into a central factory https://review.opendev.org/c/openstack/nova/+/998571 | |
| 04:22:06 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Shutdown thread pool executors during service shutdown https://review.opendev.org/c/openstack/nova/+/1000211 | |
| 05:07:18 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Shutdown thread pool executors during graceful shutdown https://review.opendev.org/c/openstack/nova/+/1000211 | |
| 05:07:47 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Add doc and release notes for the graceful shutdown task tracking https://review.opendev.org/c/openstack/nova/+/997141 | |
| 07:25:45 | opendevreview | Anton Iacobaeus proposed openstack/nova master: libvirt: Generalize memory encryption config https://review.opendev.org/c/openstack/nova/+/998466 | |
| 07:25:46 | opendevreview | Anton Iacobaeus proposed openstack/nova master: libvirt: Add memory encryption config for TDX https://review.opendev.org/c/openstack/nova/+/998608 | |
| 07:25:46 | opendevreview | Anton Iacobaeus proposed openstack/nova master: libvirt: Launch security Intel TDX https://review.opendev.org/c/openstack/nova/+/999249 | |
| 07:25:47 | opendevreview | Anton Iacobaeus proposed openstack/nova master: doc: Add documentation for Intel TDX https://review.opendev.org/c/openstack/nova/+/1000081 | |
| 07:25:47 | opendevreview | Anton Iacobaeus proposed openstack/nova master: libvirt: reject_mem_enc_instances https://review.opendev.org/c/openstack/nova/+/1000082 | |
| 07:25:48 | opendevreview | Anton Iacobaeus proposed openstack/nova master: libvirt: Add os_firmware_secure https://review.opendev.org/c/openstack/nova/+/1000472 | |
| 07:57:47 | opendevreview | Anton Iacobaeus proposed openstack/nova master: libvirt: Check only x86_64 arch for Intel TDX https://review.opendev.org/c/openstack/nova/+/1000805 | |
| 08:33:51 | opendevreview | Lajos Katona proposed openstack/nova master: libvirt: Add test for MTU overwrite during live migration https://review.opendev.org/c/openstack/nova/+/995803 | |
| 08:33:51 | opendevreview | Lajos Katona proposed openstack/nova master: libvirt: Avoid setting MTU during live migration if different https://review.opendev.org/c/openstack/nova/+/995809 | |
| 08:33:52 | opendevreview | Lajos Katona proposed openstack/nova master: Add regression test to repoduce bug 1854844 https://review.opendev.org/c/openstack/nova/+/991772 | |
| 08:33:52 | opendevreview | Lajos Katona proposed openstack/nova master: libvirt: Preserve queue_size during live migration https://review.opendev.org/c/openstack/nova/+/995841 | |
| 08:38:57 | RomanHros[m] | Hello, please someone look at the https://launchpad.net/bugs/2016173. It looks like very beneficial patch but seems to be forgotten... | |
| 09:42:24 | frickler | melwitt: sean-k-mooney[m]: ^^ any reason this is stuck except lack of focus? https://review.opendev.org/c/openstack/nova/+/880399 | |
| 09:45:19 | sean-k-mooney[m] | nothing comes to mind | |
| 09:48:39 | sean-k-mooney[m] | im still waking up but i can take a look properly when im at my desk | |
| 10:39:31 | opendevreview | Stephen Finucane proposed openstack/nova master: trivial: Fix outstanding flake8 issues https://review.opendev.org/c/openstack/nova/+/999714 | |
| 10:39:31 | opendevreview | Stephen Finucane proposed openstack/nova master: Add ruff-check https://review.opendev.org/c/openstack/nova/+/974441 | |
| 10:39:32 | opendevreview | Stephen Finucane proposed openstack/nova master: Enable G (flake8-logging-format) rules https://review.opendev.org/c/openstack/nova/+/998557 | |
| 13:24:51 | gibi | antia: tkajinam: after pulling down the latest version of the TDX series to the same machine I used before for testing I libvirt rejects the VM boot with | |
| 13:24:54 | gibi | Aug 13 16:23:40 beast01.rhos.lab.eng.brq2.redhat.com nova-compute[1108220]: ERROR nova.compute.manager [instance: e034aa57-fdca-4ea2-831f-41bcf51eb1d3] libvirt.libvirtError: operation failed: Unable to find 'efi' firmware that is compatible with the current configuration | |
| 13:25:18 | gibi | so I guess this is connected to the secure loader / secure firmware change | |
| 13:25:30 | antia | gibi: okay interesting | |