| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-02-25 | |||
| 15:51:21 | sean-k-mooney | ^ cordiating with graceful shutdown was the only topic i was going to raise | |
| 15:51:47 | sean-k-mooney | im review that serice now but honestly i was expect gmann ot have creted fucntionla test to test graceful shutdown | |
| 15:51:48 | gibi | bauzas: yeah the eventlet series is approved but gate is slow and unstable so it will take time. | |
| 15:52:05 | gibi | I'm monitoring it... | |
| 15:52:17 | sean-k-mooney | so im not sure that is ready i am also not really ok with callign it anything more then expreimetnal until we comple phase 2 and 3 | |
| 15:58:59 | bauzas | I guess I'll give another swing to vtpm live-migration series | |
| 16:41:42 | tkajinam | bauzas, thanks ! | |
| 16:42:20 | melwitt | bauzas, dansmith: for vtpm the patches targeted for FF are the fixups patch https://review.opendev.org/c/openstack/nova/+/976316 and the service version bump https://review.opendev.org/c/openstack/nova/+/975724 (since we got 'host' secret security mode available at this time). the rest will have to wait until H | |
| 16:45:47 | dansmith | melwitt: ack, I'll try to circle back soon, sorry for being pulled in multiple directions | |
| 16:46:11 | melwitt | thanks 🙏 | |
| 17:08:04 | bauzas | melwitt: noted, I'll look at those before ending my day | |
| 17:09:08 | melwitt | thanks 🙏 | |
| 17:20:56 | opendevreview | Stephen Finucane proposed openstack/nova master: docs: Update REST API version history for 2.101 https://review.opendev.org/c/openstack/nova/+/978002 | |
| 17:36:32 | opendevreview | Takashi Kajinami proposed openstack/nova master: Remove unused gabbi https://review.opendev.org/c/openstack/nova/+/978006 | |
| 17:47:48 | opendevreview | Merged openstack/nova master: libvirt: Add capability to load smm feature from existing xml https://review.opendev.org/c/openstack/nova/+/969131 | |
| 18:13:11 | opendevreview | Merged openstack/nova master: libvirt: Use firmware auto-selection by libvirt https://review.opendev.org/c/openstack/nova/+/969132 | |
| 19:47:38 | gmaan | sean-k-mooney: thanks for review on graceful shutdown, i am ok to callout the experimental in releasenotes and once we finish the all phases we will have exact list/things which finish or move to safe state in shitdown | |
| 19:48:26 | gmaan | sean-k-mooney: bauzas gibi seeing the gate eventlet series might take time to merge and rebasing the graceful shutdown can miss the FF date. | |
| 19:48:57 | gmaan | as eventlet changes are with +w let me rebase in advance so that graceful shutdown can be in gate in parallel | |
| 19:49:35 | sean-k-mooney | ack | |
| 19:50:17 | sean-k-mooney | i really jsut dont want to get a custoemr case in the future whre the go "you said shutdown was graceful" and the interupted a live migration and got into an underined state | |
| 19:50:55 | sean-k-mooney | so if we call it experimetal untill pahse 2 or 3 is done i think its fine to merge although i woudl really like to see functional tests added next cycle to buidl conficance in this | |
| 19:51:08 | gmaan | for me interrupted is fine as it can be solved with the configured timeout but an underined state i agree that we should make sure | |
| 19:51:10 | sean-k-mooney | the ansible/bash tests help | |
| 19:51:56 | gmaan | sean-k-mooney: functional test for phase 2 and 3 will cover the things but for phase1 which is all about RPC redesign cannot be cover by functional tests as they use fake driver | |
| 19:52:01 | sean-k-mooney | oh ya i have said its totally fine with me if we just abort the ongoing live migration or interupt them in a fixable state. | |
| 19:52:13 | gmaan | yeah | |
| 19:52:21 | sean-k-mooney | the funcitonl tests can use the livbirt driver | |
| 19:52:28 | sean-k-mooney | and we have live migration fucntional test already | |
| 19:52:32 | gmaan | I mean fake RPC driver | |
| 19:52:51 | gmaan | they do not use real RPC | |
| 19:53:37 | sean-k-mooney | thye use the fake rpc yes, the in memory one but ya the reaosn i bring it up was it was stated we would have them in the spec and i was expecting to see them in the code when i went to review | |
| 19:54:16 | sean-k-mooney | so we need to modify oslo to make the fake tansport supprot more the one rpc server ? | |
| 19:54:29 | sean-k-mooney | or can we make that work form the nova side? | |
| 19:55:02 | gmaan | I will say integration tests which is doable are ok for the RPC part | |
| 19:55:42 | gmaan | sand for phase 2/3 yes, it can be covered in functional with better control | |
| 19:56:18 | sean-k-mooney | ack, i can live with that. i dont relaly trust the integration tests to test this on its own | |
| 19:56:40 | sean-k-mooney | but we agree to make incremental progress so thats ok | |
| 19:57:24 | gmaan | k, let me rebase and fix the other comments you have | |
| 20:22:52 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Use 2nd RPC server in compute operations https://review.opendev.org/c/openstack/nova/+/975588 | |
| 20:32:58 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Use 2nd RPC server in compute operations https://review.opendev.org/c/openstack/nova/+/975588 | |
| 20:36:08 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Prepare resize/cold migration for graceful shutdown https://review.opendev.org/c/openstack/nova/+/977182 | |
| 20:46:25 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Add more test for graceful shutdown https://review.opendev.org/c/openstack/nova/+/977405 | |
| 21:00:51 | opendevreview | Merged openstack/nova master: api: Add response body schemas for servers APIs (5/6) https://review.opendev.org/c/openstack/nova/+/956239 | |
| 21:19:06 | gmaan | bauzas: sean-k-mooney gibi it seems, there is no merge conflict between eventlet and graceful shutdown changes, but anyways I have rebased latter one on eventlet. | |
| 21:19:12 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Add operator document for graceful shutdown https://review.opendev.org/c/openstack/nova/+/977590 | |
| 21:19:18 | gmaan | sean-k-mooney: updated ^^ doc change also | |
| 21:33:13 | sean-k-mooney | gmaan: im +2 on all of your changes thanks for the update | |
| 21:34:32 | sean-k-mooney | gmaan: gibi is unfutently not have great luck with the ci gods at the moment but maybe you can recheck the eventlet patches later in NA time when the ci might be less flaky | |
| 21:45:00 | gmaan | sean-k-mooney: thanks, I will keep eyes on CI | |
| 22:02:15 | cardoe | sean-k-mooney: ping https://review.opendev.org/c/openstack/nova/+/972601 | |
| 22:02:47 | cardoe | I just posted a reply to you but I wanted to discuss it. | |
| 22:03:49 | cardoe | I believe you're misunderstanding what my change did. My change is matching the behavior of KVM/libvirt with the ironic driver. And making it so that the full backtrace is unnecessary to the end user to troubleshoot. | |
| 22:09:25 | cardoe | There's half a dozen bug reports / issues today where if I do X with KVM/libvirt, I get a clear error message explaining why it didn't work. But when I use the ironic driver via nova, the error message I get as a user is either wrong or generic and the only way to work out what happened is to read the full backtrace. Inside of that full backtrace is the same one that's being delivered back to the user in the KVM/libvirt case. | |
| 22:32:07 | gmaan | sean-k-mooney: thanks | |
| 22:32:33 | gmaan | gibi: if you can re-review it https://review.opendev.org/c/openstack/nova/+/975588 this first one | |
| 22:49:36 | opendevreview | melanie witt proposed openstack/nova master: TPM: bump service version to enable live migration https://review.opendev.org/c/openstack/nova/+/975724 | |
| 22:49:37 | opendevreview | melanie witt proposed openstack/nova master: TPM: support live migration of `deployment` secret security https://review.opendev.org/c/openstack/nova/+/925771 | |
| 22:49:37 | opendevreview | melanie witt proposed openstack/nova master: TPM: test live migration between hosts with different security https://review.opendev.org/c/openstack/nova/+/952629 | |
| 22:49:38 | opendevreview | melanie witt proposed openstack/nova master: TPM: add late check for supported TPM secret security https://review.opendev.org/c/openstack/nova/+/956975 | |
| 22:49:38 | opendevreview | melanie witt proposed openstack/nova master: TPM: enable conversion of secret security modes via resize https://review.opendev.org/c/openstack/nova/+/962052 | |
| 22:49:39 | opendevreview | melanie witt proposed openstack/nova master: DNM vtpm tempest https://review.opendev.org/c/openstack/nova/+/957477 | |
| 23:54:42 | cardoe | gmaan: neat. So we'll be able to shutdown gracefully and let it continue to manage things. | |
| 23:59:03 | gmaan | cardoe: yes, we introduced the 2 RPC server in compute so that compute stop accepting the new request and finish the in-progress one. in this phase-1, services wait for configuration timeout which will be replaced by task tracking in next part. | |
| #openstack-nova - 2026-02-26 | |||
| 00:05:12 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Add operator document for graceful shutdown https://review.opendev.org/c/openstack/nova/+/977590 | |
| 01:33:05 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Add operator document for graceful shutdown https://review.opendev.org/c/openstack/nova/+/977590 | |
| 01:35:56 | gmaan | gate is in its worst situation | |
| 02:38:41 | gmaan | gibi: sean-k-mooney and this doc also need re-review, fixed pep8 spell things https://review.opendev.org/c/openstack/nova/+/977590 | |
| 03:00:11 | opendevreview | Merged openstack/nova master: api: Add response body schemas for servers APIs (6/6) https://review.opendev.org/c/openstack/nova/+/956240 | |
| 08:50:26 | opendevreview | Merged openstack/nova master: api: Add response body schemas for server shares APIs https://review.opendev.org/c/openstack/nova/+/956266 | |
| 08:50:39 | opendevreview | Merged openstack/nova master: tests: Invert validation check https://review.opendev.org/c/openstack/nova/+/956241 | |
| 08:59:22 | gibi | I'm envy about the folks who can land patches through the gate. The eventlet series now stuck for more than 24 hours behind random failures. I feel bad about grinding rechecks but I will do it anyhow :/ | |
| 09:15:37 | frickler | let me know if you need help like re-enqueueing changes directly back into gate or even force-merging. once you're sufficiently sure the failures are unrelated IMO it doesn't make sense to keep doing rechecks and using up CI resources | |
| 09:16:11 | frickler | and I say this knowing that getting tests more stable is a high priority for the nova team anyway | |
| 09:18:51 | gibi | yeah, we need to stabilize our gate | |
| 09:19:08 | gibi | and thanks I might eventually ask for help requeuing | |
| 09:28:56 | sean-k-mooney | frickler: gibi os we can remove a class of error form the ceph jobs it we just move them to useing nested virt (kvm) | |
| 09:29:12 | sean-k-mooney | nova doesnt do that but nuetorn and some other teams do | |
| 09:30:03 | sean-k-mooney | for exampel the kernel panics on volume detach | |
| 09:31:00 | sean-k-mooney | the ssh timeout might go away if we enabel the creation fo the tap devices by os-vif. i need to try enableing that in the gate now that the neutron bits are all in place | |
| 09:32:23 | sean-k-mooney | but yes we take some time to adress more of those failures. the issue nova has is those failure are often not form nova/hypervior comonents | |
| 09:32:34 | sean-k-mooney | i.e. the etc/ceph contianer download fialures | |
| 09:33:16 | sean-k-mooney | *etcd | |
| 09:33:46 | sean-k-mooney | i think we can improve ^ in devstack by stragicly retrying some fo the downloads | |
| 09:33:51 | bauzas | do we have at least an idea on why the failures are more recurrent those days ? those look like usual suspects | |
| 09:34:36 | bauzas | (speaking of the volume detach kernel panics and the ssh timeouts - the latter already being WA'd a lot) | |
| 09:34:47 | sean-k-mooney | bauzas: i dont think any of the failure gibi has been hitting were new. they all looked like the "ususal suspects" | |
| 09:35:13 | sean-k-mooney | so my guess is load make it manifest more | |
| 09:35:29 | sean-k-mooney | the kernel panics dont happen if we use either kvm or the uec images | |
| 09:35:52 | sean-k-mooney | both of which change (speedup) the timing of the kernel loading in cirrus | |
| 09:36:06 | gibi | yeah load does not help | |
| 09:36:18 | gibi | I just saw a guest bootop where every step took a second | |
| 09:36:20 | gibi | SEND | |
| 09:36:30 | gibi | [ 4.017889] clocksource: Switched to clocksource tsc | |
| 09:36:30 | gibi | [ 5.258926] ACPI: \_SB_.LNKB: Enabled at IRQ 10 | |
| 09:36:30 | gibi | [ 5.553620] Freeing initrd memory: 8864K | |
| 09:36:30 | gibi | [ 6.239466] ACPI: \_SB_.LNKC: Enabled at IRQ 11 | |
| 09:36:30 | gibi | [ 7.080979] ACPI: \_SB_.LNKD: Enabled at IRQ 11 | |
| 09:36:33 | gibi | [ 7.966979] ACPI: \_SB_.LNKA: Enabled at IRQ 10 | |
| 09:36:35 | gibi | [ 8.834095] Free page reporting enabled | |