| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-02-25 | |||
| 19:56:40 | sean-k-mooney | but we agree to make incremental progress so thats ok | |
| 19:57:24 | gmaan | k, let me rebase and fix the other comments you have | |
| 20:22:52 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Use 2nd RPC server in compute operations https://review.opendev.org/c/openstack/nova/+/975588 | |
| 20:32:58 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Use 2nd RPC server in compute operations https://review.opendev.org/c/openstack/nova/+/975588 | |
| 20:36:08 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Prepare resize/cold migration for graceful shutdown https://review.opendev.org/c/openstack/nova/+/977182 | |
| 20:46:25 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Add more test for graceful shutdown https://review.opendev.org/c/openstack/nova/+/977405 | |
| 21:00:51 | opendevreview | Merged openstack/nova master: api: Add response body schemas for servers APIs (5/6) https://review.opendev.org/c/openstack/nova/+/956239 | |
| 21:19:06 | gmaan | bauzas: sean-k-mooney gibi it seems, there is no merge conflict between eventlet and graceful shutdown changes, but anyways I have rebased latter one on eventlet. | |
| 21:19:12 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Add operator document for graceful shutdown https://review.opendev.org/c/openstack/nova/+/977590 | |
| 21:19:18 | gmaan | sean-k-mooney: updated ^^ doc change also | |
| 21:33:13 | sean-k-mooney | gmaan: im +2 on all of your changes thanks for the update | |
| 21:34:32 | sean-k-mooney | gmaan: gibi is unfutently not have great luck with the ci gods at the moment but maybe you can recheck the eventlet patches later in NA time when the ci might be less flaky | |
| 21:45:00 | gmaan | sean-k-mooney: thanks, I will keep eyes on CI | |
| 22:02:15 | cardoe | sean-k-mooney: ping https://review.opendev.org/c/openstack/nova/+/972601 | |
| 22:02:47 | cardoe | I just posted a reply to you but I wanted to discuss it. | |
| 22:03:49 | cardoe | I believe you're misunderstanding what my change did. My change is matching the behavior of KVM/libvirt with the ironic driver. And making it so that the full backtrace is unnecessary to the end user to troubleshoot. | |
| 22:09:25 | cardoe | There's half a dozen bug reports / issues today where if I do X with KVM/libvirt, I get a clear error message explaining why it didn't work. But when I use the ironic driver via nova, the error message I get as a user is either wrong or generic and the only way to work out what happened is to read the full backtrace. Inside of that full backtrace is the same one that's being delivered back to the user in the KVM/libvirt case. | |
| 22:32:07 | gmaan | sean-k-mooney: thanks | |
| 22:32:33 | gmaan | gibi: if you can re-review it https://review.opendev.org/c/openstack/nova/+/975588 this first one | |
| 22:49:36 | opendevreview | melanie witt proposed openstack/nova master: TPM: bump service version to enable live migration https://review.opendev.org/c/openstack/nova/+/975724 | |
| 22:49:37 | opendevreview | melanie witt proposed openstack/nova master: TPM: test live migration between hosts with different security https://review.opendev.org/c/openstack/nova/+/952629 | |
| 22:49:37 | opendevreview | melanie witt proposed openstack/nova master: TPM: support live migration of `deployment` secret security https://review.opendev.org/c/openstack/nova/+/925771 | |
| 22:49:38 | opendevreview | melanie witt proposed openstack/nova master: TPM: enable conversion of secret security modes via resize https://review.opendev.org/c/openstack/nova/+/962052 | |
| 22:49:38 | opendevreview | melanie witt proposed openstack/nova master: TPM: add late check for supported TPM secret security https://review.opendev.org/c/openstack/nova/+/956975 | |
| 22:49:39 | opendevreview | melanie witt proposed openstack/nova master: DNM vtpm tempest https://review.opendev.org/c/openstack/nova/+/957477 | |
| 23:54:42 | cardoe | gmaan: neat. So we'll be able to shutdown gracefully and let it continue to manage things. | |
| 23:59:03 | gmaan | cardoe: yes, we introduced the 2 RPC server in compute so that compute stop accepting the new request and finish the in-progress one. in this phase-1, services wait for configuration timeout which will be replaced by task tracking in next part. | |
| #openstack-nova - 2026-02-26 | |||
| 00:05:12 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Add operator document for graceful shutdown https://review.opendev.org/c/openstack/nova/+/977590 | |
| 01:33:05 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Add operator document for graceful shutdown https://review.opendev.org/c/openstack/nova/+/977590 | |
| 01:35:56 | gmaan | gate is in its worst situation | |
| 02:38:41 | gmaan | gibi: sean-k-mooney and this doc also need re-review, fixed pep8 spell things https://review.opendev.org/c/openstack/nova/+/977590 | |
| 03:00:11 | opendevreview | Merged openstack/nova master: api: Add response body schemas for servers APIs (6/6) https://review.opendev.org/c/openstack/nova/+/956240 | |
| 08:50:26 | opendevreview | Merged openstack/nova master: api: Add response body schemas for server shares APIs https://review.opendev.org/c/openstack/nova/+/956266 | |
| 08:50:39 | opendevreview | Merged openstack/nova master: tests: Invert validation check https://review.opendev.org/c/openstack/nova/+/956241 | |
| 08:59:22 | gibi | I'm envy about the folks who can land patches through the gate. The eventlet series now stuck for more than 24 hours behind random failures. I feel bad about grinding rechecks but I will do it anyhow :/ | |
| 09:15:37 | frickler | let me know if you need help like re-enqueueing changes directly back into gate or even force-merging. once you're sufficiently sure the failures are unrelated IMO it doesn't make sense to keep doing rechecks and using up CI resources | |
| 09:16:11 | frickler | and I say this knowing that getting tests more stable is a high priority for the nova team anyway | |
| 09:18:51 | gibi | yeah, we need to stabilize our gate | |
| 09:19:08 | gibi | and thanks I might eventually ask for help requeuing | |
| 09:28:56 | sean-k-mooney | frickler: gibi os we can remove a class of error form the ceph jobs it we just move them to useing nested virt (kvm) | |
| 09:29:12 | sean-k-mooney | nova doesnt do that but nuetorn and some other teams do | |
| 09:30:03 | sean-k-mooney | for exampel the kernel panics on volume detach | |
| 09:31:00 | sean-k-mooney | the ssh timeout might go away if we enabel the creation fo the tap devices by os-vif. i need to try enableing that in the gate now that the neutron bits are all in place | |
| 09:32:23 | sean-k-mooney | but yes we take some time to adress more of those failures. the issue nova has is those failure are often not form nova/hypervior comonents | |
| 09:32:34 | sean-k-mooney | i.e. the etc/ceph contianer download fialures | |
| 09:33:16 | sean-k-mooney | *etcd | |
| 09:33:46 | sean-k-mooney | i think we can improve ^ in devstack by stragicly retrying some fo the downloads | |
| 09:33:51 | bauzas | do we have at least an idea on why the failures are more recurrent those days ? those look like usual suspects | |
| 09:34:36 | bauzas | (speaking of the volume detach kernel panics and the ssh timeouts - the latter already being WA'd a lot) | |
| 09:34:47 | sean-k-mooney | bauzas: i dont think any of the failure gibi has been hitting were new. they all looked like the "ususal suspects" | |
| 09:35:13 | sean-k-mooney | so my guess is load make it manifest more | |
| 09:35:29 | sean-k-mooney | the kernel panics dont happen if we use either kvm or the uec images | |
| 09:35:52 | sean-k-mooney | both of which change (speedup) the timing of the kernel loading in cirrus | |
| 09:36:06 | gibi | yeah load does not help | |
| 09:36:18 | gibi | I just saw a guest bootop where every step took a second | |
| 09:36:20 | gibi | SEND | |
| 09:36:30 | gibi | [ 7.080979] ACPI: \_SB_.LNKD: Enabled at IRQ 11 | |
| 09:36:30 | gibi | [ 6.239466] ACPI: \_SB_.LNKC: Enabled at IRQ 11 | |
| 09:36:30 | gibi | [ 5.553620] Freeing initrd memory: 8864K | |
| 09:36:30 | gibi | [ 5.258926] ACPI: \_SB_.LNKB: Enabled at IRQ 10 | |
| 09:36:30 | gibi | [ 4.017889] clocksource: Switched to clocksource tsc | |
| 09:36:33 | gibi | [ 7.966979] ACPI: \_SB_.LNKA: Enabled at IRQ 10 | |
| 09:36:35 | gibi | [ 8.834095] Free page reporting enabled | |
| 09:36:38 | gibi | [ 8.839828] Serial: 8250/16550 driver, 32 p | |
| 09:36:52 | sean-k-mooney | ya... | |
| 09:36:54 | gibi | this is not sustainable with our current ssh timeouts | |
| 09:38:20 | sean-k-mooney | so the ceph jobs i think we shoudl consdier movign to the nested virt lables to compenstate for the ceph cpu load or look at the perfromence of them in general because those are the ones that hit those time out most often | |
| 09:38:29 | sean-k-mooney | and the kernel panincs most often | |
| 09:38:57 | sean-k-mooney | it would be intersting to compare the boot time form one of those jobs vs the non ceph jobs | |
| 09:39:06 | sean-k-mooney | or look at the load average for the jobs | |
| 09:40:04 | gibi | eah | |
| 09:40:06 | gibi | yeah | |
| 09:40:18 | sean-k-mooney | gibi: i take it the eventlet changes didnt merge last night | |
| 09:51:12 | sean-k-mooney | amazed that any of the jobs pass? :) | |
| 09:51:46 | bauzas | gibi: were you able to see the vmstat values of the host at the time the VM was booting that fast ? | |
| 09:53:20 | bauzas | sean-k-mooney: looks like we play a Claw machine game | |
| 11:07:27 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Remove unnecessary action method prefix https://review.opendev.org/c/openstack/nova/+/958863 | |
| 11:07:28 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Deprecate os-volumes_boot API https://review.opendev.org/c/openstack/nova/+/958864 | |
| 11:09:54 | opendevreview | Stephen Finucane proposed openstack/nova master: conf: Deprecate AggregateImagePropertiesIsolation opts https://review.opendev.org/c/openstack/nova/+/783396 | |
| 11:09:55 | opendevreview | Stephen Finucane proposed openstack/nova master: conf: Deprecate AggregateImagePropertiesIsolation opts https://review.opendev.org/c/openstack/nova/+/783396 | |
| 11:28:13 | opendevreview | Stephen Finucane proposed openstack/nova master: tests: Fix typo https://review.opendev.org/c/openstack/nova/+/958855 | |
| 11:28:13 | opendevreview | Stephen Finucane proposed openstack/nova master: docs: Update REST API version history for 2.101 https://review.opendev.org/c/openstack/nova/+/978002 | |
| 11:28:14 | opendevreview | Stephen Finucane proposed openstack/nova master: tests: Add missing test coverage https://review.opendev.org/c/openstack/nova/+/958857 | |
| 11:28:14 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Add runtime check for query additionalProperties https://review.opendev.org/c/openstack/nova/+/958856 | |
| 12:14:31 | opendevreview | sean mooney proposed openstack/nova master: ci: Define nested nodesets, audit KVM/UEC, move nova-next https://review.opendev.org/c/openstack/nova/+/978066 | |
| 12:21:49 | gibi | sean-k-mooney: it didn't. I still rechecking :/ | |
| 12:22:27 | gibi | bauzas: I did not dig deeper. I see overall slowness so it is probably infra overload due to FF churn | |
| 12:22:28 | sean-k-mooney | ^ might help but it has not run in ci yet to see if its functionatl | |
| 12:22:41 | sean-k-mooney | lol nope | |
| 12:22:51 | sean-k-mooney | it has zuul sytax errors | |
| 12:22:56 | sean-k-mooney | which i thinkis a typo | |
| 12:34:53 | opendevreview | sean mooney proposed openstack/nova master: ci: Define nested nodesets, audit KVM/UEC, move nova-next https://review.opendev.org/c/openstack/nova/+/978066 | |
| 12:37:14 | sean-k-mooney | was missign a 1 in compute1 | |
| 12:47:52 | opendevreview | sean mooney proposed openstack/nova master: ci: Define nested nodesets, audit KVM/UEC, move nova-next https://review.opendev.org/c/openstack/nova/+/978066 | |
| 13:00:37 | opendevreview | sean mooney proposed openstack/nova master: ci: Define nested nodesets, audit KVM/UEC, move nova-next https://review.opendev.org/c/openstack/nova/+/978066 | |
| 13:00:52 | sean-k-mooney | i shoudl not try to update patches while also in an irc meeting... | |
| 13:25:53 | gmaan | oh, i hoped at least a few will be merged overnight... | |
| 13:26:10 | opendevreview | Merged openstack/nova master: api: Fix issue with instance usage audit log schema https://review.opendev.org/c/openstack/nova/+/977967 | |
| 13:26:27 | opendevreview | Merged openstack/nova master: api: Add runtime check for general additionalProperties https://review.opendev.org/c/openstack/nova/+/958858 | |
| 13:26:44 | opendevreview | Merged openstack/nova stable/2024.2: api: Fix validators for hw:cpu_max_* extra specs https://review.opendev.org/c/openstack/nova/+/957744 | |