| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-02-26 | |||
| 09:33:16 | sean-k-mooney | *etcd | |
| 09:33:46 | sean-k-mooney | i think we can improve ^ in devstack by stragicly retrying some fo the downloads | |
| 09:33:51 | bauzas | do we have at least an idea on why the failures are more recurrent those days ? those look like usual suspects | |
| 09:34:36 | bauzas | (speaking of the volume detach kernel panics and the ssh timeouts - the latter already being WA'd a lot) | |
| 09:34:47 | sean-k-mooney | bauzas: i dont think any of the failure gibi has been hitting were new. they all looked like the "ususal suspects" | |
| 09:35:13 | sean-k-mooney | so my guess is load make it manifest more | |
| 09:35:29 | sean-k-mooney | the kernel panics dont happen if we use either kvm or the uec images | |
| 09:35:52 | sean-k-mooney | both of which change (speedup) the timing of the kernel loading in cirrus | |
| 09:36:06 | gibi | yeah load does not help | |
| 09:36:18 | gibi | I just saw a guest bootop where every step took a second | |
| 09:36:20 | gibi | SEND | |
| 09:36:30 | gibi | [ 4.017889] clocksource: Switched to clocksource tsc | |
| 09:36:30 | gibi | [ 5.258926] ACPI: \_SB_.LNKB: Enabled at IRQ 10 | |
| 09:36:30 | gibi | [ 5.553620] Freeing initrd memory: 8864K | |
| 09:36:30 | gibi | [ 6.239466] ACPI: \_SB_.LNKC: Enabled at IRQ 11 | |
| 09:36:30 | gibi | [ 7.080979] ACPI: \_SB_.LNKD: Enabled at IRQ 11 | |
| 09:36:33 | gibi | [ 7.966979] ACPI: \_SB_.LNKA: Enabled at IRQ 10 | |
| 09:36:35 | gibi | [ 8.834095] Free page reporting enabled | |
| 09:36:38 | gibi | [ 8.839828] Serial: 8250/16550 driver, 32 p | |
| 09:36:52 | sean-k-mooney | ya... | |
| 09:36:54 | gibi | this is not sustainable with our current ssh timeouts | |
| 09:38:20 | sean-k-mooney | so the ceph jobs i think we shoudl consdier movign to the nested virt lables to compenstate for the ceph cpu load or look at the perfromence of them in general because those are the ones that hit those time out most often | |
| 09:38:29 | sean-k-mooney | and the kernel panincs most often | |
| 09:38:57 | sean-k-mooney | it would be intersting to compare the boot time form one of those jobs vs the non ceph jobs | |
| 09:39:06 | sean-k-mooney | or look at the load average for the jobs | |
| 09:40:04 | gibi | eah | |
| 09:40:06 | gibi | yeah | |
| 09:40:18 | sean-k-mooney | gibi: i take it the eventlet changes didnt merge last night | |
| 09:51:12 | sean-k-mooney | amazed that any of the jobs pass? :) | |
| 09:51:46 | bauzas | gibi: were you able to see the vmstat values of the host at the time the VM was booting that fast ? | |
| 09:53:20 | bauzas | sean-k-mooney: looks like we play a Claw machine game | |
| 11:07:27 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Remove unnecessary action method prefix https://review.opendev.org/c/openstack/nova/+/958863 | |
| 11:07:28 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Deprecate os-volumes_boot API https://review.opendev.org/c/openstack/nova/+/958864 | |
| 11:09:54 | opendevreview | Stephen Finucane proposed openstack/nova master: conf: Deprecate AggregateImagePropertiesIsolation opts https://review.opendev.org/c/openstack/nova/+/783396 | |
| 11:09:55 | opendevreview | Stephen Finucane proposed openstack/nova master: conf: Deprecate AggregateImagePropertiesIsolation opts https://review.opendev.org/c/openstack/nova/+/783396 | |
| 11:28:13 | opendevreview | Stephen Finucane proposed openstack/nova master: docs: Update REST API version history for 2.101 https://review.opendev.org/c/openstack/nova/+/978002 | |
| 11:28:13 | opendevreview | Stephen Finucane proposed openstack/nova master: tests: Fix typo https://review.opendev.org/c/openstack/nova/+/958855 | |
| 11:28:14 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Add runtime check for query additionalProperties https://review.opendev.org/c/openstack/nova/+/958856 | |
| 11:28:14 | opendevreview | Stephen Finucane proposed openstack/nova master: tests: Add missing test coverage https://review.opendev.org/c/openstack/nova/+/958857 | |
| 12:14:31 | opendevreview | sean mooney proposed openstack/nova master: ci: Define nested nodesets, audit KVM/UEC, move nova-next https://review.opendev.org/c/openstack/nova/+/978066 | |
| 12:21:49 | gibi | sean-k-mooney: it didn't. I still rechecking :/ | |
| 12:22:27 | gibi | bauzas: I did not dig deeper. I see overall slowness so it is probably infra overload due to FF churn | |
| 12:22:28 | sean-k-mooney | ^ might help but it has not run in ci yet to see if its functionatl | |
| 12:22:41 | sean-k-mooney | lol nope | |
| 12:22:51 | sean-k-mooney | it has zuul sytax errors | |
| 12:22:56 | sean-k-mooney | which i thinkis a typo | |
| 12:34:53 | opendevreview | sean mooney proposed openstack/nova master: ci: Define nested nodesets, audit KVM/UEC, move nova-next https://review.opendev.org/c/openstack/nova/+/978066 | |
| 12:37:14 | sean-k-mooney | was missign a 1 in compute1 | |
| 12:47:52 | opendevreview | sean mooney proposed openstack/nova master: ci: Define nested nodesets, audit KVM/UEC, move nova-next https://review.opendev.org/c/openstack/nova/+/978066 | |
| 13:00:37 | opendevreview | sean mooney proposed openstack/nova master: ci: Define nested nodesets, audit KVM/UEC, move nova-next https://review.opendev.org/c/openstack/nova/+/978066 | |
| 13:00:52 | sean-k-mooney | i shoudl not try to update patches while also in an irc meeting... | |
| 13:25:53 | gmaan | oh, i hoped at least a few will be merged overnight... | |
| 13:26:10 | opendevreview | Merged openstack/nova master: api: Fix issue with instance usage audit log schema https://review.opendev.org/c/openstack/nova/+/977967 | |
| 13:26:27 | opendevreview | Merged openstack/nova master: api: Add runtime check for general additionalProperties https://review.opendev.org/c/openstack/nova/+/958858 | |
| 13:26:44 | opendevreview | Merged openstack/nova stable/2024.2: api: Fix validators for hw:cpu_max_* extra specs https://review.opendev.org/c/openstack/nova/+/957744 | |
| 13:26:57 | sean-k-mooney | so it si posibel to merge things ^ | |
| 13:27:06 | sean-k-mooney | but that seriese is apprently cursed | |
| 13:27:19 | DominikDanelski[m] | sean-k-mooney: In https://review.opendev.org/c/openstack/nova/+/971176/comment/e96c41d6_d900c91a/ you mentioned 'repurposing' the spec. Is it done by essentially making a modified copy that targets a newer release and has no direct link to the previous one, or should git move the accepted one from Ussuri, since it wasn't implemented then, modify it, and add an appropriate line to redirects file? | |
| 13:28:07 | sean-k-mooney | DominikDanelski[m]: yes you copy the spec to the new folder and update the history section at the end to note its re proposed | |
| 13:28:47 | sean-k-mooney | we do not move them as we want to keep every revsion fo the spec that were prevsouly approved so we can clearly see any deltas | |
| 13:29:23 | DominikDanelski[m] | All right, thank you. | |
| 13:49:26 | opendevreview | Merged openstack/nova master: libvirt: Remove unnecessary arg https://review.opendev.org/c/openstack/nova/+/881817 | |
| 14:16:07 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Remove unnecessary action method prefix https://review.opendev.org/c/openstack/nova/+/958863 | |
| 14:16:08 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Deprecate os-volumes_boot API https://review.opendev.org/c/openstack/nova/+/958864 | |
| 14:19:32 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Remove unnecessary action method prefix https://review.opendev.org/c/openstack/nova/+/958863 | |
| 14:19:32 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Deprecate os-volumes_boot API https://review.opendev.org/c/openstack/nova/+/958864 | |
| 14:35:17 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Remove unnecessary action method prefix https://review.opendev.org/c/openstack/nova/+/958863 | |
| 14:35:17 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Deprecate os-volumes_boot API https://review.opendev.org/c/openstack/nova/+/958864 | |
| 14:42:41 | gibi | dansmith: o/ I'm +2 on the making the volume attach async in https://review.opendev.org/c/openstack/nova/+/971068 | |
| 14:43:08 | dansmith | ack | |
| 14:50:08 | cardoe | sean-k-mooney: around? Wanted to follow up about what I posted here. | |
| 14:51:13 | Uggla | gibi, bauzas, any chance we can merge https://review.opendev.org/q/topic:%22bp/generalize-sev-code%22 to my mind it is ok except maybe the latest one. | |
| 14:53:21 | bauzas | Uggla: I was trying to look at it | |
| 14:54:15 | Uggla | dansmith it would be super cool if you can check https://review.opendev.org/c/openstack/nova/+/971068 | |
| 14:54:16 | gibi | Uggla: I have couple of +2s in the series | |
| 14:54:26 | gibi | but not all the way top | |
| 14:54:45 | gibi | so I'm not sure if I will have the energy today to go up to the top | |
| 14:54:56 | gibi | but I will try | |
| 14:55:11 | Uggla | gibi, i guess all comments were addressed and that looks in a good shape. | |
| 14:55:16 | dansmith | Uggla: isn't that the one gibi just poked me on? | |
| 14:55:42 | Uggla | dansmith yes, just re-enforcing the msg. :) | |
| 14:59:54 | Uggla | sean-k-mooney, gmaan super cool for openapi. Thanks for the effort. 👍 | |
| 15:06:17 | cardoe | Uggla: I'm hoping we can get https://review.opendev.org/c/openstack/nova/+/972601 into the next release. | |
| 15:06:40 | cardoe | I'm trying to follow up with sean-k-mooney cause his -1 reason is literally what this patch is correcting. | |
| 15:07:33 | cardoe | For context, KVM/libvirt driver does not use the prepare_networks_before_block_device_mapping() while the ironic driver does. Which is what is the root cause of this. | |
| 15:07:51 | bauzas | cardoe: maybe you don't know but today is FeatureFreeze, we prioritize for the moment reviews for blueprints | |
| 15:08:07 | bauzas | cardoe: during the next two weeks, we can then look at bugfixes like this one | |
| 15:08:14 | Uggla | cardoe, that's a bug right. So it is not impacted by FF. | |
| 15:08:46 | cardoe | bauzas: okay I didn't know. | |
| 15:09:05 | cardoe | I was just trying to provide context. | |
| 15:09:16 | bauzas | cardoe: no worries, you can usually find the release cadence there : https://releases.openstack.org/gazpacho/schedule.html | |
| 15:09:51 | Uggla | cardoe, so as bauzas explains we could merge it in the following weeks. Assuming all comments will be addressed | |
| 15:09:53 | bauzas | cardoe: again, no worries, my only concern is about context switching, like for the moment I'm trying to make sure we could accept a SEV-ES usecase | |
| 15:10:16 | bauzas | cardoe: but please ping us next week around Tuesday, we could look at the bug | |
| 15:10:21 | cardoe | Not asking you folks to context switch right now. Just putting info in here so that you can follow up async. | |
| 15:10:27 | cardoe | But I can ping next Tuesday. | |
| 15:10:38 | bauzas | thanks | |
| 15:14:39 | Uggla | cardoe also good to put it in our dev plan: https://etherpad.opendev.org/p/nova-2026.1-status so we should not missed it. (note: I did it for you) | |
| 15:15:04 | sean-k-mooney | cardoe: i was confused by the wroding you had in relese note | |
| 15:15:12 | sean-k-mooney | ill take anouther look | |