| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-08-13 | |||
| 20:10:38 | dansmith | gmaan: yeah I thought gibi had a -1 on the next patch.. is that resolved? | |
| 20:11:14 | gmaan | dansmith: I updated it but gibi have not checked the updated one yet | |
| 20:11:54 | gmaan | i resolved his feedback except one which i replied and waiting for him to check before we need any update there | |
| 20:18:15 | dansmith | ack | |
| 21:37:02 | melwitt | seems like the nova-graceful-shutdown job fails intermittently "Timed out waiting for compute service on npb4f32edc7a464 to be inactive (current: failed)" https://zuul.opendev.org/t/openstack/build/065f1dfcb6604871b71c9b7f84392275 but I don't see any gate-failure bugs open about it | |
| 21:39:21 | gmaan | melwitt: i have one but that is race between some test stopping service and other need it but this looks new to me, checking | |
| 22:03:40 | gmaan | it seems two issue here 1. graceful shutdown is timeout andit seems lot of periodic tasks were running, this is solved in task tracking system where we stop running any new periodic tasks during shutdown | |
| 22:04:22 | gmaan | Aug 13 19:11:12.544060 npb4f32edc7a464 systemd[1]: devstack@n-cpu.service: Failed with result 'exit-code'. | |
| 22:04:22 | gmaan | Aug 13 19:11:12.543995 npb4f32edc7a464 systemd[1]: devstack@n-cpu.service: Main process exited, code=exited, status=1/FAILURE | |
| 22:04:22 | gmaan | Aug 13 19:11:12.515025 npb4f32edc7a464 nova-compute[92418]: INFO oslo_service.backend._threading.service [None req-e086152e-d33a-491b-8053-3d300bc9a11e None None] Graceful shutdown timeout exceeded, instantaneous exiting | |
| 22:04:22 | gmaan | 2, when timeout it seems privsep-helper still running | |
| 22:04:24 | gmaan | Aug 13 19:11:12.544562 npb4f32edc7a464 systemd[1]: devstack@n-cpu.service: Unit process 94748 (privsep-helper) remains running after unit stopped. | |
| 22:04:24 | gmaan | Aug 13 19:11:12.544388 npb4f32edc7a464 systemd[1]: devstack@n-cpu.service: Unit process 93426 (privsep-helper) remains running after unit stopped. | |
| 22:04:26 | gmaan | Aug 13 19:11:12.545602 npb4f32edc7a464 systemd[1]: devstack@n-cpu.service: Consumed 12.751s CPU time, 277.1M memory peak, 0B memory swap peak. | |
| 22:04:42 | gmaan | this seems something new | |
| 22:06:07 | gmaan | thats does not seems the issue, i can see "Unit process 82937 (privsep-helper) remains running after unit stopped." in successful case also | |
| 22:11:26 | gmaan | not sure why service status is updated in DB, because after 180 sec of graceful shutdown, test wait for another 180 sec timeout for serviec to be inactive | |
| 22:17:01 | gmaan | i see, I think its the status update interval time report_interval and service_down_time is racing with test timeout. report_interval = 120 service_down_time = 720 | |
| 22:17:02 | gmaan | https://zuul.opendev.org/t/openstack/build/065f1dfcb6604871b71c9b7f84392275/log/controller/logs/screen-n-cpu.txt#101-115 | |
| 22:17:46 | gmaan | i think i need to update the test script to consider the report_interval and service_down_time for service status update timeout | |
| 22:28:21 | gmaan | melwitt: reported the bug, I will propose fix sometime tomorrow otherwise next week https://bugs.launchpad.net/nova/+bug/2163448 | |
| 22:44:02 | melwitt | thanks for doing that gmaan | |
| #openstack-nova - 2026-08-14 | |||
| 08:29:56 | opendevreview | Rico Lin proposed openstack/nova master: compute: add skip_inventory_in_use_errors opt https://review.opendev.org/c/openstack/nova/+/985750 | |
| 09:12:26 | opendevreview | Bhavana proposed openstack/nova-specs master: Add spec for MIG-aware PCI Host Weigher https://review.opendev.org/c/openstack/nova-specs/+/998753 | |
| 10:08:33 | opendevreview | Kamil Sambor proposed openstack/nova master: Fix pickling error that forces oslo.service to fall back to fork https://review.opendev.org/c/openstack/nova/+/1000956 | |
| 12:09:44 | opendevreview | Pierre Libeau proposed openstack/nova master: libvirt: Add reproducer for unstable vNIC PCI addresses on hard reboot https://review.opendev.org/c/openstack/nova/+/1000961 | |
| 12:09:45 | opendevreview | Pierre Libeau proposed openstack/nova master: libvirt: Preserve vNIC guest PCI address on domain XML regeneration https://review.opendev.org/c/openstack/nova/+/1000963 | |
| 12:09:45 | opendevreview | Pierre Libeau proposed openstack/nova master: libvirt: Render guest interface PCI address in config https://review.opendev.org/c/openstack/nova/+/1000962 | |
| 12:43:40 | plibeau | hi — opinions wanted on https://bugs.launchpad.net/nova/+bug/2163417 : vNIC guest PCI slots get reshuffled on hard reboot (multi-NIC appliances see port1/port2 swapped). nova never pins the VIF guest PCI address, libvirt renumbers from XML order; MAC is the only invariant. reproduced on clean master | |
| 12:45:19 | plibeau | I have a WIP opt-in (3 commits above) [libvirt] preserve_vif_pci_addresses re-pinning by MAC from old_guest (hard reboot/power_on/resume only). is that the right direction, or should the slot be persisted per-VIF? or is "use device role tagging" the intended answer? | |
| 12:47:34 | gibi | plibeau: I think we never promised the guest that the pci slot ordering is stable. Tagging devices are the way to map ports to devices afaik | |
| 12:53:59 | plibeau | yes and linux by default manage that correctly because it use mac address and not pci slot to manage NIC but it's not the case of some and specially appliance. What do you think to try to improve that for this type of use case? | |
| 12:56:21 | gibi | plibeau: I mean you can tag the nic and block devices in the nova api when add it to your server, then the VM can read the tagging from the metadata | |
| 12:56:28 | gibi | that should be OS independent | |
| 12:57:43 | amorin | yes, but some OS are not able to do that, the proposition from plibeau is to try to keep the order as much as possible to avoid order change when it's possible | |
| 13:01:26 | gibi | amorin: you mean there are OSes that are not able to read the metadata API? | |
| 13:01:47 | gibi | or read the metadat from the config drive? | |
| 13:02:00 | amorin | yup | |
| 13:02:10 | amorin | some of our customers complain about this | |
| 13:02:49 | gibi | is it possible that they could read but don't want to? Sorry I'm suspicious here as it is a very basic fuctionality to read a file from an attached disk or do http request | |
| 13:03:40 | amorin | I totally agree with you :) They usually rely on providers to give them images and they seems not able to do so in an easy way | |
| 13:03:54 | amorin | plibeau will have more info, I dont remember the name of the image? | |
| 13:05:11 | gibi | amorin: you can propose a feature (or ask this on the IRC meeting) to start providing a stable order of PCI devices to guests to get the oppinion of others but this was rejected before with the above argument | |
| 13:05:52 | plibeau | Fortinet is the name :) but not only and the customer can't fix also because it's appliance, he don't have root access on it | |
| 13:07:48 | gibi | can this be brougth up to Frotinet then? to integrate with cloud VMs by reading the metadata? :) | |
| 13:09:58 | gibi | I feel like either Fortinet supports to run in OpenStack or isn't :) | |
| 13:15:21 | plibeau | yes, fix on Fortinet side is a solution but we need to fix also probably other case. I think yes Fortinet is supported on openstack I see some documentation how to deploy it above openstack | |
| 13:38:29 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Unify the thread pool executors into a central factory https://review.opendev.org/c/openstack/nova/+/998571 | |
| 13:58:28 | gmaan | gibi: ^^ added the test coverage | |
| 13:59:31 | gibi | gmaan: thanks. I will check it soon | |
| 14:01:33 | gmaan | thanks | |
| 14:01:39 | gibi | +2 | |
| 14:01:42 | gibi | thanks | |
| 14:01:50 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Shutdown thread pool executors during graceful shutdown https://review.opendev.org/c/openstack/nova/+/1000211 | |
| 14:01:56 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Add doc and release notes for the graceful shutdown task tracking https://review.opendev.org/c/openstack/nova/+/997141 | |
| 14:15:18 | opendevreview | Tobias Urdin proposed openstack/nova master: Catch ValueError from from_image_ref in rescue https://review.opendev.org/c/openstack/nova/+/1000969 | |
| 14:48:47 | gmaan | gibi: and this is graceful shutdown part to shutdown executors which i separated from the original change https://review.opendev.org/c/openstack/nova/+/1000211 | |
| 14:59:06 | opendevreview | Balazs Gibizer proposed openstack/nova master: Stabilize functional test for slow instance.save https://review.opendev.org/c/openstack/nova/+/999121 | |
| 14:59:07 | opendevreview | Balazs Gibizer proposed openstack/nova master: Fix test_fill_vifs_migration under native threading https://review.opendev.org/c/openstack/nova/+/1000780 | |
| 14:59:07 | opendevreview | Balazs Gibizer proposed openstack/nova master: Make functional-py313-threading voting https://review.opendev.org/c/openstack/nova/+/997216 | |
| 14:59:08 | opendevreview | Balazs Gibizer proposed openstack/nova master: Stabilize func test waiting for VM ERROR state https://review.opendev.org/c/openstack/nova/+/1000971 | |
| 15:00:11 | opendevreview | Balazs Gibizer proposed openstack/nova master: Stabilize functional test for slow instance.save https://review.opendev.org/c/openstack/nova/+/999121 | |
| 15:00:11 | opendevreview | Balazs Gibizer proposed openstack/nova master: Stabilize func test waiting for VM ERROR state https://review.opendev.org/c/openstack/nova/+/998822 | |
| 15:00:12 | opendevreview | Balazs Gibizer proposed openstack/nova master: Fix test_fill_vifs_migration under native threading https://review.opendev.org/c/openstack/nova/+/1000780 | |
| 15:00:12 | opendevreview | Balazs Gibizer proposed openstack/nova master: Make functional-py313-threading voting https://review.opendev.org/c/openstack/nova/+/997216 | |
| 15:05:08 | gibi | gmaan: thank for the suggestion in the stabilization series ^^ I applied it | |
| 15:05:25 | gmaan | gibi: ack will check today | |
| 15:49:48 | opendevreview | Pierre Libeau proposed openstack/nova master: libvirt: Add reproducer for unstable vNIC PCI addresses on hard reboot https://review.opendev.org/c/openstack/nova/+/1000961 | |
| 15:49:49 | opendevreview | Pierre Libeau proposed openstack/nova master: libvirt: Preserve vNIC guest PCI address on domain XML regeneration https://review.opendev.org/c/openstack/nova/+/1000963 | |
| 15:49:49 | opendevreview | Pierre Libeau proposed openstack/nova master: libvirt: Render guest interface PCI address in config https://review.opendev.org/c/openstack/nova/+/1000962 | |
| 16:31:20 | opendevreview | Merged openstack/os-traits master: Add HW_NVME capability traits https://review.opendev.org/c/openstack/os-traits/+/999921 | |
| 17:48:37 | opendevreview | Merged openstack/nova master: Task tracking mechanism for graceful shutdown https://review.opendev.org/c/openstack/nova/+/996299 | |
| 17:58:05 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Shutdown thread pool executors during graceful shutdown https://review.opendev.org/c/openstack/nova/+/1000211 | |
| 18:29:29 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Unify the thread pool executors into a central factory https://review.opendev.org/c/openstack/nova/+/998571 | |
| 18:30:15 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Shutdown thread pool executors during graceful shutdown https://review.opendev.org/c/openstack/nova/+/1000211 | |
| #openstack-nova - 2026-08-15 | |||
| 02:59:02 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Use devstack set service shutdown related config options https://review.opendev.org/c/openstack/nova/+/1001017 | |
| 03:19:25 | gmaan | it seems nova-grenade-multinode job is failing more frequently on test_shelve_unshelve_server, any known issue? i reported bug for that but did not debug it yet https://bugs.launchpad.net/nova/+bug/2163552 | |
| 23:50:00 | opendevreview | Merged openstack/nova master: Track and log the main RPC server in-progress task https://review.opendev.org/c/openstack/nova/+/996583 | |
| #openstack-nova - 2026-08-17 | |||
| 11:26:03 | opendevreview | chandan kumar proposed openstack/nova master: Report OWNER_NOVA trait on resource providers https://review.opendev.org/c/openstack/nova/+/994299 | |
| 11:34:02 | opendevreview | chandan kumar proposed openstack/nova master: Report OWNER_NOVA trait on resource providers https://review.opendev.org/c/openstack/nova/+/994299 | |
| 11:56:34 | opendevreview | Joan Gilabert proposed openstack/nova master: Add libvirt driver MDEV ARQ support for Cyborg https://review.opendev.org/c/openstack/nova/+/995840 | |
| 13:40:12 | opendevreview | Pierre Libeau proposed openstack/nova master: libvirt: Render guest interface PCI address in config https://review.opendev.org/c/openstack/nova/+/1000962 | |
| 13:40:12 | opendevreview | Pierre Libeau proposed openstack/nova master: libvirt: Add reproducer for unstable vNIC PCI addresses on hard reboot https://review.opendev.org/c/openstack/nova/+/1000961 | |
| 13:40:13 | opendevreview | Pierre Libeau proposed openstack/nova master: libvirt: Preserve vNIC guest PCI address on domain XML regeneration https://review.opendev.org/c/openstack/nova/+/1000963 | |
| 14:51:46 | melwitt | patch? | |
| 14:51:46 | melwitt | I saw that somebody posted a new comment on this bug https://bugs.launchpad.net/nova/+bug/2076614 saying they hit the problem while upgrading 2025.1 => 2026.1 ... there was a fix proposed but it's in merge conflict now and sean-k-mooney had a comment on it with some follow up steps, I wondered are those steps still valid now (> 1 year later) https://review.opendev.org/c/openstack/nova/+/948328 or is it that we just need to rebase the | |
| 14:54:36 | sean-k-mooney | melwitt: triging to remeber context | |
| 14:55:02 | sean-k-mooney | but reading my old comment i was just noting we could start droping older migrations in a followup for things older then caracal | |
| 14:55:27 | sean-k-mooney | i think as wriggen the archive/pruge woudl still be required before doign the online migrations | |
| 14:56:21 | sean-k-mooney | and it might still be worth addign an upgrade release note | |
| 14:56:30 | sean-k-mooney | but i think just rebaseing that commit would be fine | |
| 14:57:05 | sean-k-mooney | melwitt: i.e. if you wanted to progress that i would be ok wiht just resolving the merge conflict and proceeding with it | |
| 14:57:46 | sean-k-mooney | with that said this only affect upgrades form unsupproted releases to my knowldage | |
| 14:57:52 | sean-k-mooney | so it has not been a high priorty | |
| 14:58:29 | sean-k-mooney | oh ok the latest comment confirm its an issue for current release | |
| 14:58:35 | melwitt | sean-k-mooney: ok cool thank you. yeah ... it caught my eye bc they are upgrading _to_ a supported release though so it set off my alarm bells | |
| 14:58:36 | sean-k-mooney | when upgrading after caracal so ya | |
| 14:59:15 | sean-k-mooney | right it was the other way around its only an issue after caracal but also not sure why we dont see this grenade | |
| 15:01:45 | melwitt | it happens due to the "fake" marker instance for that fill_virtual_interface_list online migration right ... so the grenade test wouldn't have that old of a migration artifact in its database | |
| 15:02:27 | melwitt | it would have to be a long lived deployment that has been upgraded over the years, IIUC | |