| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-08-13 | |||
| 22:04:22 | gmaan | Aug 13 19:11:12.515025 npb4f32edc7a464 nova-compute[92418]: INFO oslo_service.backend._threading.service [None req-e086152e-d33a-491b-8053-3d300bc9a11e None None] Graceful shutdown timeout exceeded, instantaneous exiting | |
| 22:04:22 | gmaan | 2, when timeout it seems privsep-helper still running | |
| 22:04:24 | gmaan | Aug 13 19:11:12.544562 npb4f32edc7a464 systemd[1]: devstack@n-cpu.service: Unit process 94748 (privsep-helper) remains running after unit stopped. | |
| 22:04:24 | gmaan | Aug 13 19:11:12.544388 npb4f32edc7a464 systemd[1]: devstack@n-cpu.service: Unit process 93426 (privsep-helper) remains running after unit stopped. | |
| 22:04:26 | gmaan | Aug 13 19:11:12.545602 npb4f32edc7a464 systemd[1]: devstack@n-cpu.service: Consumed 12.751s CPU time, 277.1M memory peak, 0B memory swap peak. | |
| 22:04:42 | gmaan | this seems something new | |
| 22:06:07 | gmaan | thats does not seems the issue, i can see "Unit process 82937 (privsep-helper) remains running after unit stopped." in successful case also | |
| 22:11:26 | gmaan | not sure why service status is updated in DB, because after 180 sec of graceful shutdown, test wait for another 180 sec timeout for serviec to be inactive | |
| 22:17:01 | gmaan | i see, I think its the status update interval time report_interval and service_down_time is racing with test timeout. report_interval = 120 service_down_time = 720 | |
| 22:17:02 | gmaan | https://zuul.opendev.org/t/openstack/build/065f1dfcb6604871b71c9b7f84392275/log/controller/logs/screen-n-cpu.txt#101-115 | |
| 22:17:46 | gmaan | i think i need to update the test script to consider the report_interval and service_down_time for service status update timeout | |
| 22:28:21 | gmaan | melwitt: reported the bug, I will propose fix sometime tomorrow otherwise next week https://bugs.launchpad.net/nova/+bug/2163448 | |
| 22:44:02 | melwitt | thanks for doing that gmaan | |
| #openstack-nova - 2026-08-14 | |||
| 08:29:56 | opendevreview | Rico Lin proposed openstack/nova master: compute: add skip_inventory_in_use_errors opt https://review.opendev.org/c/openstack/nova/+/985750 | |
| 09:12:26 | opendevreview | Bhavana proposed openstack/nova-specs master: Add spec for MIG-aware PCI Host Weigher https://review.opendev.org/c/openstack/nova-specs/+/998753 | |
| 10:08:33 | opendevreview | Kamil Sambor proposed openstack/nova master: Fix pickling error that forces oslo.service to fall back to fork https://review.opendev.org/c/openstack/nova/+/1000956 | |
| 12:09:44 | opendevreview | Pierre Libeau proposed openstack/nova master: libvirt: Add reproducer for unstable vNIC PCI addresses on hard reboot https://review.opendev.org/c/openstack/nova/+/1000961 | |
| 12:09:45 | opendevreview | Pierre Libeau proposed openstack/nova master: libvirt: Preserve vNIC guest PCI address on domain XML regeneration https://review.opendev.org/c/openstack/nova/+/1000963 | |
| 12:09:45 | opendevreview | Pierre Libeau proposed openstack/nova master: libvirt: Render guest interface PCI address in config https://review.opendev.org/c/openstack/nova/+/1000962 | |
| 12:43:40 | plibeau | hi — opinions wanted on https://bugs.launchpad.net/nova/+bug/2163417 : vNIC guest PCI slots get reshuffled on hard reboot (multi-NIC appliances see port1/port2 swapped). nova never pins the VIF guest PCI address, libvirt renumbers from XML order; MAC is the only invariant. reproduced on clean master | |
| 12:45:19 | plibeau | I have a WIP opt-in (3 commits above) [libvirt] preserve_vif_pci_addresses re-pinning by MAC from old_guest (hard reboot/power_on/resume only). is that the right direction, or should the slot be persisted per-VIF? or is "use device role tagging" the intended answer? | |
| 12:47:34 | gibi | plibeau: I think we never promised the guest that the pci slot ordering is stable. Tagging devices are the way to map ports to devices afaik | |
| 12:53:59 | plibeau | yes and linux by default manage that correctly because it use mac address and not pci slot to manage NIC but it's not the case of some and specially appliance. What do you think to try to improve that for this type of use case? | |
| 12:56:21 | gibi | plibeau: I mean you can tag the nic and block devices in the nova api when add it to your server, then the VM can read the tagging from the metadata | |
| 12:56:28 | gibi | that should be OS independent | |
| 12:57:43 | amorin | yes, but some OS are not able to do that, the proposition from plibeau is to try to keep the order as much as possible to avoid order change when it's possible | |
| 13:01:26 | gibi | amorin: you mean there are OSes that are not able to read the metadata API? | |
| 13:01:47 | gibi | or read the metadat from the config drive? | |
| 13:02:00 | amorin | yup | |
| 13:02:10 | amorin | some of our customers complain about this | |
| 13:02:49 | gibi | is it possible that they could read but don't want to? Sorry I'm suspicious here as it is a very basic fuctionality to read a file from an attached disk or do http request | |
| 13:03:40 | amorin | I totally agree with you :) They usually rely on providers to give them images and they seems not able to do so in an easy way | |
| 13:03:54 | amorin | plibeau will have more info, I dont remember the name of the image? | |
| 13:05:11 | gibi | amorin: you can propose a feature (or ask this on the IRC meeting) to start providing a stable order of PCI devices to guests to get the oppinion of others but this was rejected before with the above argument | |
| 13:05:52 | plibeau | Fortinet is the name :) but not only and the customer can't fix also because it's appliance, he don't have root access on it | |
| 13:07:48 | gibi | can this be brougth up to Frotinet then? to integrate with cloud VMs by reading the metadata? :) | |
| 13:09:58 | gibi | I feel like either Fortinet supports to run in OpenStack or isn't :) | |
| 13:15:21 | plibeau | yes, fix on Fortinet side is a solution but we need to fix also probably other case. I think yes Fortinet is supported on openstack I see some documentation how to deploy it above openstack | |
| 13:38:29 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Unify the thread pool executors into a central factory https://review.opendev.org/c/openstack/nova/+/998571 | |
| 13:58:28 | gmaan | gibi: ^^ added the test coverage | |
| 13:59:31 | gibi | gmaan: thanks. I will check it soon | |
| 14:01:33 | gmaan | thanks | |
| 14:01:39 | gibi | +2 | |
| 14:01:42 | gibi | thanks | |
| 14:01:50 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Shutdown thread pool executors during graceful shutdown https://review.opendev.org/c/openstack/nova/+/1000211 | |
| 14:01:56 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Add doc and release notes for the graceful shutdown task tracking https://review.opendev.org/c/openstack/nova/+/997141 | |
| 14:15:18 | opendevreview | Tobias Urdin proposed openstack/nova master: Catch ValueError from from_image_ref in rescue https://review.opendev.org/c/openstack/nova/+/1000969 | |
| 14:48:47 | gmaan | gibi: and this is graceful shutdown part to shutdown executors which i separated from the original change https://review.opendev.org/c/openstack/nova/+/1000211 | |
| 14:59:06 | opendevreview | Balazs Gibizer proposed openstack/nova master: Stabilize functional test for slow instance.save https://review.opendev.org/c/openstack/nova/+/999121 | |
| 14:59:07 | opendevreview | Balazs Gibizer proposed openstack/nova master: Fix test_fill_vifs_migration under native threading https://review.opendev.org/c/openstack/nova/+/1000780 | |
| 14:59:07 | opendevreview | Balazs Gibizer proposed openstack/nova master: Make functional-py313-threading voting https://review.opendev.org/c/openstack/nova/+/997216 | |
| 14:59:08 | opendevreview | Balazs Gibizer proposed openstack/nova master: Stabilize func test waiting for VM ERROR state https://review.opendev.org/c/openstack/nova/+/1000971 | |
| 15:00:11 | opendevreview | Balazs Gibizer proposed openstack/nova master: Stabilize functional test for slow instance.save https://review.opendev.org/c/openstack/nova/+/999121 | |
| 15:00:11 | opendevreview | Balazs Gibizer proposed openstack/nova master: Stabilize func test waiting for VM ERROR state https://review.opendev.org/c/openstack/nova/+/998822 | |
| 15:00:12 | opendevreview | Balazs Gibizer proposed openstack/nova master: Fix test_fill_vifs_migration under native threading https://review.opendev.org/c/openstack/nova/+/1000780 | |
| 15:00:12 | opendevreview | Balazs Gibizer proposed openstack/nova master: Make functional-py313-threading voting https://review.opendev.org/c/openstack/nova/+/997216 | |
| 15:05:08 | gibi | gmaan: thank for the suggestion in the stabilization series ^^ I applied it | |
| 15:05:25 | gmaan | gibi: ack will check today | |
| 15:49:48 | opendevreview | Pierre Libeau proposed openstack/nova master: libvirt: Add reproducer for unstable vNIC PCI addresses on hard reboot https://review.opendev.org/c/openstack/nova/+/1000961 | |
| 15:49:49 | opendevreview | Pierre Libeau proposed openstack/nova master: libvirt: Preserve vNIC guest PCI address on domain XML regeneration https://review.opendev.org/c/openstack/nova/+/1000963 | |
| 15:49:49 | opendevreview | Pierre Libeau proposed openstack/nova master: libvirt: Render guest interface PCI address in config https://review.opendev.org/c/openstack/nova/+/1000962 | |
| 16:31:20 | opendevreview | Merged openstack/os-traits master: Add HW_NVME capability traits https://review.opendev.org/c/openstack/os-traits/+/999921 | |
| 17:48:37 | opendevreview | Merged openstack/nova master: Task tracking mechanism for graceful shutdown https://review.opendev.org/c/openstack/nova/+/996299 | |
| 17:58:05 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Shutdown thread pool executors during graceful shutdown https://review.opendev.org/c/openstack/nova/+/1000211 | |
| 18:29:29 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Unify the thread pool executors into a central factory https://review.opendev.org/c/openstack/nova/+/998571 | |
| 18:30:15 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Shutdown thread pool executors during graceful shutdown https://review.opendev.org/c/openstack/nova/+/1000211 | |
| #openstack-nova - 2026-08-15 | |||
| 02:59:02 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Use devstack set service shutdown related config options https://review.opendev.org/c/openstack/nova/+/1001017 | |
| 03:19:25 | gmaan | it seems nova-grenade-multinode job is failing more frequently on test_shelve_unshelve_server, any known issue? i reported bug for that but did not debug it yet https://bugs.launchpad.net/nova/+bug/2163552 | |
| 23:50:00 | opendevreview | Merged openstack/nova master: Track and log the main RPC server in-progress task https://review.opendev.org/c/openstack/nova/+/996583 | |
| #openstack-nova - 2026-08-17 | |||
| 11:26:03 | opendevreview | chandan kumar proposed openstack/nova master: Report OWNER_NOVA trait on resource providers https://review.opendev.org/c/openstack/nova/+/994299 | |
| 11:34:02 | opendevreview | chandan kumar proposed openstack/nova master: Report OWNER_NOVA trait on resource providers https://review.opendev.org/c/openstack/nova/+/994299 | |
| 11:56:34 | opendevreview | Joan Gilabert proposed openstack/nova master: Add libvirt driver MDEV ARQ support for Cyborg https://review.opendev.org/c/openstack/nova/+/995840 | |
| 13:40:12 | opendevreview | Pierre Libeau proposed openstack/nova master: libvirt: Render guest interface PCI address in config https://review.opendev.org/c/openstack/nova/+/1000962 | |
| 13:40:12 | opendevreview | Pierre Libeau proposed openstack/nova master: libvirt: Add reproducer for unstable vNIC PCI addresses on hard reboot https://review.opendev.org/c/openstack/nova/+/1000961 | |
| 13:40:13 | opendevreview | Pierre Libeau proposed openstack/nova master: libvirt: Preserve vNIC guest PCI address on domain XML regeneration https://review.opendev.org/c/openstack/nova/+/1000963 | |
| 14:51:46 | melwitt | patch? | |
| 14:51:46 | melwitt | I saw that somebody posted a new comment on this bug https://bugs.launchpad.net/nova/+bug/2076614 saying they hit the problem while upgrading 2025.1 => 2026.1 ... there was a fix proposed but it's in merge conflict now and sean-k-mooney had a comment on it with some follow up steps, I wondered are those steps still valid now (> 1 year later) https://review.opendev.org/c/openstack/nova/+/948328 or is it that we just need to rebase the | |
| 14:54:36 | sean-k-mooney | melwitt: triging to remeber context | |
| 14:55:02 | sean-k-mooney | but reading my old comment i was just noting we could start droping older migrations in a followup for things older then caracal | |
| 14:55:27 | sean-k-mooney | i think as wriggen the archive/pruge woudl still be required before doign the online migrations | |
| 14:56:21 | sean-k-mooney | and it might still be worth addign an upgrade release note | |
| 14:56:30 | sean-k-mooney | but i think just rebaseing that commit would be fine | |
| 14:57:05 | sean-k-mooney | melwitt: i.e. if you wanted to progress that i would be ok wiht just resolving the merge conflict and proceeding with it | |
| 14:57:46 | sean-k-mooney | with that said this only affect upgrades form unsupproted releases to my knowldage | |
| 14:57:52 | sean-k-mooney | so it has not been a high priorty | |
| 14:58:29 | sean-k-mooney | oh ok the latest comment confirm its an issue for current release | |
| 14:58:35 | melwitt | sean-k-mooney: ok cool thank you. yeah ... it caught my eye bc they are upgrading _to_ a supported release though so it set off my alarm bells | |
| 14:58:36 | sean-k-mooney | when upgrading after caracal so ya | |
| 14:59:15 | sean-k-mooney | right it was the other way around its only an issue after caracal but also not sure why we dont see this grenade | |
| 15:01:45 | melwitt | it happens due to the "fake" marker instance for that fill_virtual_interface_list online migration right ... so the grenade test wouldn't have that old of a migration artifact in its database | |
| 15:02:27 | melwitt | it would have to be a long lived deployment that has been upgraded over the years, IIUC | |
| 15:11:46 | sean-k-mooney | i guess i tought that marker was adde by the older migration | |
| 15:11:48 | sean-k-mooney | but ok | |
| 15:14:06 | sean-k-mooney | its still a valid bug so it makes sesne to fix it on master in anycase | |
| 15:15:24 | melwitt | yes I think that is correct that the marker was added by the older migration (rocky => stein). I was trying to say that CI wouldn't show it bc CI would not ever have that old of a migration remnant in it | |
| 15:16:52 | melwitt | maybe i'm misunderstanding it, I guess you are saying the old migration still running today. I have to think about it more | |
| 15:17:45 | sean-k-mooney | i dont recall if alembic/sql aclahme in devstack is instcating the db directly from the model or if it apples the base schema and then runs the migrations | |
| 15:18:09 | sean-k-mooney | i thinink sqlalchemy-migrate may have done the latter | |
| 15:18:19 | sean-k-mooney | as i recall seeign devstack run migraitons locally in teh past | |
| 15:18:33 | sean-k-mooney | but i dont knwo if we skip that step in a normal install with alembic | |