| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-08-13 | |||
| 15:45:46 | dansmith | I'm on 3c23390cc8 | |
| 15:45:53 | dansmith | wait a damn second | |
| 15:46:41 | dansmith | lol, okay I did a local clone for a new working tree and I think it doesn't know what actual origin/master is so I'm based on something way old.. wtf | |
| 15:47:40 | gmaan | dansmith: yeah that's too old Jan 26 | |
| 15:47:53 | dansmith | yeah, stupid local shallow clone | |
| 15:47:56 | dansmith | sorry for the noise | |
| 15:50:03 | gmaan | np! I used to have same issue in my multiple local copies and forgot to rebase before change and finds it during git review which adds more work. I tried to stick to have 1 or 2 local working repo now :) | |
| 15:52:07 | dansmith | yeah I almost always work out of one repo, but for something destructive I'm using another.. I did a local clone and so it's (long stale) master branch becomes origin/master in the new clone so I thought I was rebased on master but wasn't | |
| 15:52:37 | dansmith | which is stupid rookie mistake, I just rarely do this so I was on autopilot I guess | |
| 17:04:26 | gmaan | gibi: i updated the thread pool executor refactoring, i replied to this comment, I think there are more chance that different executors creation can happen in parallel compare to before this change where most of them were created during init https://review.opendev.org/c/openstack/nova/+/998571/comment/0f4554fd_8780afaf/ | |
| 17:05:44 | gmaan | that is why I think a another lock is worth here to make them parallel but i do not have much performance data/impact so we can leave that with single lock (serial creation) and can change if any issue we see. let me know and i can update it accordingly | |
| 18:18:35 | opendevreview | melanie witt proposed openstack/nova master: Reproducer for bug 2016173 https://review.opendev.org/c/openstack/nova/+/946222 | |
| 18:18:36 | opendevreview | melanie witt proposed openstack/nova master: Call volume detach rollback API if detach fails https://review.opendev.org/c/openstack/nova/+/880399 | |
| 19:40:09 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Shutdown thread pool executors during graceful shutdown https://review.opendev.org/c/openstack/nova/+/1000211 | |
| 19:55:11 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Add doc and release notes for the graceful shutdown task tracking https://review.opendev.org/c/openstack/nova/+/997141 | |
| 20:03:31 | gmaan | melwitt: i saw you +w on the 2nd change but this base change is also ready, just in case you missed it, but if you are still reviewing that, please ignore this ping :) https://review.opendev.org/c/openstack/nova/+/996299/ | |
| 20:04:23 | melwitt | gmaan: yes, sorry I am mid review still. thanks for checking :) | |
| 20:04:35 | gmaan | melwitt: ohk, thanks | |
| 20:08:55 | gmaan | dansmith: there are other two changes are also ready to complete the graceful shutdown series, https://review.opendev.org/c/openstack/nova/+/1000211 and doc/releasenotes on top of it. | |
| 20:09:52 | gmaan | dansmith: Those are dependent on thread pool executor refactoring which makes the executors shudown easy to handle during graceful shutdown, you are welcome to review the same but just to clarify that it is part of eventlet-removal and not directly to graceful shutdown BP https://review.opendev.org/c/openstack/nova/+/998571 | |
| 20:10:24 | gmaan | that is why i set the different topic for those even they are in same series | |
| 20:10:38 | dansmith | gmaan: yeah I thought gibi had a -1 on the next patch.. is that resolved? | |
| 20:11:14 | gmaan | dansmith: I updated it but gibi have not checked the updated one yet | |
| 20:11:54 | gmaan | i resolved his feedback except one which i replied and waiting for him to check before we need any update there | |
| 20:18:15 | dansmith | ack | |
| 21:37:02 | melwitt | seems like the nova-graceful-shutdown job fails intermittently "Timed out waiting for compute service on npb4f32edc7a464 to be inactive (current: failed)" https://zuul.opendev.org/t/openstack/build/065f1dfcb6604871b71c9b7f84392275 but I don't see any gate-failure bugs open about it | |
| 21:39:21 | gmaan | melwitt: i have one but that is race between some test stopping service and other need it but this looks new to me, checking | |
| 22:03:40 | gmaan | it seems two issue here 1. graceful shutdown is timeout andit seems lot of periodic tasks were running, this is solved in task tracking system where we stop running any new periodic tasks during shutdown | |
| 22:04:22 | gmaan | Aug 13 19:11:12.544060 npb4f32edc7a464 systemd[1]: devstack@n-cpu.service: Failed with result 'exit-code'. | |
| 22:04:22 | gmaan | Aug 13 19:11:12.543995 npb4f32edc7a464 systemd[1]: devstack@n-cpu.service: Main process exited, code=exited, status=1/FAILURE | |
| 22:04:22 | gmaan | Aug 13 19:11:12.515025 npb4f32edc7a464 nova-compute[92418]: INFO oslo_service.backend._threading.service [None req-e086152e-d33a-491b-8053-3d300bc9a11e None None] Graceful shutdown timeout exceeded, instantaneous exiting | |
| 22:04:22 | gmaan | 2, when timeout it seems privsep-helper still running | |
| 22:04:24 | gmaan | Aug 13 19:11:12.544562 npb4f32edc7a464 systemd[1]: devstack@n-cpu.service: Unit process 94748 (privsep-helper) remains running after unit stopped. | |
| 22:04:24 | gmaan | Aug 13 19:11:12.544388 npb4f32edc7a464 systemd[1]: devstack@n-cpu.service: Unit process 93426 (privsep-helper) remains running after unit stopped. | |
| 22:04:26 | gmaan | Aug 13 19:11:12.545602 npb4f32edc7a464 systemd[1]: devstack@n-cpu.service: Consumed 12.751s CPU time, 277.1M memory peak, 0B memory swap peak. | |
| 22:04:42 | gmaan | this seems something new | |
| 22:06:07 | gmaan | thats does not seems the issue, i can see "Unit process 82937 (privsep-helper) remains running after unit stopped." in successful case also | |
| 22:11:26 | gmaan | not sure why service status is updated in DB, because after 180 sec of graceful shutdown, test wait for another 180 sec timeout for serviec to be inactive | |
| 22:17:01 | gmaan | i see, I think its the status update interval time report_interval and service_down_time is racing with test timeout. report_interval = 120 service_down_time = 720 | |
| 22:17:02 | gmaan | https://zuul.opendev.org/t/openstack/build/065f1dfcb6604871b71c9b7f84392275/log/controller/logs/screen-n-cpu.txt#101-115 | |
| 22:17:46 | gmaan | i think i need to update the test script to consider the report_interval and service_down_time for service status update timeout | |
| 22:28:21 | gmaan | melwitt: reported the bug, I will propose fix sometime tomorrow otherwise next week https://bugs.launchpad.net/nova/+bug/2163448 | |
| 22:44:02 | melwitt | thanks for doing that gmaan | |
| #openstack-nova - 2026-08-14 | |||
| 08:29:56 | opendevreview | Rico Lin proposed openstack/nova master: compute: add skip_inventory_in_use_errors opt https://review.opendev.org/c/openstack/nova/+/985750 | |
| 09:12:26 | opendevreview | Bhavana proposed openstack/nova-specs master: Add spec for MIG-aware PCI Host Weigher https://review.opendev.org/c/openstack/nova-specs/+/998753 | |
| 10:08:33 | opendevreview | Kamil Sambor proposed openstack/nova master: Fix pickling error that forces oslo.service to fall back to fork https://review.opendev.org/c/openstack/nova/+/1000956 | |
| 12:09:44 | opendevreview | Pierre Libeau proposed openstack/nova master: libvirt: Add reproducer for unstable vNIC PCI addresses on hard reboot https://review.opendev.org/c/openstack/nova/+/1000961 | |
| 12:09:45 | opendevreview | Pierre Libeau proposed openstack/nova master: libvirt: Preserve vNIC guest PCI address on domain XML regeneration https://review.opendev.org/c/openstack/nova/+/1000963 | |
| 12:09:45 | opendevreview | Pierre Libeau proposed openstack/nova master: libvirt: Render guest interface PCI address in config https://review.opendev.org/c/openstack/nova/+/1000962 | |
| 12:43:40 | plibeau | hi — opinions wanted on https://bugs.launchpad.net/nova/+bug/2163417 : vNIC guest PCI slots get reshuffled on hard reboot (multi-NIC appliances see port1/port2 swapped). nova never pins the VIF guest PCI address, libvirt renumbers from XML order; MAC is the only invariant. reproduced on clean master | |
| 12:45:19 | plibeau | I have a WIP opt-in (3 commits above) [libvirt] preserve_vif_pci_addresses re-pinning by MAC from old_guest (hard reboot/power_on/resume only). is that the right direction, or should the slot be persisted per-VIF? or is "use device role tagging" the intended answer? | |
| 12:47:34 | gibi | plibeau: I think we never promised the guest that the pci slot ordering is stable. Tagging devices are the way to map ports to devices afaik | |
| 12:53:59 | plibeau | yes and linux by default manage that correctly because it use mac address and not pci slot to manage NIC but it's not the case of some and specially appliance. What do you think to try to improve that for this type of use case? | |
| 12:56:21 | gibi | plibeau: I mean you can tag the nic and block devices in the nova api when add it to your server, then the VM can read the tagging from the metadata | |
| 12:56:28 | gibi | that should be OS independent | |
| 12:57:43 | amorin | yes, but some OS are not able to do that, the proposition from plibeau is to try to keep the order as much as possible to avoid order change when it's possible | |
| 13:01:26 | gibi | amorin: you mean there are OSes that are not able to read the metadata API? | |
| 13:01:47 | gibi | or read the metadat from the config drive? | |
| 13:02:00 | amorin | yup | |
| 13:02:10 | amorin | some of our customers complain about this | |
| 13:02:49 | gibi | is it possible that they could read but don't want to? Sorry I'm suspicious here as it is a very basic fuctionality to read a file from an attached disk or do http request | |
| 13:03:40 | amorin | I totally agree with you :) They usually rely on providers to give them images and they seems not able to do so in an easy way | |
| 13:03:54 | amorin | plibeau will have more info, I dont remember the name of the image? | |
| 13:05:11 | gibi | amorin: you can propose a feature (or ask this on the IRC meeting) to start providing a stable order of PCI devices to guests to get the oppinion of others but this was rejected before with the above argument | |
| 13:05:52 | plibeau | Fortinet is the name :) but not only and the customer can't fix also because it's appliance, he don't have root access on it | |
| 13:07:48 | gibi | can this be brougth up to Frotinet then? to integrate with cloud VMs by reading the metadata? :) | |
| 13:09:58 | gibi | I feel like either Fortinet supports to run in OpenStack or isn't :) | |
| 13:15:21 | plibeau | yes, fix on Fortinet side is a solution but we need to fix also probably other case. I think yes Fortinet is supported on openstack I see some documentation how to deploy it above openstack | |
| 13:38:29 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Unify the thread pool executors into a central factory https://review.opendev.org/c/openstack/nova/+/998571 | |
| 13:58:28 | gmaan | gibi: ^^ added the test coverage | |
| 13:59:31 | gibi | gmaan: thanks. I will check it soon | |
| 14:01:33 | gmaan | thanks | |
| 14:01:39 | gibi | +2 | |
| 14:01:42 | gibi | thanks | |
| 14:01:50 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Shutdown thread pool executors during graceful shutdown https://review.opendev.org/c/openstack/nova/+/1000211 | |
| 14:01:56 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Add doc and release notes for the graceful shutdown task tracking https://review.opendev.org/c/openstack/nova/+/997141 | |
| 14:15:18 | opendevreview | Tobias Urdin proposed openstack/nova master: Catch ValueError from from_image_ref in rescue https://review.opendev.org/c/openstack/nova/+/1000969 | |
| 14:48:47 | gmaan | gibi: and this is graceful shutdown part to shutdown executors which i separated from the original change https://review.opendev.org/c/openstack/nova/+/1000211 | |
| 14:59:06 | opendevreview | Balazs Gibizer proposed openstack/nova master: Stabilize functional test for slow instance.save https://review.opendev.org/c/openstack/nova/+/999121 | |
| 14:59:07 | opendevreview | Balazs Gibizer proposed openstack/nova master: Fix test_fill_vifs_migration under native threading https://review.opendev.org/c/openstack/nova/+/1000780 | |
| 14:59:07 | opendevreview | Balazs Gibizer proposed openstack/nova master: Make functional-py313-threading voting https://review.opendev.org/c/openstack/nova/+/997216 | |
| 14:59:08 | opendevreview | Balazs Gibizer proposed openstack/nova master: Stabilize func test waiting for VM ERROR state https://review.opendev.org/c/openstack/nova/+/1000971 | |
| 15:00:11 | opendevreview | Balazs Gibizer proposed openstack/nova master: Stabilize functional test for slow instance.save https://review.opendev.org/c/openstack/nova/+/999121 | |
| 15:00:11 | opendevreview | Balazs Gibizer proposed openstack/nova master: Stabilize func test waiting for VM ERROR state https://review.opendev.org/c/openstack/nova/+/998822 | |
| 15:00:12 | opendevreview | Balazs Gibizer proposed openstack/nova master: Fix test_fill_vifs_migration under native threading https://review.opendev.org/c/openstack/nova/+/1000780 | |
| 15:00:12 | opendevreview | Balazs Gibizer proposed openstack/nova master: Make functional-py313-threading voting https://review.opendev.org/c/openstack/nova/+/997216 | |
| 15:05:08 | gibi | gmaan: thank for the suggestion in the stabilization series ^^ I applied it | |
| 15:05:25 | gmaan | gibi: ack will check today | |
| 15:49:48 | opendevreview | Pierre Libeau proposed openstack/nova master: libvirt: Add reproducer for unstable vNIC PCI addresses on hard reboot https://review.opendev.org/c/openstack/nova/+/1000961 | |
| 15:49:49 | opendevreview | Pierre Libeau proposed openstack/nova master: libvirt: Preserve vNIC guest PCI address on domain XML regeneration https://review.opendev.org/c/openstack/nova/+/1000963 | |
| 15:49:49 | opendevreview | Pierre Libeau proposed openstack/nova master: libvirt: Render guest interface PCI address in config https://review.opendev.org/c/openstack/nova/+/1000962 | |
| 16:31:20 | opendevreview | Merged openstack/os-traits master: Add HW_NVME capability traits https://review.opendev.org/c/openstack/os-traits/+/999921 | |
| 17:48:37 | opendevreview | Merged openstack/nova master: Task tracking mechanism for graceful shutdown https://review.opendev.org/c/openstack/nova/+/996299 | |
| 17:58:05 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Shutdown thread pool executors during graceful shutdown https://review.opendev.org/c/openstack/nova/+/1000211 | |
| 18:29:29 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Unify the thread pool executors into a central factory https://review.opendev.org/c/openstack/nova/+/998571 | |
| 18:30:15 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Shutdown thread pool executors during graceful shutdown https://review.opendev.org/c/openstack/nova/+/1000211 | |
| #openstack-nova - 2026-08-15 | |||
| 02:59:02 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Use devstack set service shutdown related config options https://review.opendev.org/c/openstack/nova/+/1001017 | |
| 03:19:25 | gmaan | it seems nova-grenade-multinode job is failing more frequently on test_shelve_unshelve_server, any known issue? i reported bug for that but did not debug it yet https://bugs.launchpad.net/nova/+bug/2163552 | |
| 23:50:00 | opendevreview | Merged openstack/nova master: Track and log the main RPC server in-progress task https://review.opendev.org/c/openstack/nova/+/996583 | |
| #openstack-nova - 2026-08-17 | |||
| 11:26:03 | opendevreview | chandan kumar proposed openstack/nova master: Report OWNER_NOVA trait on resource providers https://review.opendev.org/c/openstack/nova/+/994299 | |