| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-08-24 | |||
| 16:52:54 | gmaan | gibi: the shutdown executors change is ready for review https://review.opendev.org/c/openstack/nova/+/1000211 | |
| 16:53:08 | gmaan | and the last change of gracefult shutdown series, doc/releasenotes https://review.opendev.org/c/openstack/nova/+/997141/11 | |
| 16:53:28 | gmaan | gibi: will you be ok to review that doc change also ^^ (its not big) | |
| 16:55:07 | gmaan | if not then also fine, I think melwitt (^^) can check it as she reviewed the main change for tracking things | |
| 16:55:08 | opendevreview | Merged openstack/nova master: Add libvirt driver MDEV ARQ support for Cyborg https://review.opendev.org/c/openstack/nova/+/995840 | |
| 16:55:18 | gmaan | these two left for this series | |
| 16:55:42 | gibi | I will review them but not today | |
| 16:55:47 | gibi | is it OK tomorrow? | |
| 16:55:57 | gmaan | yeah sure thanks. | |
| 16:56:08 | gibi | cool | |
| 16:59:55 | melwitt | yes I can look | |
| 17:02:37 | gmaan | melwitt: thanks | |
| 17:32:43 | opendevreview | sean mooney proposed openstack/nova master: doc: Add guide for NVIDIA vGPU with SR-IOV variant drivers https://review.opendev.org/c/openstack/nova/+/994160 | |
| 17:32:43 | opendevreview | sean mooney proposed openstack/nova master: libvirt: Support Cyborg PCI ARQ managed mode https://review.opendev.org/c/openstack/nova/+/994579 | |
| 17:58:42 | gmaan | seems like one more test unstable, reported new bug https://bugs.launchpad.net/nova/+bug/2164931 | |
| 18:00:40 | dansmith | man.. wonder if something else is going on.. I would have expected more instability from eventlet stuff that all seemed to go smoothly.. | |
| 18:00:52 | opendevreview | melanie witt proposed openstack/nova master: Switch Cyborg client from keystoneauth1 to openstacksdk https://review.opendev.org/c/openstack/nova/+/1002007 | |
| 18:09:55 | melwitt | yeah, the gate has been pretty painful | |
| 18:53:23 | JayF | we had a bug filed, against ironic, by the openstacksdks team about a fail that had gotten to a reasonable % of time. I dug in, it was an old race that we were more reliably losing due to slower I/O | |
| 18:53:43 | JayF | so I would believe we're seeing some performance characteristics change and impact runs | |
| 18:54:33 | sean-k-mooney | so load near FF causling races that normally dont matrialise to show up? | |
| 18:55:08 | JayF | sean-k-mooney: I suggest we just replay the chat logs from every previous 6 months ago going back a decade instead of re-saying all the things at FF busy time ;) | |
| 18:55:26 | sean-k-mooney | :) | |
| 18:55:32 | JayF | [standard cycle-end CI complaint #24987] | |
| 18:55:56 | JayF | (honestly thanks for mentioning that, I had "forgotten" about that tendency and it puts the last brick in place about "why now?" for that CI fix) | |
| 18:57:05 | sean-k-mooney | nova is still doting the Is and crosing the Ts on eventlet -> threading by default move in ci | |
| 18:57:25 | sean-k-mooney | so its less of a clear picture fo is io slow beacuse of ci or threading | |
| 19:22:26 | opendevreview | melanie witt proposed openstack/nova master: Switch Cyborg client from keystoneauth1 to openstacksdk https://review.opendev.org/c/openstack/nova/+/1002007 | |
| 20:30:33 | opendevreview | Clif Houck proposed openstack/nova master: perf(ironic): eliminate O(N²) ProviderTree deepcopy at startup https://review.opendev.org/c/openstack/nova/+/980676 | |
| 20:30:37 | opendevreview | Clif Houck proposed openstack/nova master: Parallelize per-node resource updates https://review.opendev.org/c/openstack/nova/+/980679 | |
| 20:54:23 | opendevreview | melanie witt proposed openstack/nova master: Switch Cyborg client from keystoneauth1 to openstacksdk https://review.opendev.org/c/openstack/nova/+/1002007 | |
| 21:37:16 | opendevreview | Clif Houck proposed openstack/nova master: Parallelize per-node resource updates https://review.opendev.org/c/openstack/nova/+/980679 | |
| 21:39:06 | clif | I think the per-node resource updates patch is in a good spot now, all review comments should be addressed | |
| 23:29:05 | opendevreview | Merged openstack/nova master: Reproducer for bug 2016173 https://review.opendev.org/c/openstack/nova/+/946222 | |
| 23:48:50 | melwitt | sean-k-mooney, stephenfin: fyi https://review.opendev.org/c/openstack/nova/+/1002007 is finally happy with the cyborg-tempest-py3 job. needed a couple of get_endpoint_data() things for openstacksdk | |
| #openstack-nova - 2026-08-25 | |||
| 00:10:34 | opendevreview | Merged openstack/nova master: doc: Add releases information for SEV-ES and SEV-SNP https://review.opendev.org/c/openstack/nova/+/1002164 | |
| 00:10:47 | opendevreview | Merged openstack/nova master: weights: add regression test for JSON log TypeError (bug 2028518) https://review.opendev.org/c/openstack/nova/+/989029 | |
| 00:11:59 | sean-k-mooney | melwitt: ya i had to special case get_endpoint_data in my patch as weel https://review.opendev.org/c/openstack/nova/+/994579/4/nova/accelerator/cyborg.py#148 | |
| 00:12:20 | sean-k-mooney | .get_endpoint() does not incldue the microversion info | |
| 00:12:53 | sean-k-mooney | so you need to extract the info via the session | |
| 00:45:10 | opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Use a per-instance cephx identity for CephFS shares https://review.opendev.org/c/openstack/nova/+/1001900 | |
| 00:45:11 | opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Support cold migration with virtiofs shares https://review.opendev.org/c/openstack/nova/+/989633 | |
| 00:45:11 | opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Extract share management from ComputeManager https://review.opendev.org/c/openstack/nova/+/999349 | |
| 00:45:12 | opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Reconcile stale Manila share access rules https://review.opendev.org/c/openstack/nova/+/1001916 | |
| 01:05:50 | opendevreview | Merged openstack/nova master: Fix PEP-765 syntax warning https://review.opendev.org/c/openstack/nova/+/988636 | |
| 01:06:02 | opendevreview | Merged openstack/nova master: Drop description for ancient horizon https://review.opendev.org/c/openstack/nova/+/989419 | |
| 01:06:14 | opendevreview | Merged openstack/nova master: Update contributor guide for 2026.1 Gazpacho https://review.opendev.org/c/openstack/nova/+/961896 | |
| 01:06:26 | opendevreview | Merged openstack/nova master: Update contributor guide for 2026.2 Hibiscus https://review.opendev.org/c/openstack/nova/+/997870 | |
| 01:31:04 | opendevreview | Merged openstack/nova master: Filter out deleted instances when looking for build timouts https://review.opendev.org/c/openstack/nova/+/880125 | |
| 02:27:56 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Fix graceful shutdown functional unstable tests https://review.opendev.org/c/openstack/nova/+/1002223 | |
| 02:30:02 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Fix graceful shutdown functional unstable tests https://review.opendev.org/c/openstack/nova/+/1002223 | |
| 02:31:03 | gmaan | gibi: melwitt ^^ this is possible fix for the graceful shutdown unstable tests. We can see how it goes, I am not closing bug in case there are other race exist there. | |
| 02:31:13 | gmaan | I added it on top of other fixes | |
| 03:13:24 | opendevreview | Clif Houck proposed openstack/nova master: Parallelize per-node resource updates https://review.opendev.org/c/openstack/nova/+/980679 | |
| 03:13:24 | opendevreview | Clif Houck proposed openstack/nova master: perf(ironic): eliminate O(N²) ProviderTree deepcopy at startup https://review.opendev.org/c/openstack/nova/+/980676 | |
| 03:40:51 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Add functional test for cleanup_host cancelling queued live migrations https://review.opendev.org/c/openstack/nova/+/986477 | |
| 05:02:33 | opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Reconcile stale Manila share access rules https://review.opendev.org/c/openstack/nova/+/1001916 | |
| 05:47:35 | opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Use a per-instance cephx identity for CephFS shares https://review.opendev.org/c/openstack/nova/+/1001900 | |
| 05:47:36 | opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Support cold migration with virtiofs shares https://review.opendev.org/c/openstack/nova/+/989633 | |
| 05:47:36 | opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Extract share management from ComputeManager https://review.opendev.org/c/openstack/nova/+/999349 | |
| 05:47:37 | opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Reconcile stale Manila share access rules https://review.opendev.org/c/openstack/nova/+/1001916 | |
| 08:43:37 | gibi | I'm pretty sure that some of the instability is due to threading allows switching between tasks at any point of execution and then we have the CI slowdown that forces more switches resulting in more race conditions materializing now | |
| 08:44:58 | gibi | I ran a lot of functional-threading tests locally and reasonable amount of it in CI before we made the functional-py313-threading voting and it was all shiny 3 weeks ago. Now I have multiple bugs filed on new race conditions | |
| 08:55:18 | gibi | gmaan: https://review.opendev.org/c/openstack/nova/+/1002223 needs more tweaking according to CI | |
| 09:04:34 | gibi | I assigned two bugs to myself and started looking for a root cause / fix https://bugs.launchpad.net/nova/+bug/2164606 and https://bugs.launchpad.net/nova/+bug/2164893 they are more frequent than the rest of non-graceful hits | |
| 09:51:30 | gibi | btw I can trigger some new race conditions locally if I artificially restric the amount of cpu and io I give to tox via | |
| 09:51:41 | gibi | systemd-run --user --scope -p CPUQuota=400% -p IOReadBandwidthMax="/ 2M" tox -e functional-py313-threading | |
| 09:55:51 | opendevreview | Balazs Gibizer proposed openstack/nova master: Stabilize TestInstanceNotificationSample.test_instance_action https://review.opendev.org/c/openstack/nova/+/1002246 | |
| 09:56:10 | gibi | OK that is one fix for the many ^^ | |
| 10:07:40 | opendevreview | Takashi Kajinami proposed openstack/nova master: Remove unused default argument https://review.opendev.org/c/openstack/nova/+/922073 | |
| 11:16:17 | sean-k-mooney | 'compute.exception' is a notification? | |
| 11:16:34 | sean-k-mooney | is that like any uncuaght excpetion on the compaute agent | |
| 11:16:56 | sean-k-mooney | or something we send at a specific time | |
| 11:19:18 | sean-k-mooney | i mean that definetly a think that exists and we are waitign for it in rebuild filaers https://github.com/openstack/nova/blob/9aa9a54e04fdfec9a8775bf90488558783179d4b/nova/tests/functional/test_servers_resource_request.py#L2230-L2232 | |
| 11:19:23 | sean-k-mooney | so the pattern looks right | |
| 12:03:35 | gibi | sean-k-mooney: yepp compute.exception is a catch all notification | |
| 12:04:10 | sean-k-mooney | i +2'd with a note about doing it for our only other *_with_error helper as well | |
| 12:05:00 | sean-k-mooney | i assume we may have a similar race for share detach as share attach | |
| 12:06:11 | gibi | yepp you are right and I'm updating the patch | |
| 12:06:12 | sean-k-mooney | i dont know if it has a simialr assertion however hence +2, if you think its needed ill happilry re review | |
| 12:06:39 | sean-k-mooney | cool i didnt really want ot dig into that to confirm | |
| 12:15:06 | opendevreview | Balazs Gibizer proposed openstack/nova master: Stabilize TestInstanceNotificationSample.test_instance_action https://review.opendev.org/c/openstack/nova/+/1002246 | |
| 12:15:15 | gibi | sean-k-mooney: fixed it up ^^ | |
| 12:32:37 | sean-k-mooney | oh it was listing them but only asserting 2 | |
| 12:32:40 | sean-k-mooney | fun | |
| 12:32:48 | gibi | :) | |
| 13:06:05 | opendevreview | Balazs Gibizer proposed openstack/nova master: Stabilize TestInstanceNotificationSampleWithMultipleCompute https://review.opendev.org/c/openstack/nova/+/1002281 | |
| 13:06:15 | gibi | another fix ^^ | |
| 13:10:00 | gibi | I'm taking https://bugs.launchpad.net/nova/+bug/2164888 as well now | |
| 13:25:22 | clif | anyone familiar with a mypy failure around "ExceptionGroup" it complains the name is not defined despite it being a builtin... | |
| 13:31:39 | clif | also are nova tox functional tests able to be run locally? | |
| 13:31:53 | sean-k-mooney | an excpetion group is not a type | |
| 13:32:00 | sean-k-mooney | its a collection of types | |
| 13:32:19 | sean-k-mooney | as far as im aware at least | |
| 13:32:23 | sean-k-mooney | but no i ahve not seen that | |
| 13:32:34 | clif | wdym, it is a type isn't it? https://docs.python.org/3/library/exceptions.html#ExceptionGroup | |
| 13:33:02 | sean-k-mooney | hum ok added in 3.11 | |
| 13:33:19 | sean-k-mooney | it wasnt a type before this is the first release we have been abot to use 3.11 features | |
| 13:33:38 | sean-k-mooney | before expction groups were jsut a tuple of possible expctions | |
| 13:34:18 | clif | it seems to work fine based on local testing but mypy blows up on it in tox pep8 | |