| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-05-27 | |||
| 15:14:43 | sean-k-mooney | im doign one more revieison of the agentic coding scafolls which ill push shortly | |
| 15:14:49 | sean-k-mooney | that what i was distcted by | |
| 15:14:57 | sean-k-mooney | but cool i shoudl be doen before the triage call | |
| 15:15:05 | sean-k-mooney | i think i adress most of your feedback | |
| 15:19:44 | opendevreview | sean mooney proposed openstack/nova master: Add agentic coding guidance and docs https://review.opendev.org/c/openstack/nova/+/984540 | |
| 15:28:18 | opendevreview | sean mooney proposed openstack/nova master: Add agentic coding guidance and docs https://review.opendev.org/c/openstack/nova/+/984540 | |
| 15:49:14 | opendevreview | Lajos Katona proposed openstack/nova master: Use SDK for Neutron Ports https://review.opendev.org/c/openstack/nova/+/969298 | |
| 15:49:29 | opendevreview | Lajos Katona proposed openstack/nova master: Use SDK for Neutron security-groups https://review.opendev.org/c/openstack/nova/+/981141 | |
| 15:52:18 | opendevreview | sean mooney proposed openstack/nova stable/2025.2: ironic: Abort build on fatal image validation errors https://review.opendev.org/c/openstack/nova/+/990319 | |
| 16:31:11 | Uggla | sean-k-mooney, gibi the reproducer and fix I mentionned in triage call 982675: Add reproducer for bug #2111320: soft-deleted services in Instance.services | https://review.opendev.org/c/openstack/nova/+/982675 && 982676: Filter soft-deleted services from Instance.services ORM relationship | https://review.opendev.org/c/openstack/nova/+/982676 if | |
| 16:31:13 | Uggla | you can have a look, no hurries though. | |
| 17:17:57 | gmaan | dansmith: this is graceful shutdown part2 (task tracking) spec, please review https://review.opendev.org/c/openstack/nova-specs/+/986447 | |
| 17:24:28 | dansmith | gmaan: ack | |
| 19:06:32 | gmaan | dansmith: thanks for the review. I replied to your comments. can you please check the 3 open comments before I update it for rest of other | |
| 19:22:16 | opendevreview | Merged openstack/nova-specs master: Add virtiofs cold migration spec https://review.opendev.org/c/openstack/nova-specs/+/985738 | |
| 19:43:16 | gmaan | dansmith: thanks for catching that, I forgot to mention in spec but the manager wrapper will exclude the async tasks (tracking), replied there- https://review.opendev.org/c/openstack/nova-specs/+/986447/comment/b0cc44d8_bcf54af5/ | |
| 19:43:27 | gmaan | does that make sense and resolve your concern? | |
| 19:44:16 | gmaan | task will be tracked once either by the manager wrapper or by the async tasks tracker when they actually finsihed | |
| 19:47:28 | sean-k-mooney | gmaan: out of scope for this cycle but i was chating to JayF about the graceful shutdown spec indirectly. on of the topics that came up was resuming ongoing operation on agent start up instead o fmarkign them as failed or other ways to reduce the impact of an agent restart on thses long runing ironic operations | |
| 19:47:53 | sean-k-mooney | i suggested that your graceful shutdown work could evengually help | |
| 19:48:28 | sean-k-mooney | but i was wondering if you had spent any time conidring that. i think we identified reumeimg operations as step 3 right | |
| 19:49:45 | sean-k-mooney | server boots and rebuilds cna take 10s of mintues in some case | |
| 19:50:08 | gmaan | sean-k-mooney: yeah that is step 3, I have not spent time on that yet. But yes, that is more of preserve the checkpoints and on startup if we can resume operations from that checkpoint | |
| 19:50:16 | sean-k-mooney | so ironic is impacted by the fact we mark the server in errror when we start up durign a rebuild more then libvirt as a result | |
| 19:51:05 | sean-k-mooney | ack cool well i have seeded that usecase back into your brain. feel free to forget about it until we are don with step 2 :) | |
| 19:53:02 | gmaan | yeah, i recorded it in backlog spec and once we finish step2 then I can plan to propose the detailed spec | |
| 20:22:45 | opendevreview | Ghanshyam Maan proposed openstack/nova-specs master: Spec for the graceful shutdown part2: Task tracking https://review.opendev.org/c/openstack/nova-specs/+/986447 | |
| 20:23:11 | gmaan | dansmith: ^^ updated, please re-review | |
| #openstack-nova - 2026-05-28 | |||
| 03:00:47 | songwenping | hi team, is NVIDIA A100 MIG devices can passthrough to KVM VM? | |
| 06:22:53 | ralonsoh | Hi folks, just a heads-up. I updated the Neutron policy file, using yours: https://github.com/openstack/nova/blob/380f657b5102707a3da676478e72fd96691e966b/doc/source/configuration/policy-concepts.rst | |
| 06:22:59 | ralonsoh | This is my patch: https://review.opendev.org/c/openstack/neutron/+/990069 | |
| 06:23:29 | ralonsoh | You should remove the references to `enforce_scope`, that is now True, deprecated and marked for removal | |
| 08:02:22 | opendevreview | yatin proposed openstack/nova master: [DNM] Do not use sqlalchemy reserved metadata https://review.opendev.org/c/openstack/nova/+/990429 | |
| 08:17:48 | sahid | o/ | |
| 08:19:59 | sahid | quick question guys, there is a missing field "tags" in the unversioned notifications, i'm wondering whether we can fix that or perhpas unversioned notif is deprecated and we don't want to do anything here? | |
| 08:22:14 | gibi | sahid: unversioned is deprecated we only add new stuff to versioned notificatoin | |
| 08:45:04 | sahid | gibi: ack thank you | |
| 08:45:43 | sahid | btw i have noticed that tags field is missing during delete if I have a moment i will try to fix that soon | |
| 09:01:39 | opendevreview | Balazs Gibizer proposed openstack/nova master: Poison eventlet import in native threading mode https://review.opendev.org/c/openstack/nova/+/986282 | |
| 09:01:57 | gibi | sahid: ack, feel free to ping me with the patc | |
| 09:01:58 | gibi | h | |
| 09:05:37 | opendevreview | huanhongda proposed openstack/nova master: Fix scheduler aggregate cache race on concurrent add_host https://review.opendev.org/c/openstack/nova/+/990437 | |
| 09:16:24 | sean-k-mooney | im wondering if we coudl run a little experiment in the nova gate for a week or two to see if we could finally get rid of the kernel paincs | |
| 09:17:11 | sean-k-mooney | i was chating to frickler in the infra channel and they had the insight that we may have been thinking about the root cause incorrectly and that the panic was actully cause by file system currption | |
| 09:18:25 | sean-k-mooney | they pointed out that the current behiovor looks as if the inital copy of the filesystem form the initram was interupted | |
| 09:18:47 | sean-k-mooney | cirros ships with a mostly empty rootfs (jsut a partion table) | |
| 09:18:57 | sean-k-mooney | and copies the root on first boot | |
| 09:19:15 | sean-k-mooney | if we interupt that then we will get a panic when we resize | |
| 09:19:26 | sean-k-mooney | so we inheringly have a race in the ci right now | |
| 09:19:40 | sean-k-mooney | once we normally win because the root fs is like 10 MB or so | |
| 09:20:09 | sean-k-mooney | based on there suggestion i built a test image https://github.com/SeanMooney/cirros/releases/tag/cirros-d260527-x86_64-5a75ef2-test | |
| 09:20:26 | sean-k-mooney | where the root is prepopulated and tested it with a quick devstack change | |
| 09:20:36 | sean-k-mooney | https://review.opendev.org/c/openstack/devstack/+/990348 | |
| 09:20:44 | sean-k-mooney | so the experimetn i would like to run in nova gate | |
| 09:21:04 | sean-k-mooney | is override the cirrors image we use in teh 2-3 jobs we normally see the issue in | |
| 09:21:15 | sean-k-mooney | and poitn it at my image | |
| 09:21:44 | sean-k-mooney | while i speraly chat to frickler and the other cirros folks about how to do this upstram in the cirros project | |
| 09:22:29 | sean-k-mooney | we typically see 10-20 kernel panics a day on nova jobs | |
| 09:22:48 | sean-k-mooney | so what im hopign is if we run that experimetn for a week or two that woudl drop to 0 | |
| 09:23:48 | sean-k-mooney | im going to create a patch to override the cirros image path for the relevent jobs so we can dicsuss mroe but i would be intersted in what folks think | |
| 09:24:43 | sean-k-mooney | are we willing to run the experiment for 2 weeks to get the data? we can obvioulsy quickly revert if needed | |
| 09:28:50 | sean-k-mooney | my corrent workign thory is we actully had 2 diffent kernel paninc, the first was resovled with the waiting ofr sshable changes for volume attach/detach. and the second was this file system currptionissue that we see with BFV + resize or BFV + any reboot i guess | |
| 09:29:32 | gibi | sean-k-mooney: your plan sounds sane to me | |
| 09:39:49 | sean-k-mooney | what i can do is prepare the url overied patch and add the topic to the irc meeting adjenda for monday. if we are happy to proceed we can merge it after the meeting adn then monitor it and see | |
| 11:06:37 | opendevreview | Joan Gilabert proposed openstack/nova-specs master: Repropose and update cyborg vGPU (mdev) support https://review.opendev.org/c/openstack/nova-specs/+/967515 | |
| 12:05:48 | opendevreview | Thibaut Démaret proposed openstack/nova master: libvirt: add disk rotation_rate support for local disks https://review.opendev.org/c/openstack/nova/+/979693 | |
| 12:37:41 | kklimaszewski | Hello, I've recently proposed a new nova spec for SPDK-based nova-provisioned storage backend: https://review.opendev.org/c/openstack/nova-specs/+/985676. Would anyone here be willing to review it/be its liaison? Also: Dominik (known here as Etua) has reproposed a spec from Ussuri for NUMA topology with resource providers: https://review.opendev.org/c/openstack/nova-specs/+/978570. Could someone here please review it? | |
| 14:49:47 | JayF | sean-k-mooney: I think I need to find time, regardless of interruptions and urgency, to get your patches cleaned up and landed upstream. That's a radical improvement. https://usercontent.irccloud-cdn.com/file/i2G0mxRP/image.png | |
| 14:50:45 | JayF | tl;dr: G-Research's (somewhat patched) 2024.1, one nova-compute that supported 1000 Ironic nodes was taking 65 minutes to restart. With three changes, that was reduced to 2 minutes. | |
| 14:51:11 | sean-k-mooney | damb | |
| 14:51:13 | sean-k-mooney | ok | |
| 14:51:18 | JayF | (NCI = "nova-compute-ironic" -- the internal GR moniker for nova-computes w/Ironic driver) | |
| 14:51:46 | JayF | given step #1 is also the first thing we wanna remove -- moving the resource updates to Ironic -- gives me hope as well that we're on the right track | |
| 14:52:56 | sean-k-mooney | so step 1 is the dispatching of the udpate to the thread pool right | |
| 14:53:12 | sean-k-mooney | so you can run N threads to do the work | |
| 14:53:23 | sean-k-mooney | you mentioned 32 workser but were they all used | |
| 14:53:55 | JayF | passed on the ask, I don't know | |
| 14:53:59 | sean-k-mooney | i assume you have not had time to test if say 8 or 16 has a linear effect? | |
| 14:54:02 | JayF | to be clear: I don't have operational access to this cluster at all | |
| 14:54:09 | sean-k-mooney | no worries | |
| 14:54:18 | sean-k-mooney | jsut wondering | |
| 14:54:45 | JayF | Honestly though, if we really get it down to 2 minutes | |
| 14:54:57 | JayF | something like a graceful shutdown starts looking really good for HA updates | |
| 14:55:03 | sean-k-mooney | JayF: the vtpm thing is also interesting i think we may have fixed that laze loadign thing on master recently | |
| 14:55:24 | JayF | if we could push Ironic queries to a secondary nova-compute while letting the first one complete actions in progress (or as discussed; hand off those in-progress actions from checkpoints) | |
| 14:57:27 | sean-k-mooney | JayF: https://review.opendev.org/c/openstack/nova/+/977037 | |
| 14:57:32 | sean-k-mooney | so not merged yet | |
| 14:57:47 | sean-k-mooney | JayF: but htat will fix the vtpm issue | |
| 14:57:56 | sean-k-mooney | its an issue for libivrt as well | |
| 14:58:29 | sean-k-mooney | added you to that so you shoudl get a notificaion whn that lands | |
| 14:59:46 | sean-k-mooney | i kind of forgot about that patch if im being honest but looks like i called out the ironic inpact the last tiem i looked :) | |
| 15:07:38 | sean-k-mooney | JayF: so the 3 patchs ye really need are https://review.opendev.org/c/openstack/nova/+/977037 https://review.opendev.org/c/openstack/nova/+/980676 and https://review.opendev.org/c/openstack/nova/+/980679/1 | |
| 15:16:15 | sean-k-mooney | on other thing that we coudl perhaps look into in the futre is allow nova-compute too run in a active stadby mode or a better way to have muliple nova-computes per shared then the previous hashring approch | |
| 15:16:21 | JayF | sean-k-mooney: I'm not convinced https://review.opendev.org/c/openstack/nova/+/977037 does the same as the downstream does | |
| 15:17:02 | sean-k-mooney | it does not skip the check but it but it remove the laze load overhead | |
| 15:17:05 | JayF | if not self.driver.capabilities.get('can_have_vtpm_instances', True):; return <--- he essentially just added this to the top of _validate_vtpm_configuration | |
| 15:17:35 | JayF | and added that as a driver capability (Which ironic has set to false) | |
| 15:17:38 | sean-k-mooney | ya that replace the expensive lazeload with an if | |
| 15:17:41 | sean-k-mooney | which is also valid | |
| 15:17:58 | sean-k-mooney | yep that will also work | |