| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-03-25 | |||
| 17:49:09 | cardoe | Literally the first commit adding support for Ironic into Nova relied on that behavior. So I'm at a loss how it's unsupported. | |
| 17:49:33 | JayF | ^^ this is what I'm saying, I would love real reasoning other than "because it's always been that way" | |
| 17:49:36 | cardoe | sean-k-mooney: you can change the type of an unbound port. The port isn't yet bound. It's valid by Neutron spec and API. | |
| 17:49:57 | cardoe | That was confirmed during the last PTG. | |
| 17:50:08 | sean-k-mooney | you can but a service shoudl not the vnice tyep is a request form the end user | |
| 17:50:25 | sean-k-mooney | and a port with a nova isntane atach shoudlnot be modifed by ironic but it will be | |
| 17:50:33 | sean-k-mooney | as i said this is how it works today | |
| 17:50:51 | sean-k-mooney | but it has been a long standing pain point that ironic is not workign liek every other virt dirver | |
| 17:50:52 | cardoe | s/today/has always worked/ | |
| 17:51:04 | sean-k-mooney | right and it has alwasy been a mild security issue | |
| 17:51:22 | sean-k-mooney | nova actully did have a security issue which we have fix related to vnic types beign changed | |
| 17:51:54 | sean-k-mooney | anywya im not agains supprotign specifying a vnic type but other are | |
| 17:52:11 | sean-k-mooney | so this would be an api change and one requiring a spec | |
| 17:52:34 | sean-k-mooney | valid to dicuss just ohters woudl prefer not too change this | |
| 17:52:41 | cardoe | Well I've been trying to get a spec approved for 2 years and now ya want me to write another? | |
| 17:52:58 | sean-k-mooney | all api changes require specs that unconditional | |
| 17:53:01 | JayF | maybe we can find what needs to change in your spec to be what sean-k-mooney needs? | |
| 17:53:06 | sean-k-mooney | so this woudl never be conidered a bug in nova | |
| 17:53:06 | JayF | can that be put in review? | |
| 17:53:38 | sean-k-mooney | JayF: im +2 on the ohter spec by the way and have been for 3 cycles :) | |
| 17:53:49 | cardoe | It's absolutely a Nova bug. The Nova Ironic driver creates a Port that cannot work with Ironic. | |
| 17:55:16 | sean-k-mooney | no the compute manger does | |
| 17:55:38 | sean-k-mooney | vnic_type=normal is the only thing that nova can create | |
| 17:55:50 | sean-k-mooney | and we have alwasy say that to use ironic you must pass a port | |
| 17:55:59 | sean-k-mooney | as a reasult of the requrieemnt for vnic tyep baremental | |
| 17:57:30 | cardoe | Can you point me to the docs which say to use Ironic with Nova you must pass your own Port? | |
| 17:57:36 | cardoe | https://docs.openstack.org/nova/2025.2/admin/configuration/hypervisor-ironic.html doesn't show it | |
| 17:59:14 | cardoe | https://docs.openstack.org/nova/2025.2/user/support-matrix.html even shows the operation where Nova is creating the Port itself and as supported. | |
| 18:02:12 | sean-k-mooney | cardoe: its partly a sideffe of nova only supprot vnic_type=normal prot when it creats them and https://review.opendev.org/c/openstack/nova-specs/+/672400 was the last tiem we propsed this as a spec | |
| 18:02:26 | sean-k-mooney | cardoe: ill see fi we have an explcit doc or not | |
| 18:21:06 | sean-k-mooney | cardoe: ironic has https://cdn.maingear.com/docs/ironic/2025.1/admin/networking.html#vif-attachment-flow the srong recoemdation to alwasy create the prot in advnce. one thing to note is while teh current flow was orgianly approved in https://github.com/openstack/nova-specs/blob/master/specs/mitaka/approved/ironic-networks-support.rst#tenant-networking-support-for-ironic-driver | |
| 18:21:08 | sean-k-mooney | with the modification of the vnic type out of band by ironic that spec was never reappovoed and accpated for a later release but the code wa smerged for some unknow reason | |
| 18:21:53 | sean-k-mooney | at the time this was approve the only usecase for a non defautl vnic type was neutron sriov prot which requried the port to be precreated. | |
| 18:21:59 | cardoe | Let's just abandon my proposed change. I'll go away. | |
| 18:22:20 | sean-k-mooney | cardoe: i think the idea of a hypervior depenent defualt vnic type is not a bad one | |
| 18:22:50 | sean-k-mooney | so im not against this just provideign the wider context | |
| 19:00:52 | JayF | Yeah I think we have the fix in line with cardoe's change, just need to figure out the right way to market it to at least one other Nova core and if needed update the paperwork to be in line with whatever is expected if they consider that a feature and not a bugfix | |
| 20:07:24 | sean-k-mooney | JayF: next step would be nova irc meeting. it would be better in the long term if we also had a path to growign more nova cores with ironic expertiese. in the past john garbut was one but they have less time for nova these days | |
| 20:07:39 | sean-k-mooney | JayF: im +1 on the patch over all. | |
| 20:08:39 | JayF | sean-k-mooney: similarly on the Ironic side; I've tried to take the mantle of "expert in the driver", which I think I've "hit" but many of the things here are lower level than that. Between the group of us we can find a solution I'm sure :) | |
| 20:10:58 | opendevreview | Antonin Ruan proposed openstack/nova-specs master: Temp URL download spec https://review.opendev.org/c/openstack/nova-specs/+/975883 | |
| 20:22:00 | opendevreview | Erlon R. Cruz proposed openstack/nova master: Add RBD XML update functionality for migration process https://review.opendev.org/c/openstack/nova/+/974032 | |
| 22:41:11 | opendevreview | Merged openstack/nova master: Update min support for Hibiscus https://review.opendev.org/c/openstack/nova/+/979301 | |
| #openstack-nova - 2026-03-26 | |||
| 08:47:09 | priteau | Hello Nova team. I have just hit the `mkfs.vfat: Label can be no longer than 11 characters` issue while live migrating an instance. There is a fix in Nova but backports are waiting for review. Could you please merge? https://review.opendev.org/c/openstack/nova/+/952003 | |
| 10:04:22 | priteau | Uggla: Would it be possible to merge these backports? | |
| 10:09:05 | Uggla | priteau, I think so. I'll try to have them merged. | |
| 10:09:25 | priteau | Thank you! | |
| 10:16:16 | gibi | priteau: approved the backort now | |
| 10:36:15 | opendevreview | Doug Szumski proposed openstack/nova master: Stop corrupting ephemeral volumes during cold migration https://review.opendev.org/c/openstack/nova/+/940900 | |
| 10:46:11 | priteau | Thanks gibi. I might ping you for the next ones! | |
| 10:47:06 | gibi | priteau: sure | |
| 10:58:38 | nathharp | hey knowledgable nova people. I'm having problems with driving Ironic via nova-compute: when building multiple instances, i see errors like: "status": 409, "title": "Conflict", "detail": "There was a conflict when trying to complete your request.\n\n resource provider generation conflict ", "code": "placement.concurrent_update". Is this likely to be an issue with nova, or placement? This is a 2025.1 cloud | |
| 11:23:00 | nicolairuckel | https://specs.openstack.org/openstack/nova-specs/specs/2025.1/approved/vtpm-live-migration.html Is it correct, that there is nothing to be implemented for the user mode since in that case live migration is not possible? | |
| 12:50:56 | opendevreview | Merged openstack/nova stable/2025.1: libvirt: Use common naming convention for ephemeral disk labels https://review.opendev.org/c/openstack/nova/+/952003 | |
| 12:56:24 | priteau | gibi: 2025.1 backport has merged, thanks! 2024.2 backport is ready too: https://review.opendev.org/c/openstack/nova/+/952004 | |
| 13:29:51 | opendevreview | Balazs Gibizer proposed openstack/nova master: Default native threading for conductor and compute https://review.opendev.org/c/openstack/nova/+/982237 | |
| 14:32:20 | opendevreview | Kamil Sambor proposed openstack/nova master: Replace eventlet with standard threading in novncproxy https://review.opendev.org/c/openstack/nova/+/976089 | |
| 15:09:45 | opendevreview | Merged openstack/nova stable/2024.2: libvirt: Use common naming convention for ephemeral disk labels https://review.opendev.org/c/openstack/nova/+/952004 | |
| 16:06:11 | opendevreview | Kamil Sambor proposed openstack/nova master: Replace eventlet with standard threading in novncproxy https://review.opendev.org/c/openstack/nova/+/976089 | |
| 17:11:13 | clif | sean-k-mooney: could you give me some context on ProviderTree? does the tree represent a heirarchical relationship between nodes? I also see ProviderTree calls out the ironic driver specifically for having 'large number of roots' | |
| 17:15:09 | sean-k-mooney | clif: placement is a forset data stucre where we have a forest of provider that are them selve aranged in a tree stucture with root RPs representign a comptue node and potically nested resouce provider as childer repcesting gpus or pci device on that compute host | |
| 17:15:50 | sean-k-mooney | clif: in the context of ironic each ironic node is create in placement as a seperate resouce provider with invetory of 1 resouce class fo custom_baremtal_something | |
| 17:16:18 | sean-k-mooney | so a single nova compute agent with the ironic driver manage a set of trees of depht 1 | |
| 17:16:35 | clif | ah, ok | |
| 17:16:48 | sean-k-mooney | the uuid of the placment resouce provider is the ironic node uuid which is alsoce the nova compute node uuid | |
| 17:17:20 | sean-k-mooney | so it a 1:1:1 mapping between compute_node in the nova db, resouce provider in the placment db and node in the ironic db | |
| 17:17:52 | clif | so each tree is a single ironic node and each tree's children is information about the ironic node? | |
| 17:18:00 | sean-k-mooney | clif: for almost every other dirver each nova comptue agent manages exactly 1 compute node and 1 root resouce provider | |
| 17:18:35 | sean-k-mooney | clif: for hte ironic diver we dont create childer sinc you cant subdevide ironic hosts | |
| 17:18:39 | JayF | (this is, I presume, the root cause for how so much perf was left on the table in the Ironic case?) | |
| 17:18:45 | sean-k-mooney | we could in the future but we didnt have a suecase today | |
| 17:19:03 | sean-k-mooney | JayF: yes more or less | |
| 17:20:00 | sean-k-mooney | JayF: the first botlenec (the deep copy of the providers manged by this compute agent) | |
| 17:20:19 | sean-k-mooney | is cheap if the number of provider is 1-8 max in 99% of cases | |
| 17:20:42 | sean-k-mooney | its really expensive if you have 100s or 100s espcially if your doing it in a loop | |
| 17:21:05 | clif | could we create a seperate provider implementation for ironic driver use? | |
| 17:21:06 | sean-k-mooney | JayF: so i think that is 1 of the hopefully simple things to fix | |
| 17:21:31 | sean-k-mooney | clif: we try not to have any virt driver awareness in teh compute manager or resouce tracker | |
| 17:21:41 | sean-k-mooney | and that is where the provider stuff lives | |
| 17:22:01 | sean-k-mooney | but short answer is yes i think we can optimize it | |
| 17:22:06 | clif | alright | |
| 17:22:10 | sean-k-mooney | jsut without the need for an if ironic check | |
| 17:22:39 | sean-k-mooney | clif: that is what https://review.opendev.org/c/openstack/nova/+/980676 is ment to do | |
| 17:23:08 | sean-k-mooney | specifically https://review.opendev.org/c/openstack/nova/+/980676/1/nova/compute/resource_tracker.py#1314 | |
| 17:23:27 | clif | right, I'm trying to load enough nova context into my noggin to get to the point where I can understand what's going on with that patch, the perf sim, etc | |
| 17:24:34 | sean-k-mooney | when we get the provider tree for a spcific node that shoudl copy only that node RP insated fo every provider in our provider cash https://review.opendev.org/c/openstack/nova/+/980676/1/nova/scheduler/client/report.py#935 | |
| 17:25:29 | clif | it seems reasonable on the surface | |
| 17:25:35 | sean-k-mooney | clif: ya that a non zeor amount of context to load | |
| 17:26:07 | sean-k-mooney | the other optimisation that i think has prommis is this patch https://review.opendev.org/c/openstack/nova/+/980679/1 | |
| 17:26:19 | sean-k-mooney | so that that one does is very simple | |
| 17:26:47 | sean-k-mooney | today we have a for loop over every ironic node managed by this agent (limited via shard key or peer_list) | |
| 17:27:06 | sean-k-mooney | the patch just kick the processing of each node out into a thread pool | |
| 17:27:19 | sean-k-mooney | so you can actully do them in parallel an tune it via a config option | |
| 17:27:28 | sean-k-mooney | defaulting to 1 for backward compatiablity | |
| 17:29:08 | JayF | And I hope this is all building up, once the identified low hanging fruit are picked, to a deeper look at if there's things we can do to improve it further. I am skeptical that this will be enough to bring us within an acceptable startup time :) | |
| 17:29:17 | clif | I will continue to delve | |
| 17:29:28 | clif | thanks for the context sean | |
| 17:30:17 | sean-k-mooney | JayF: these were the hotspots in the current profiling but if we can get a beter repoducer with the ironic fake driver or soemthign elese we can see what the next bottel neck is | |
| 17:30:53 | JayF | ironic devstack plugin support for fake driver nodes is now #2 on my todo list (I got the PRs up removing 15k worth of retired driver code lines from Ironic) | |
| 17:31:02 | sean-k-mooney | JayF: we could as you suggested skip the placement management entirly in nova and delegate that to ironic but i dont know if that will really help | |