| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-03-26 | |||
| 17:25:35 | sean-k-mooney | clif: ya that a non zeor amount of context to load | |
| 17:26:07 | sean-k-mooney | the other optimisation that i think has prommis is this patch https://review.opendev.org/c/openstack/nova/+/980679/1 | |
| 17:26:19 | sean-k-mooney | so that that one does is very simple | |
| 17:26:47 | sean-k-mooney | today we have a for loop over every ironic node managed by this agent (limited via shard key or peer_list) | |
| 17:27:06 | sean-k-mooney | the patch just kick the processing of each node out into a thread pool | |
| 17:27:19 | sean-k-mooney | so you can actully do them in parallel an tune it via a config option | |
| 17:27:28 | sean-k-mooney | defaulting to 1 for backward compatiablity | |
| 17:29:08 | JayF | And I hope this is all building up, once the identified low hanging fruit are picked, to a deeper look at if there's things we can do to improve it further. I am skeptical that this will be enough to bring us within an acceptable startup time :) | |
| 17:29:17 | clif | I will continue to delve | |
| 17:29:28 | clif | thanks for the context sean | |
| 17:30:17 | sean-k-mooney | JayF: these were the hotspots in the current profiling but if we can get a beter repoducer with the ironic fake driver or soemthign elese we can see what the next bottel neck is | |
| 17:30:53 | JayF | ironic devstack plugin support for fake driver nodes is now #2 on my todo list (I got the PRs up removing 15k worth of retired driver code lines from Ironic) | |
| 17:31:02 | sean-k-mooney | JayF: we could as you suggested skip the placement management entirly in nova and delegate that to ironic but i dont know if that will really help | |
| 17:31:26 | sean-k-mooney | we may need to consider other change liek batch updating placement which may need an api change | |
| 17:31:31 | JayF | sean-k-mooney: basically in the places we poll Ironic currently, any way to switch that to something more evented | |
| 17:31:37 | JayF | sean-k-mooney: or batching into placement, yep | |
| 17:31:58 | sean-k-mooney | well yes so we coudl extend the nova external event api for ironic | |
| 17:32:00 | JayF | sean-k-mooney: either way once we pluck the low hanging fruit I want us to take a look at doing the bigger stuff to help; myself and clif can be the hands making the action work | |
| 17:32:08 | sean-k-mooney | that woudl allow use to get ride of poolign for power state for example | |
| 17:32:11 | JayF | I just don't always know which way is up in Nova :) | |
| 17:32:54 | sean-k-mooney | JayF: if you are not familary with https://docs.openstack.org/api-ref/compute/#create-external-events-os-server-external-events | |
| 17:33:19 | sean-k-mooney | that is how cidner and nova tell nova that the sate of soemthign has changed or a workflow has completed in an event based manner | |
| 17:33:44 | JayF | Interesting | |
| 17:33:53 | sean-k-mooney | power-update might actuly be used by ironic already | |
| 17:34:06 | sean-k-mooney | but there is no reason that we coudl not add other event if we had a reason too | |
| 17:34:07 | JayF | I am 99.999999% it's not | |
| 17:34:46 | sean-k-mooney | im just tyring to think what ealse is power related that it could be | |
| 17:34:57 | JayF | yep we 100% do | |
| 17:35:00 | sean-k-mooney | https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id69 | |
| 17:35:00 | JayF | TIL | |
| 17:35:27 | JayF | https://opendev.org/openstack/ironic/src/branch/master/ironic/common/nova.py#L91 | |
| 17:35:51 | sean-k-mooney | yep so i think this was added so we can turn off the power monitring perodic in ironic deployments | |
| 17:35:52 | JayF | if we could just trigger update of placement relevant info from Ironic e.g. on certain state changes | |
| 17:36:12 | JayF | I don't care if any of that comm goes Ironic<>Nova<>Placement or Ironic<>Placement | |
| 17:36:30 | sean-k-mooney | well ironic can go do placmeent adn do some limited things today | |
| 17:36:43 | JayF | especially if we were able to somehow get those calls to impact the node cache inside the virt driver itself | |
| 17:37:01 | sean-k-mooney | JayF: so if you tell me waht you want to update there i can say if it doable today or not without nova changes | |
| 17:37:05 | JayF | because then we can do the "reconciliation job on a timer" change like you did that I already -1'd | |
| 17:37:50 | JayF | sean-k-mooney: I don't know that answer (yet); but whatever happens when a node goes unavailable (e.g. cleaning or move out of AVAILABLE state) and whatever happens when a node goes available (anything->AVAILABLE state) | |
| 17:38:16 | JayF | Plus maybe something that would trigger on a node trait add to expose that trait to placement | |
| 17:38:29 | sean-k-mooney | so the way to model that in placement today is to set reserved=total on the rp | |
| 17:38:49 | sean-k-mooney | that is something that ironic can do directly | |
| 17:38:58 | JayF | Right now my goal is to get all these written down so we can evaluate them after the lower hanging fruit have been plucked | |
| 17:39:09 | JayF | because hopefully by then we'll have devstack w/fake driver nodes | |
| 17:39:10 | sean-k-mooney | ack sound good | |
| 17:39:19 | JayF | and can have real data for what is helpful vs what just seems helpful at a surface read | |
| 17:43:58 | sean-k-mooney | JayF: you know if we do deploy with the fake driver with a 1000s inically devstack is definlly going to cail because it will time out waiting for nova-comptue to start :P | |
| 17:44:25 | sean-k-mooney | so maybe ill try with 10 and then add them once it up | |
| 17:44:42 | JayF | sean-k-mooney: Ironic has a long, long, long history of knowing where all devstack timeouts are so we can set them high enough to make an infra team member weep | |
| 17:44:52 | JayF | lol | |
| 17:45:15 | JayF | sean-k-mooney: maybe add 1000 fake nodes, only put 3 of them in a shard managed by nova-compute by default or something :D either way I'll figure it out | |
| 17:47:05 | JayF | sean-k-mooney: clif: I put some notes in here https://etherpad.opendev.org/p/ironic-nova-compute-startup-2026.2 | |
| 17:47:47 | sean-k-mooney | clif: if you want to rebase of rearrage those wip patches feel free | |
| 17:48:29 | sean-k-mooney | i wont get back to them for a while but if they are helpful to you go for it | |
| 17:51:06 | clif | thank you, will do | |
| 18:51:17 | melwitt | gmaan: hey, I have a hopefully easy patch to fix openstacksdk usage for using endpoint discovery for unified limits if you might be able to look at your convenience https://review.opendev.org/c/openstack/nova/+/975106 | |
| 19:04:00 | erlon | sean-k-mooney++ melwitt++ thanks for the througly review on the RBD XML update functionality for migration process patch | |
| 19:05:27 | sean-k-mooney | erlon: im not sur eif keeping the sepreate small funciton for nova provisioned rbd storage is bettere or not for what its worth | |
| 19:05:34 | sean-k-mooney | i could be preswaded either way | |
| 19:05:56 | sean-k-mooney | but i woudl really prefer to see a functional repoducer for that | |
| 19:07:50 | erlon | melwitt: did you tried the XML "<ip family="ipv6" address="[2001:db8:ca2:2::1]" prefix="64"/>" with a ipv6 addr without the []? did it pass the checks? | |
| 19:10:44 | erlon | sean-k-mooney: you mean to have that code inside other function like _update_volume_xml? | |
| 19:11:34 | erlon | I believe that having the code in a separate function is better for readability and testing | |
| 19:14:19 | erlon | Another thing I was considering, is that when I brought it up in the meeting, we discussed wanting another managed command to perform the bulk update. I didn't start to think about how I'll do that implementation, but having things modularized usually help to reuse code. | |
| 19:15:55 | sean-k-mooney | erlon: my conder is driffet between the nova and cidner voulem management | |
| 19:16:27 | sean-k-mooney | erlon: like currently you are just updating the xml but not th bdms right | |
| 19:16:52 | melwitt | erlon: yes I did I pasted the output in the code block of my comment, I ran the virt-xml-validate CLI without brackets (passed) and with brackets (failed) | |
| 19:16:54 | sean-k-mooney | i dont recall if we store rbd info in teh bdms for nova provisioned stroage | |
| 19:17:22 | sean-k-mooney | melwitt: its possible that libvirt version depenent | |
| 19:17:23 | melwitt | no we don't, BDMs with rbd info is for cinder volume only | |
| 19:17:40 | sean-k-mooney | ok then that not an issue with the current patch so | |
| 19:18:05 | melwitt | sean-k-mooney: yes true. it would be ideal if we could test it for real somehow but regardless I think it's safest to do it without brackets bc that's how XML is currently generated for rbd | |
| 19:18:26 | sean-k-mooney | we have a tempest-ipv6 job in our gate | |
| 19:18:32 | sean-k-mooney | i just doen not ahve ceph in it | |
| 19:18:45 | sean-k-mooney | but i think we can look at that in a followup | |
| 19:18:54 | melwitt | I posted a comment on the review but I am wondering if the patch is actually needed right? bc AFAICT all cases should be covered already for nova local/ephemeral rbd | |
| 19:19:33 | melwitt | I wonder if people just want the patch to force an update to XML right away instead of waiting for a lifecycle event. just not sure I get the use case | |
| 19:20:10 | sean-k-mooney | so its not if you reboot the vm | |
| 19:20:35 | sean-k-mooney | but if we try to live migrate with the old mons i think the dest vm will fail to connect to the storage | |
| 19:20:45 | melwitt | ceph handles all of mon IP update transparently while the guest is running and it's not clear from the bug report if the lack of XML update there results in a failure or error of some kind | |
| 19:20:49 | melwitt | right and if you reboot the VM the existing code already grabs fresh mon IPs from ceph | |
| 19:20:49 | sean-k-mooney | and the migration will fail as a result if those mons are note aviabel | |
| 19:21:07 | melwitt | ok, that is what I wanted to know. I wish they had pasted the error in the bug | |
| 19:21:44 | sean-k-mooney | ya so unless libvirt is proxiyng that info without telling us i suspect that is the failure mode | |
| 19:21:48 | melwitt | ok, if it's actually a failure and not just a "I will feel better if the XML is updated now vs later" then I think the patch is useful | |
| 19:21:59 | erlon | melwitt, what would be the lifecicle events that would update that? | |
| 19:23:15 | melwitt | erlon: basically anything that calls _hard_reboot() internally so a stop/start, a hard reboot, an unshelve, I'm not sure what else off the top of my head. the main point is as long as the guest is running, the xml does not need an update, that's for later when it's starting up from being shutdown | |
| 19:23:23 | erlon | afic, ephemeral rbds are not updated unless the instance is sheved/unshelved | |
| 19:23:51 | sean-k-mooney | it should be updated any tiem teh domain is redefiend | |
| 19:24:16 | melwitt | erlon: however, if live migration would in fact _fail_ without the xml updated, then that is the problem that makes the patch needed. it was just not clear to me from the bug report whether this is a real failure or if someone did dumpxml and saw old mon IPs and that made them believe there is a problem | |
| 19:24:21 | sean-k-mooney | so cold migrate, shleve hard,reboot,evacuate | |
| 19:24:21 | erlon | yes, but the use case is to keep the instance running. | |
| 19:24:41 | melwitt | yup every time it's redefined. for local/ephemeral RBD to be clear. Cinder volume is NOT that way | |
| 19:25:22 | melwitt | for Cinder volume, currently, you have to live migrate to make it consume new mon IPs for its next xml redefinition. and I wonder if that might be the confusion | |
| 19:25:43 | erlon | melwitt: it does fail since, when RBD is trying to connect to one of the monitors, if the monitor times out, rbd will not retry | |
| 19:26:27 | melwitt | ack ok | |
| 19:27:19 | erlon | melwitt: no, the patch clearly handles that difference between Cinder back the volumes and RBD ephemerals and config drivers | |
| 19:27:56 | erlon | *cinder backed | |
| 19:28:16 | sean-k-mooney | erlon: sure but its addign supprot for somethign we do not supprot | |
| 19:28:28 | sean-k-mooney | changign the mon ips for ceph cluster attached to a nova instnace | |