Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2026-03-26
17:21:05 clif could we create a seperate provider implementation for ironic driver use?
17:21:06 sean-k-mooney JayF: so i think that is 1 of the hopefully simple things to fix
17:21:31 sean-k-mooney clif: we try not to have any virt driver awareness in teh compute manager or resouce tracker
17:21:41 sean-k-mooney and that is where the provider stuff lives
17:22:01 sean-k-mooney but short answer is yes i think we can optimize it
17:22:06 clif alright
17:22:10 sean-k-mooney jsut without the need for an if ironic check
17:22:39 sean-k-mooney clif: that is what https://review.opendev.org/c/openstack/nova/+/980676 is ment to do
17:23:08 sean-k-mooney specifically https://review.opendev.org/c/openstack/nova/+/980676/1/nova/compute/resource_tracker.py#1314
17:23:27 clif right, I'm trying to load enough nova context into my noggin to get to the point where I can understand what's going on with that patch, the perf sim, etc
17:24:34 sean-k-mooney when we get the provider tree for a spcific node that shoudl copy only that node RP insated fo every provider in our provider cash https://review.opendev.org/c/openstack/nova/+/980676/1/nova/scheduler/client/report.py#935
17:25:29 clif it seems reasonable on the surface
17:25:35 sean-k-mooney clif: ya that a non zeor amount of context to load
17:26:07 sean-k-mooney the other optimisation that i think has prommis is this patch https://review.opendev.org/c/openstack/nova/+/980679/1
17:26:19 sean-k-mooney so that that one does is very simple
17:26:47 sean-k-mooney today we have a for loop over every ironic node managed by this agent (limited via shard key or peer_list)
17:27:06 sean-k-mooney the patch just kick the processing of each node out into a thread pool
17:27:19 sean-k-mooney so you can actully do them in parallel an tune it via a config option
17:27:28 sean-k-mooney defaulting to 1 for backward compatiablity
17:29:08 JayF And I hope this is all building up, once the identified low hanging fruit are picked, to a deeper look at if there's things we can do to improve it further. I am skeptical that this will be enough to bring us within an acceptable startup time :)
17:29:17 clif I will continue to delve
17:29:28 clif thanks for the context sean
17:30:17 sean-k-mooney JayF: these were the hotspots in the current profiling but if we can get a beter repoducer with the ironic fake driver or soemthign elese we can see what the next bottel neck is
17:30:53 JayF ironic devstack plugin support for fake driver nodes is now #2 on my todo list (I got the PRs up removing 15k worth of retired driver code lines from Ironic)
17:31:02 sean-k-mooney JayF: we could as you suggested skip the placement management entirly in nova and delegate that to ironic but i dont know if that will really help
17:31:26 sean-k-mooney we may need to consider other change liek batch updating placement which may need an api change
17:31:31 JayF sean-k-mooney: basically in the places we poll Ironic currently, any way to switch that to something more evented
17:31:37 JayF sean-k-mooney: or batching into placement, yep
17:31:58 sean-k-mooney well yes so we coudl extend the nova external event api for ironic
17:32:00 JayF sean-k-mooney: either way once we pluck the low hanging fruit I want us to take a look at doing the bigger stuff to help; myself and clif can be the hands making the action work
17:32:08 sean-k-mooney that woudl allow use to get ride of poolign for power state for example
17:32:11 JayF I just don't always know which way is up in Nova :)
17:32:54 sean-k-mooney JayF: if you are not familary with https://docs.openstack.org/api-ref/compute/#create-external-events-os-server-external-events
17:33:19 sean-k-mooney that is how cidner and nova tell nova that the sate of soemthign has changed or a workflow has completed in an event based manner
17:33:44 JayF Interesting
17:33:53 sean-k-mooney power-update might actuly be used by ironic already
17:34:06 sean-k-mooney but there is no reason that we coudl not add other event if we had a reason too
17:34:07 JayF I am 99.999999% it's not
17:34:46 sean-k-mooney im just tyring to think what ealse is power related that it could be
17:34:57 JayF yep we 100% do
17:35:00 sean-k-mooney https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id69
17:35:00 JayF TIL
17:35:27 JayF https://opendev.org/openstack/ironic/src/branch/master/ironic/common/nova.py#L91
17:35:51 sean-k-mooney yep so i think this was added so we can turn off the power monitring perodic in ironic deployments
17:35:52 JayF if we could just trigger update of placement relevant info from Ironic e.g. on certain state changes
17:36:12 JayF I don't care if any of that comm goes Ironic<>Nova<>Placement or Ironic<>Placement
17:36:30 sean-k-mooney well ironic can go do placmeent adn do some limited things today
17:36:43 JayF especially if we were able to somehow get those calls to impact the node cache inside the virt driver itself
17:37:01 sean-k-mooney JayF: so if you tell me waht you want to update there i can say if it doable today or not without nova changes
17:37:05 JayF because then we can do the "reconciliation job on a timer" change like you did that I already -1'd
17:37:50 JayF sean-k-mooney: I don't know that answer (yet); but whatever happens when a node goes unavailable (e.g. cleaning or move out of AVAILABLE state) and whatever happens when a node goes available (anything->AVAILABLE state)
17:38:16 JayF Plus maybe something that would trigger on a node trait add to expose that trait to placement
17:38:29 sean-k-mooney so the way to model that in placement today is to set reserved=total on the rp
17:38:49 sean-k-mooney that is something that ironic can do directly
17:38:58 JayF Right now my goal is to get all these written down so we can evaluate them after the lower hanging fruit have been plucked
17:39:09 JayF because hopefully by then we'll have devstack w/fake driver nodes
17:39:10 sean-k-mooney ack sound good
17:39:19 JayF and can have real data for what is helpful vs what just seems helpful at a surface read
17:43:58 sean-k-mooney JayF: you know if we do deploy with the fake driver with a 1000s inically devstack is definlly going to cail because it will time out waiting for nova-comptue to start :P
17:44:25 sean-k-mooney so maybe ill try with 10 and then add them once it up
17:44:42 JayF sean-k-mooney: Ironic has a long, long, long history of knowing where all devstack timeouts are so we can set them high enough to make an infra team member weep
17:44:52 JayF lol
17:45:15 JayF sean-k-mooney: maybe add 1000 fake nodes, only put 3 of them in a shard managed by nova-compute by default or something :D either way I'll figure it out
17:47:05 JayF sean-k-mooney: clif: I put some notes in here https://etherpad.opendev.org/p/ironic-nova-compute-startup-2026.2
17:47:47 sean-k-mooney clif: if you want to rebase of rearrage those wip patches feel free
17:48:29 sean-k-mooney i wont get back to them for a while but if they are helpful to you go for it
17:51:06 clif thank you, will do
18:51:17 melwitt gmaan: hey, I have a hopefully easy patch to fix openstacksdk usage for using endpoint discovery for unified limits if you might be able to look at your convenience https://review.opendev.org/c/openstack/nova/+/975106
19:04:00 erlon sean-k-mooney++ melwitt++ thanks for the througly review on the RBD XML update functionality for migration process patch
19:05:27 sean-k-mooney erlon: im not sur eif keeping the sepreate small funciton for nova provisioned rbd storage is bettere or not for what its worth
19:05:34 sean-k-mooney i could be preswaded either way
19:05:56 sean-k-mooney but i woudl really prefer to see a functional repoducer for that
19:07:50 erlon melwitt: did you tried the XML "<ip family="ipv6" address="[2001:db8:ca2:2::1]" prefix="64"/>" with a ipv6 addr without the []? did it pass the checks?
19:10:44 erlon sean-k-mooney: you mean to have that code inside other function like _update_volume_xml?
19:11:34 erlon I believe that having the code in a separate function is better for readability and testing
19:14:19 erlon Another thing I was considering, is that when I brought it up in the meeting, we discussed wanting another managed command to perform the bulk update. I didn't start to think about how I'll do that implementation, but having things modularized usually help to reuse code.
19:15:55 sean-k-mooney erlon: my conder is driffet between the nova and cidner voulem management
19:16:27 sean-k-mooney erlon: like currently you are just updating the xml but not th bdms right
19:16:52 melwitt erlon: yes I did I pasted the output in the code block of my comment, I ran the virt-xml-validate CLI without brackets (passed) and with brackets (failed)
19:16:54 sean-k-mooney i dont recall if we store rbd info in teh bdms for nova provisioned stroage
19:17:22 sean-k-mooney melwitt: its possible that libvirt version depenent
19:17:23 melwitt no we don't, BDMs with rbd info is for cinder volume only
19:17:40 sean-k-mooney ok then that not an issue with the current patch so
19:18:05 melwitt sean-k-mooney: yes true. it would be ideal if we could test it for real somehow but regardless I think it's safest to do it without brackets bc that's how XML is currently generated for rbd
19:18:26 sean-k-mooney we have a tempest-ipv6 job in our gate
19:18:32 sean-k-mooney i just doen not ahve ceph in it
19:18:45 sean-k-mooney but i think we can look at that in a followup
19:18:54 melwitt I posted a comment on the review but I am wondering if the patch is actually needed right? bc AFAICT all cases should be covered already for nova local/ephemeral rbd
19:19:33 melwitt I wonder if people just want the patch to force an update to XML right away instead of waiting for a lifecycle event. just not sure I get the use case
19:20:10 sean-k-mooney so its not if you reboot the vm
19:20:35 sean-k-mooney but if we try to live migrate with the old mons i think the dest vm will fail to connect to the storage
19:20:45 melwitt ceph handles all of mon IP update transparently while the guest is running and it's not clear from the bug report if the lack of XML update there results in a failure or error of some kind
19:20:49 melwitt right and if you reboot the VM the existing code already grabs fresh mon IPs from ceph
19:20:49 sean-k-mooney and the migration will fail as a result if those mons are note aviabel
19:21:07 melwitt ok, that is what I wanted to know. I wish they had pasted the error in the bug
19:21:44 sean-k-mooney ya so unless libvirt is proxiyng that info without telling us i suspect that is the failure mode
19:21:48 melwitt ok, if it's actually a failure and not just a "I will feel better if the XML is updated now vs later" then I think the patch is useful
19:21:59 erlon melwitt, what would be the lifecicle events that would update that?
19:23:15 melwitt erlon: basically anything that calls _hard_reboot() internally so a stop/start, a hard reboot, an unshelve, I'm not sure what else off the top of my head. the main point is as long as the guest is running, the xml does not need an update, that's for later when it's starting up from being shutdown
19:23:23 erlon afic, ephemeral rbds are not updated unless the instance is sheved/unshelved

Earlier   Later