Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2026-03-26
12:56:24 priteau gibi: 2025.1 backport has merged, thanks! 2024.2 backport is ready too: https://review.opendev.org/c/openstack/nova/+/952004
13:29:51 opendevreview Balazs Gibizer proposed openstack/nova master: Default native threading for conductor and compute https://review.opendev.org/c/openstack/nova/+/982237
14:32:20 opendevreview Kamil Sambor proposed openstack/nova master: Replace eventlet with standard threading in novncproxy https://review.opendev.org/c/openstack/nova/+/976089
15:09:45 opendevreview Merged openstack/nova stable/2024.2: libvirt: Use common naming convention for ephemeral disk labels https://review.opendev.org/c/openstack/nova/+/952004
16:06:11 opendevreview Kamil Sambor proposed openstack/nova master: Replace eventlet with standard threading in novncproxy https://review.opendev.org/c/openstack/nova/+/976089
17:11:13 clif sean-k-mooney: could you give me some context on ProviderTree? does the tree represent a heirarchical relationship between nodes? I also see ProviderTree calls out the ironic driver specifically for having 'large number of roots'
17:15:09 sean-k-mooney clif: placement is a forset data stucre where we have a forest of provider that are them selve aranged in a tree stucture with root RPs representign a comptue node and potically nested resouce provider as childer repcesting gpus or pci device on that compute host
17:15:50 sean-k-mooney clif: in the context of ironic each ironic node is create in placement as a seperate resouce provider with invetory of 1 resouce class fo custom_baremtal_something
17:16:18 sean-k-mooney so a single nova compute agent with the ironic driver manage a set of trees of depht 1
17:16:35 clif ah, ok
17:16:48 sean-k-mooney the uuid of the placment resouce provider is the ironic node uuid which is alsoce the nova compute node uuid
17:17:20 sean-k-mooney so it a 1:1:1 mapping between compute_node in the nova db, resouce provider in the placment db and node in the ironic db
17:17:52 clif so each tree is a single ironic node and each tree's children is information about the ironic node?
17:18:00 sean-k-mooney clif: for almost every other dirver each nova comptue agent manages exactly 1 compute node and 1 root resouce provider
17:18:35 sean-k-mooney clif: for hte ironic diver we dont create childer sinc you cant subdevide ironic hosts
17:18:39 JayF (this is, I presume, the root cause for how so much perf was left on the table in the Ironic case?)
17:18:45 sean-k-mooney we could in the future but we didnt have a suecase today
17:19:03 sean-k-mooney JayF: yes more or less
17:20:00 sean-k-mooney JayF: the first botlenec (the deep copy of the providers manged by this compute agent)
17:20:19 sean-k-mooney is cheap if the number of provider is 1-8 max in 99% of cases
17:20:42 sean-k-mooney its really expensive if you have 100s or 100s espcially if your doing it in a loop
17:21:05 clif could we create a seperate provider implementation for ironic driver use?
17:21:06 sean-k-mooney JayF: so i think that is 1 of the hopefully simple things to fix
17:21:31 sean-k-mooney clif: we try not to have any virt driver awareness in teh compute manager or resouce tracker
17:21:41 sean-k-mooney and that is where the provider stuff lives
17:22:01 sean-k-mooney but short answer is yes i think we can optimize it
17:22:06 clif alright
17:22:10 sean-k-mooney jsut without the need for an if ironic check
17:22:39 sean-k-mooney clif: that is what https://review.opendev.org/c/openstack/nova/+/980676 is ment to do
17:23:08 sean-k-mooney specifically https://review.opendev.org/c/openstack/nova/+/980676/1/nova/compute/resource_tracker.py#1314
17:23:27 clif right, I'm trying to load enough nova context into my noggin to get to the point where I can understand what's going on with that patch, the perf sim, etc
17:24:34 sean-k-mooney when we get the provider tree for a spcific node that shoudl copy only that node RP insated fo every provider in our provider cash https://review.opendev.org/c/openstack/nova/+/980676/1/nova/scheduler/client/report.py#935
17:25:29 clif it seems reasonable on the surface
17:25:35 sean-k-mooney clif: ya that a non zeor amount of context to load
17:26:07 sean-k-mooney the other optimisation that i think has prommis is this patch https://review.opendev.org/c/openstack/nova/+/980679/1
17:26:19 sean-k-mooney so that that one does is very simple
17:26:47 sean-k-mooney today we have a for loop over every ironic node managed by this agent (limited via shard key or peer_list)
17:27:06 sean-k-mooney the patch just kick the processing of each node out into a thread pool
17:27:19 sean-k-mooney so you can actully do them in parallel an tune it via a config option
17:27:28 sean-k-mooney defaulting to 1 for backward compatiablity
17:29:08 JayF And I hope this is all building up, once the identified low hanging fruit are picked, to a deeper look at if there's things we can do to improve it further. I am skeptical that this will be enough to bring us within an acceptable startup time :)
17:29:17 clif I will continue to delve
17:29:28 clif thanks for the context sean
17:30:17 sean-k-mooney JayF: these were the hotspots in the current profiling but if we can get a beter repoducer with the ironic fake driver or soemthign elese we can see what the next bottel neck is
17:30:53 JayF ironic devstack plugin support for fake driver nodes is now #2 on my todo list (I got the PRs up removing 15k worth of retired driver code lines from Ironic)
17:31:02 sean-k-mooney JayF: we could as you suggested skip the placement management entirly in nova and delegate that to ironic but i dont know if that will really help
17:31:26 sean-k-mooney we may need to consider other change liek batch updating placement which may need an api change
17:31:31 JayF sean-k-mooney: basically in the places we poll Ironic currently, any way to switch that to something more evented
17:31:37 JayF sean-k-mooney: or batching into placement, yep
17:31:58 sean-k-mooney well yes so we coudl extend the nova external event api for ironic
17:32:00 JayF sean-k-mooney: either way once we pluck the low hanging fruit I want us to take a look at doing the bigger stuff to help; myself and clif can be the hands making the action work
17:32:08 sean-k-mooney that woudl allow use to get ride of poolign for power state for example
17:32:11 JayF I just don't always know which way is up in Nova :)
17:32:54 sean-k-mooney JayF: if you are not familary with https://docs.openstack.org/api-ref/compute/#create-external-events-os-server-external-events
17:33:19 sean-k-mooney that is how cidner and nova tell nova that the sate of soemthign has changed or a workflow has completed in an event based manner
17:33:44 JayF Interesting
17:33:53 sean-k-mooney power-update might actuly be used by ironic already
17:34:06 sean-k-mooney but there is no reason that we coudl not add other event if we had a reason too
17:34:07 JayF I am 99.999999% it's not
17:34:46 sean-k-mooney im just tyring to think what ealse is power related that it could be
17:34:57 JayF yep we 100% do
17:35:00 sean-k-mooney https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id69
17:35:00 JayF TIL
17:35:27 JayF https://opendev.org/openstack/ironic/src/branch/master/ironic/common/nova.py#L91
17:35:51 sean-k-mooney yep so i think this was added so we can turn off the power monitring perodic in ironic deployments
17:35:52 JayF if we could just trigger update of placement relevant info from Ironic e.g. on certain state changes
17:36:12 JayF I don't care if any of that comm goes Ironic<>Nova<>Placement or Ironic<>Placement
17:36:30 sean-k-mooney well ironic can go do placmeent adn do some limited things today
17:36:43 JayF especially if we were able to somehow get those calls to impact the node cache inside the virt driver itself
17:37:01 sean-k-mooney JayF: so if you tell me waht you want to update there i can say if it doable today or not without nova changes
17:37:05 JayF because then we can do the "reconciliation job on a timer" change like you did that I already -1'd
17:37:50 JayF sean-k-mooney: I don't know that answer (yet); but whatever happens when a node goes unavailable (e.g. cleaning or move out of AVAILABLE state) and whatever happens when a node goes available (anything->AVAILABLE state)
17:38:16 JayF Plus maybe something that would trigger on a node trait add to expose that trait to placement
17:38:29 sean-k-mooney so the way to model that in placement today is to set reserved=total on the rp
17:38:49 sean-k-mooney that is something that ironic can do directly
17:38:58 JayF Right now my goal is to get all these written down so we can evaluate them after the lower hanging fruit have been plucked
17:39:09 JayF because hopefully by then we'll have devstack w/fake driver nodes
17:39:10 sean-k-mooney ack sound good
17:39:19 JayF and can have real data for what is helpful vs what just seems helpful at a surface read
17:43:58 sean-k-mooney JayF: you know if we do deploy with the fake driver with a 1000s inically devstack is definlly going to cail because it will time out waiting for nova-comptue to start :P
17:44:25 sean-k-mooney so maybe ill try with 10 and then add them once it up
17:44:42 JayF sean-k-mooney: Ironic has a long, long, long history of knowing where all devstack timeouts are so we can set them high enough to make an infra team member weep
17:44:52 JayF lol
17:45:15 JayF sean-k-mooney: maybe add 1000 fake nodes, only put 3 of them in a shard managed by nova-compute by default or something :D either way I'll figure it out
17:47:05 JayF sean-k-mooney: clif: I put some notes in here https://etherpad.opendev.org/p/ironic-nova-compute-startup-2026.2
17:47:47 sean-k-mooney clif: if you want to rebase of rearrage those wip patches feel free
17:48:29 sean-k-mooney i wont get back to them for a while but if they are helpful to you go for it
17:51:06 clif thank you, will do
18:51:17 melwitt gmaan: hey, I have a hopefully easy patch to fix openstacksdk usage for using endpoint discovery for unified limits if you might be able to look at your convenience https://review.opendev.org/c/openstack/nova/+/975106
19:04:00 erlon sean-k-mooney++ melwitt++ thanks for the througly review on the RBD XML update functionality for migration process patch
19:05:27 sean-k-mooney erlon: im not sur eif keeping the sepreate small funciton for nova provisioned rbd storage is bettere or not for what its worth
19:05:34 sean-k-mooney i could be preswaded either way
19:05:56 sean-k-mooney but i woudl really prefer to see a functional repoducer for that
19:07:50 erlon melwitt: did you tried the XML "<ip family="ipv6" address="[2001:db8:ca2:2::1]" prefix="64"/>" with a ipv6 addr without the []? did it pass the checks?
19:10:44 erlon sean-k-mooney: you mean to have that code inside other function like _update_volume_xml?
19:11:34 erlon I believe that having the code in a separate function is better for readability and testing
19:14:19 erlon Another thing I was considering, is that when I brought it up in the meeting, we discussed wanting another managed command to perform the bulk update. I didn't start to think about how I'll do that implementation, but having things modularized usually help to reuse code.
19:15:55 sean-k-mooney erlon: my conder is driffet between the nova and cidner voulem management
19:16:27 sean-k-mooney erlon: like currently you are just updating the xml but not th bdms right
19:16:52 melwitt erlon: yes I did I pasted the output in the code block of my comment, I ran the virt-xml-validate CLI without brackets (passed) and with brackets (failed)

Earlier   Later