Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2026-03-23
16:40:25 sean-k-mooney https://review.opendev.org/c/openstack/nova/+/980679 and https://review.opendev.org/c/openstack/nova/+/980676/1
16:40:30 sean-k-mooney gibi: yep
16:40:48 sean-k-mooney so for 1000 noes we copery the resouce provider for the first one 999 times i blieve
16:41:00 gibi ohh I see why that is bad :)
16:41:15 dansmith this totally doesn't surprise me :)
16:41:24 Uggla (OO°
16:41:28 Uggla (OO)
16:41:37 sean-k-mooney anyway im not sayign the poc patches are correct
16:41:40 dansmith like I bet we probably have some things like that in runtime code too
16:41:43 sean-k-mooney but locally they do sped thing up alot
16:42:16 sean-k-mooney ya so this deepcopy is in get_provider_tree_and_ensure_root
16:42:25 gibi sean-k-mooney: my only question is if this is the real bottleneck, ie the code, or is there any nova-ironic interaction at startup that is inherently slow due to network?
16:43:08 gibi I imagine 1000 tree memcopy is still faster than 1000 network call to ironic
16:43:18 sean-k-mooney on my laptop the delta between the first patch and last goes form 2 and ahalf minute sdown to 40 ish seconds
16:43:31 sean-k-mooney i didnt try tweakign the latency ineject for the api calls
16:43:51 sean-k-mooney but the similater in thory can add in some it defautl to 0 for the get node call at the momemnt
16:44:07 gibi I though you have it in funct test, I'm sure that mock ironic API calls
16:44:17 dansmith sean-k-mooney: and lots of cpu usage I guess?
16:44:44 sean-k-mooney yep as i said i only spent a few hour on this at the weekend but i wanted to see if we coudl guest some emperical data
16:44:46 gibi anyhow. Improvement is good. Question remains if we improved the critical path or not
16:44:54 sean-k-mooney before the ironic cross project session
16:45:14 sean-k-mooney the other atpch that helps is https://review.opendev.org/c/openstack/nova/+/980679/1
16:45:26 sean-k-mooney that kicks out the per node operation into a thread pool
16:45:35 dansmith well, and if we improved it without introducing other issues (like the cache patch seems the easiest, but as <human> points out it may be problematic)
16:45:39 sean-k-mooney https://review.opendev.org/c/openstack/nova/+/980680/1
16:46:00 sean-k-mooney is a basi update to the reprot before/after the sersise but as i said i have not fully validate this as correct
16:46:09 sean-k-mooney its easy to make thing fast if correctness is not a requirement
16:46:21 dansmith and honestly two minutes to start up with lots of nodes may not be unreasonable, but certainly sucks if we can avoid it (especially if it's just burning cpu time copying stuff)
16:46:38 sean-k-mooney dansmith: so on real ironci system its 15+ minutes
16:46:57 dansmith yeah, uncool :)
16:47:03 sean-k-mooney based on the commit messge i lied N=1000: 42.1 s (42.1 ms/node) — was 4 min 4 s (244 ms/node)
16:47:12 sean-k-mooney so it was 4 minutes even without any network overhead
16:47:31 sean-k-mooney i will also note the bigest boteel neck on the 1000 node case was in placmenet
16:47:42 sean-k-mooney in the jsonschema validation
16:47:57 sean-k-mooney but that partly an artifact fo me using the palcement fixture
16:48:07 dansmith I think we can also put #squarepegroundhole on some of this
16:49:02 sean-k-mooney anyway i just wanted to share one of the experiemts i did. im not sure if we coudl eveolve this simulator into a gerneicly useful perf tool
16:49:30 sean-k-mooney dansmith: you had asked about a schduer simultor in the past so this might be a foundation to build that if we had time
16:49:50 Uggla dansmith #squarepegroundhole what's that ?
16:49:52 dansmith I guess maybe I'd like to have JayF (a mere human) look over this for ironic gotchas before we look too deep at the other things
16:49:54 dansmith sean-k-mooney: yep, for sure
16:50:10 JayF the biggest issue is that it's dangerous to heavily lean on a cache
16:50:19 dansmith Uggla: ironic as a driver in nova is a square peg in a round hole (doesn't fit very well and has lots of ironic-specific issues like this)
16:50:33 JayF because it completely breaks use cases where "node completes cleaning -> immediate re-deploy" which while it's not a very openstack-y use case, is very common
16:50:43 sean-k-mooney Uggla: the reall issues is the compute agent was desgisn to have 1 pre compute node. the ironic driver has 1 agent for 1000s
16:50:50 dansmith this ^
16:50:59 JayF we'd need ironic to close that loop on more of an event-driven basis vs a poll to fix that concern before relying more heabily on cache
16:51:17 JayF admittedly my initial email was written assuming "ironic updates more stuff about ironic nodes" may be the path, I didn't expect sean-k-mooney to find low-hanging-fruit like he has
16:51:25 sean-k-mooney JayF: ya i saw your coment on the cach. i think we shoudl actully start lookign at riping out the hasring stuff
16:51:41 sean-k-mooney but also have a wider converation on this in general
16:52:02 JayF even just if we put in knobs to let operators make the tradeoff, it might help, idk
16:52:14 JayF I'll ensure I review those before the cross-project session
16:52:26 JayF if a sync between just us two to refine before PTG is valuable, we can do that too
16:52:59 sean-k-mooney ya perhaps. the one caveat ill say is im not activly working on this
16:53:12 sean-k-mooney but you peaked my interest enouch to see if there was anythign obviously wrong
16:53:27 sean-k-mooney or simple improvemtn we coudl make
16:54:03 JayF It's a major issue for Ironic/Nova users; which are 25% of nova users per the survey :) hopefully we can catch the eyes of someone who has expertise, even just to lay the path out for me and/or my team (cid or clif) to implement
16:54:46 sean-k-mooney JayF: i can chat to you after the meetign more if you like but it might be interestign to see if we could perhaps replciate this in devstack with the ironic fake driver
16:55:00 JayF yeah, that sounds like a good path
16:55:09 JayF and I've wanted to make ironic easier to hook up in devstack with fake driver
16:55:12 sean-k-mooney i.e. to see if the imovpments in this emulated env traslate to a real env
16:55:14 JayF maybe I'll spike on that between now:ptg
16:56:07 sean-k-mooney Uggla: sorry that took more time then i was planning but i think this has been productive
16:56:27 JayF we should take Uggla to "why Ironic driver is awesome and terrible 101" some day, I think :P
16:56:48 Uggla JayF sure
16:56:51 dansmith let him enjoy life, gawd
16:57:03 dansmith that's like explaining santa
16:57:30 Uggla Are we good ?
16:57:33 JayF "And here's _tooth_fairy()"
16:57:42 JayF yes, ty Uggla :)
16:57:48 Uggla Anything else ?
16:58:14 Uggla seems not.
16:58:15 dansmith JayF: wait, what's your point about the tooth fairy? If you have a contact, I'm still waiting for $10
16:58:45 Uggla Quick update on bug scrubbing because we are at he top of the hour.
16:58:53 Uggla #topic Bug scrubbing
16:58:59 Uggla #info up to 199 (-7)
16:59:09 JayF dansmith: I saw her walking out of "Budget Dentures". Scandalous! ;)
16:59:14 dansmith lol
16:59:25 Uggla I manage to go down 200.
16:59:33 Uggla #link: https://etherpad.opendev.org/p/nova-bug-triage-roster
16:59:50 Uggla #link: https://truc.uggla.fr/ if you want to see the trend.
17:00:32 Uggla JayF, dansmith, can tooth fairy do something for bugs ?
17:01:01 JayF If she is a metaphor for the Ironic driver, she can create them ;)
17:01:10 Uggla :)
17:01:56 Uggla Time to close.
17:02:10 Uggla thanks for joining this meeting. Have a nice day/evening and see you next week.
17:02:16 gibi o/
17:02:17 opendevmeet Log: https://meetings.opendev.org/meetings/nova/2026/nova.2026-03-23-16.00.log.html
17:02:17 opendevmeet Minutes (text): https://meetings.opendev.org/meetings/nova/2026/nova.2026-03-23-16.00.txt
17:02:17 opendevmeet Minutes: https://meetings.opendev.org/meetings/nova/2026/nova.2026-03-23-16.00.html
17:02:17 opendevmeet Meeting ended Mon Mar 23 17:02:17 2026 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
17:02:17 Uggla #endmeeting
17:02:24 lajoskatona o/
17:02:24 elodilles same to you, thanks o/
18:27:17 sean-k-mooney melwitt: thanks for taking a look at https://review.opendev.org/c/openstack/nova/+/975872/5
18:28:01 sean-k-mooney melwitt: gibi do you thinks we could try and get the repoducer patch merged in the next few days https://review.opendev.org/c/openstack/nova/+/975859/4 while we dicuss the fix in https://review.opendev.org/c/openstack/nova/+/975872
18:29:01 sean-k-mooney it woudl be nice to make some progress on this again. im hoping to complete that bug work in the next week or two
18:29:23 melwitt sean-k-mooney: ok yeah let me focus on that one then, I had started with the fix (obviously)
18:29:25 sean-k-mooney mainly to avoid having to load context on this over and over again
18:29:58 sean-k-mooney melwitt: ill definally look into your feedback on teh fix too

Earlier   Later