| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-03-23 | |||
| 16:46:38 | sean-k-mooney | dansmith: so on real ironci system its 15+ minutes | |
| 16:46:57 | dansmith | yeah, uncool :) | |
| 16:47:03 | sean-k-mooney | based on the commit messge i lied N=1000: 42.1 s (42.1 ms/node) — was 4 min 4 s (244 ms/node) | |
| 16:47:12 | sean-k-mooney | so it was 4 minutes even without any network overhead | |
| 16:47:31 | sean-k-mooney | i will also note the bigest boteel neck on the 1000 node case was in placmenet | |
| 16:47:42 | sean-k-mooney | in the jsonschema validation | |
| 16:47:57 | sean-k-mooney | but that partly an artifact fo me using the palcement fixture | |
| 16:48:07 | dansmith | I think we can also put #squarepegroundhole on some of this | |
| 16:49:02 | sean-k-mooney | anyway i just wanted to share one of the experiemts i did. im not sure if we coudl eveolve this simulator into a gerneicly useful perf tool | |
| 16:49:30 | sean-k-mooney | dansmith: you had asked about a schduer simultor in the past so this might be a foundation to build that if we had time | |
| 16:49:50 | Uggla | dansmith #squarepegroundhole what's that ? | |
| 16:49:52 | dansmith | I guess maybe I'd like to have JayF (a mere human) look over this for ironic gotchas before we look too deep at the other things | |
| 16:49:54 | dansmith | sean-k-mooney: yep, for sure | |
| 16:50:10 | JayF | the biggest issue is that it's dangerous to heavily lean on a cache | |
| 16:50:19 | dansmith | Uggla: ironic as a driver in nova is a square peg in a round hole (doesn't fit very well and has lots of ironic-specific issues like this) | |
| 16:50:33 | JayF | because it completely breaks use cases where "node completes cleaning -> immediate re-deploy" which while it's not a very openstack-y use case, is very common | |
| 16:50:43 | sean-k-mooney | Uggla: the reall issues is the compute agent was desgisn to have 1 pre compute node. the ironic driver has 1 agent for 1000s | |
| 16:50:50 | dansmith | this ^ | |
| 16:50:59 | JayF | we'd need ironic to close that loop on more of an event-driven basis vs a poll to fix that concern before relying more heabily on cache | |
| 16:51:17 | JayF | admittedly my initial email was written assuming "ironic updates more stuff about ironic nodes" may be the path, I didn't expect sean-k-mooney to find low-hanging-fruit like he has | |
| 16:51:25 | sean-k-mooney | JayF: ya i saw your coment on the cach. i think we shoudl actully start lookign at riping out the hasring stuff | |
| 16:51:41 | sean-k-mooney | but also have a wider converation on this in general | |
| 16:52:02 | JayF | even just if we put in knobs to let operators make the tradeoff, it might help, idk | |
| 16:52:14 | JayF | I'll ensure I review those before the cross-project session | |
| 16:52:26 | JayF | if a sync between just us two to refine before PTG is valuable, we can do that too | |
| 16:52:59 | sean-k-mooney | ya perhaps. the one caveat ill say is im not activly working on this | |
| 16:53:12 | sean-k-mooney | but you peaked my interest enouch to see if there was anythign obviously wrong | |
| 16:53:27 | sean-k-mooney | or simple improvemtn we coudl make | |
| 16:54:03 | JayF | It's a major issue for Ironic/Nova users; which are 25% of nova users per the survey :) hopefully we can catch the eyes of someone who has expertise, even just to lay the path out for me and/or my team (cid or clif) to implement | |
| 16:54:46 | sean-k-mooney | JayF: i can chat to you after the meetign more if you like but it might be interestign to see if we could perhaps replciate this in devstack with the ironic fake driver | |
| 16:55:00 | JayF | yeah, that sounds like a good path | |
| 16:55:09 | JayF | and I've wanted to make ironic easier to hook up in devstack with fake driver | |
| 16:55:12 | sean-k-mooney | i.e. to see if the imovpments in this emulated env traslate to a real env | |
| 16:55:14 | JayF | maybe I'll spike on that between now:ptg | |
| 16:56:07 | sean-k-mooney | Uggla: sorry that took more time then i was planning but i think this has been productive | |
| 16:56:27 | JayF | we should take Uggla to "why Ironic driver is awesome and terrible 101" some day, I think :P | |
| 16:56:48 | Uggla | JayF sure | |
| 16:56:51 | dansmith | let him enjoy life, gawd | |
| 16:57:03 | dansmith | that's like explaining santa | |
| 16:57:30 | Uggla | Are we good ? | |
| 16:57:33 | JayF | "And here's _tooth_fairy()" | |
| 16:57:42 | JayF | yes, ty Uggla :) | |
| 16:57:48 | Uggla | Anything else ? | |
| 16:58:14 | Uggla | seems not. | |
| 16:58:15 | dansmith | JayF: wait, what's your point about the tooth fairy? If you have a contact, I'm still waiting for $10 | |
| 16:58:45 | Uggla | Quick update on bug scrubbing because we are at he top of the hour. | |
| 16:58:53 | Uggla | #topic Bug scrubbing | |
| 16:58:59 | Uggla | #info up to 199 (-7) | |
| 16:59:09 | JayF | dansmith: I saw her walking out of "Budget Dentures". Scandalous! ;) | |
| 16:59:14 | dansmith | lol | |
| 16:59:25 | Uggla | I manage to go down 200. | |
| 16:59:33 | Uggla | #link: https://etherpad.opendev.org/p/nova-bug-triage-roster | |
| 16:59:50 | Uggla | #link: https://truc.uggla.fr/ if you want to see the trend. | |
| 17:00:32 | Uggla | JayF, dansmith, can tooth fairy do something for bugs ? | |
| 17:01:01 | JayF | If she is a metaphor for the Ironic driver, she can create them ;) | |
| 17:01:10 | Uggla | :) | |
| 17:01:56 | Uggla | Time to close. | |
| 17:02:10 | Uggla | thanks for joining this meeting. Have a nice day/evening and see you next week. | |
| 17:02:16 | gibi | o/ | |
| 17:02:17 | Uggla | #endmeeting | |
| 17:02:17 | opendevmeet | Meeting ended Mon Mar 23 17:02:17 2026 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | |
| 17:02:17 | opendevmeet | Minutes: https://meetings.opendev.org/meetings/nova/2026/nova.2026-03-23-16.00.html | |
| 17:02:17 | opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/nova/2026/nova.2026-03-23-16.00.txt | |
| 17:02:17 | opendevmeet | Log: https://meetings.opendev.org/meetings/nova/2026/nova.2026-03-23-16.00.log.html | |
| 17:02:24 | elodilles | same to you, thanks o/ | |
| 17:02:24 | lajoskatona | o/ | |
| 18:27:17 | sean-k-mooney | melwitt: thanks for taking a look at https://review.opendev.org/c/openstack/nova/+/975872/5 | |
| 18:28:01 | sean-k-mooney | melwitt: gibi do you thinks we could try and get the repoducer patch merged in the next few days https://review.opendev.org/c/openstack/nova/+/975859/4 while we dicuss the fix in https://review.opendev.org/c/openstack/nova/+/975872 | |
| 18:29:01 | sean-k-mooney | it woudl be nice to make some progress on this again. im hoping to complete that bug work in the next week or two | |
| 18:29:23 | melwitt | sean-k-mooney: ok yeah let me focus on that one then, I had started with the fix (obviously) | |
| 18:29:25 | sean-k-mooney | mainly to avoid having to load context on this over and over again | |
| 18:29:58 | sean-k-mooney | melwitt: ill definally look into your feedback on teh fix too | |
| 18:30:10 | sean-k-mooney | just want to see if we can make incremental progress | |
| 18:31:02 | gibi | sean-k-mooney: I don't immediately have capacity for that patch. Please ping me in couple of days | |
| 18:31:16 | melwitt | sean-k-mooney: sure. thanks for letting me know | |
| 18:31:21 | sean-k-mooney | melwitt: since i started wroking on this i have had time to look at the cybrog quota code and foudn its entrily non fucntional https://bugs.launchpad.net/openstack-cyborg/+bug/2143943 | |
| 18:31:40 | sean-k-mooney | gibi: ack will do | |
| 18:32:04 | melwitt | sean-k-mooney: wow that's wild | |
| 18:32:43 | sean-k-mooney | i have a writeups attached https://launchpadlibrarian.net/851194002/quota-system.md but tldr is the safolding for qutas was built in the v1 api but never ported to v2 | |
| 18:33:26 | sean-k-mooney | at this point i think im just oging to rip it out since it does not work and if we need to have native supprot in cybrog indepent of th enova fix go staight to unified limits | |
| 18:35:01 | melwitt | yeah I think that's probably the best move to just go straight to unified limits | |
| 18:35:27 | melwitt | if fixing the current thing is not trivial | |
| 18:35:52 | sean-k-mooney | well its worked for 10 months and has been broken since 2019 | |
| 18:36:36 | sean-k-mooney | so its not so much how hard is fixing it its is it worth addign a seperate quota api to cyborg at all at this point if we can just use unified limits | |
| 18:36:54 | sean-k-mooney | the entiretly of the v1 api was deleted years ago and it never existed in v2 | |
| 18:36:55 | dansmith | unified limits is easy that way.. saves a lot of api effort | |
| 18:37:18 | melwitt | sean-k-mooney: adding a separate API = not trivial :) so yeah | |
| 18:37:57 | sean-k-mooney | in cybrog qutoa was also only configurable via config before when it did work so there is also no prior art to revive | |
| 18:38:16 | dansmith | that's like glance | |
| 18:38:33 | dansmith | like glance _was_ before unified limits, I should say | |
| 18:38:42 | sean-k-mooney | ack | |
| 18:39:15 | dansmith | even if you had prior art, unified limits instantly makes you more modern and compatible with the way the other projects are going, IMHO, so that's a reason right there | |
| 18:40:29 | sean-k-mooney | right now im not fucosing on cybrog as a standalone service "without nova" so if we fix the enfocement on the nova side then that closes the gap although at some point i shoudl add quta checks indepenet of nova spawn/resize code before we add attach/detach supprot | |
| 18:41:44 | sean-k-mooney | anyway out of scope of nova but this will eventually get doen after the other higher priorty items | |
| 18:42:57 | sean-k-mooney | ARQ binding is when a resouce is actully assigne dto an instnace so that is where we should be checking to prevent over quota usage longterm. | |
| 18:43:38 | sean-k-mooney | i.e form parallel nova instance cretes | |
| 20:06:56 | opendevreview | Merged openstack/nova master: Move py312-threading to py313 https://review.opendev.org/c/openstack/nova/+/980458 | |
| #openstack-nova - 2026-03-24 | |||
| 06:59:17 | gokhan | Hi everyone, I’m working on designing a large-scale architecture and could use some advice from the community. So far, I’ve been running about 50-100 hypervisors with 3 controllers without any issues, but I'm now planning to scale up to 1,000 nodes and I'm wondering what the best path forward is. At what point does Cells v2 become a 'must-have' rather than just an option? For those of you running massive setups, how many compute nodes do | |
| 06:59:17 | gokhan | you typically manage per cell to keep things stable and limit the blast radius? I'd love to hear your experiences on whether a single-cell setup can actually handle 1,000 nodes with the right tuning, or if I should definitely start splitting things up now. Thanks a lot for any tips! | |
| 08:13:58 | gibi | gokhan: I think 1000 node in a single cell is a huge strech. Might work if you have a fairly static workload. I think our non official recommendation is to start considering cells around 200 nodes | |