| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-03-23 | |||
| 16:17:46 | Uggla | thanks gibi. | |
| 16:18:06 | Uggla | #topic Nova using openstack sdk for neutron | |
| 16:18:28 | lajoskatona | o/ thanks | |
| 16:18:40 | lajoskatona | I pushed a new ps for security-groups: https://review.opendev.org/c/openstack/nova/+/981141 | |
| 16:19:06 | lajoskatona | and I polished the older patches also to make them ready for review for early H | |
| 16:19:34 | lajoskatona | The security-group one is wip as it seems I still break something :-) | |
| 16:20:12 | lajoskatona | After that I hope I can check what remains, that is for sure the port-binding API and some other small things, so that will come soon I hope | |
| 16:20:34 | lajoskatona | that's it for the SDK work | |
| 16:21:01 | sean-k-mooney | ack. so its breaking in tempest but not in the unit/fucntional tests | |
| 16:21:32 | lajoskatona | exactly, so I have to read more logs | |
| 16:22:59 | Uggla | oh not cool. | |
| 16:23:52 | lajoskatona | life, if you touch something you break something :-( | |
| 16:24:01 | Uggla | sometimes. | |
| 16:24:32 | Uggla | #topic Open discussion | |
| 16:25:13 | Uggla | I forget to say something in review priorities. | |
| 16:25:55 | Uggla | https://review.opendev.org/q/topic:%22bp/generalize-sev-code%22 looks good to me. Except latest patch we agreed to remove from this serie. | |
| 16:26:25 | Uggla | gibi from you point of view anything still not ok with the serie ? | |
| 16:26:26 | gibi | I have to go back to that series | |
| 16:26:34 | gibi | I lost context over last week | |
| 16:27:35 | Uggla | thanks gibi, it will be great to merge that before starting new CC stuffs. | |
| 16:27:38 | gibi | I had some negative feedback before but I guess that was addressed in the meantime | |
| 16:28:01 | gibi | Uggla: sure, but that also needs two cores ;) | |
| 16:28:20 | Uggla | gibi, yes I have not noticed remaining open things. | |
| 16:28:51 | Uggla | I know, bauzas maybe ? | |
| 16:29:38 | bauzas | OK, drop me as a reviewer | |
| 16:30:13 | Uggla | OK done for this one. | |
| 16:30:45 | Uggla | Second, it seems I will be PTL again for H cycle. | |
| 16:31:01 | gibi | Uggla: thanks for your continued service | |
| 16:31:27 | elodilles | yepp, thanks Uggla \o/ | |
| 16:31:51 | Uggla | thank you. | |
| 16:32:50 | Uggla | Third point, we may have 2 x potential new slots for upstream bug triage. | |
| 16:33:46 | Uggla | It is not settled at the moment. I need to think about how we will organize that. | |
| 16:34:17 | Uggla | I will keep you posted via the ML | |
| 16:35:11 | Uggla | that's all for me. Does anyone else want to discuss something ? | |
| 16:36:11 | sean-k-mooney | am more a quick fyi | |
| 16:36:18 | Uggla | sure | |
| 16:36:48 | sean-k-mooney | so JayF raised a long stanidng issue with the ironic driver (startup time) | |
| 16:37:03 | sean-k-mooney | i pent a few hourse playing aroudn with https://review.opendev.org/q/topic:%22startup-time%22 | |
| 16:37:32 | sean-k-mooney | baisly i create a poc of a simulator to messuer the perfoamcne of it as we scaled out ironic nodes | |
| 16:38:06 | sean-k-mooney | this uses our functional tests infrastucer and then i ran some profileing of the starut and had ai analise it | |
| 16:38:38 | sean-k-mooney | so i just wanted to share that as it found we are don an quadtaric deep copy of the resouce providers | |
| 16:39:10 | sean-k-mooney | i dont know if this tool could be useful for other things liek similarting large env for schduler fileter or eventlet debuging | |
| 16:39:27 | sean-k-mooney | btu i just wanted to share that it exists as a review incase anyone found it interesting | |
| 16:39:46 | sean-k-mooney | the code quality/style is not what i woudl normally do | |
| 16:39:51 | sean-k-mooney | since this was just a quick poc | |
| 16:40:05 | sean-k-mooney | but it did produce some interesting results so ya that was all i had | |
| 16:40:21 | gibi | sean-k-mooney: does the deep-copy expensive due to multiple compute nodes per compute host? | |
| 16:40:25 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/980679 and https://review.opendev.org/c/openstack/nova/+/980676/1 | |
| 16:40:30 | sean-k-mooney | gibi: yep | |
| 16:40:48 | sean-k-mooney | so for 1000 noes we copery the resouce provider for the first one 999 times i blieve | |
| 16:41:00 | gibi | ohh I see why that is bad :) | |
| 16:41:15 | dansmith | this totally doesn't surprise me :) | |
| 16:41:24 | Uggla | (OO° | |
| 16:41:28 | Uggla | (OO) | |
| 16:41:37 | sean-k-mooney | anyway im not sayign the poc patches are correct | |
| 16:41:40 | dansmith | like I bet we probably have some things like that in runtime code too | |
| 16:41:43 | sean-k-mooney | but locally they do sped thing up alot | |
| 16:42:16 | sean-k-mooney | ya so this deepcopy is in get_provider_tree_and_ensure_root | |
| 16:42:25 | gibi | sean-k-mooney: my only question is if this is the real bottleneck, ie the code, or is there any nova-ironic interaction at startup that is inherently slow due to network? | |
| 16:43:08 | gibi | I imagine 1000 tree memcopy is still faster than 1000 network call to ironic | |
| 16:43:18 | sean-k-mooney | on my laptop the delta between the first patch and last goes form 2 and ahalf minute sdown to 40 ish seconds | |
| 16:43:31 | sean-k-mooney | i didnt try tweakign the latency ineject for the api calls | |
| 16:43:51 | sean-k-mooney | but the similater in thory can add in some it defautl to 0 for the get node call at the momemnt | |
| 16:44:07 | gibi | I though you have it in funct test, I'm sure that mock ironic API calls | |
| 16:44:17 | dansmith | sean-k-mooney: and lots of cpu usage I guess? | |
| 16:44:44 | sean-k-mooney | yep as i said i only spent a few hour on this at the weekend but i wanted to see if we coudl guest some emperical data | |
| 16:44:46 | gibi | anyhow. Improvement is good. Question remains if we improved the critical path or not | |
| 16:44:54 | sean-k-mooney | before the ironic cross project session | |
| 16:45:14 | sean-k-mooney | the other atpch that helps is https://review.opendev.org/c/openstack/nova/+/980679/1 | |
| 16:45:26 | sean-k-mooney | that kicks out the per node operation into a thread pool | |
| 16:45:35 | dansmith | well, and if we improved it without introducing other issues (like the cache patch seems the easiest, but as <human> points out it may be problematic) | |
| 16:45:39 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/980680/1 | |
| 16:46:00 | sean-k-mooney | is a basi update to the reprot before/after the sersise but as i said i have not fully validate this as correct | |
| 16:46:09 | sean-k-mooney | its easy to make thing fast if correctness is not a requirement | |
| 16:46:21 | dansmith | and honestly two minutes to start up with lots of nodes may not be unreasonable, but certainly sucks if we can avoid it (especially if it's just burning cpu time copying stuff) | |
| 16:46:38 | sean-k-mooney | dansmith: so on real ironci system its 15+ minutes | |
| 16:46:57 | dansmith | yeah, uncool :) | |
| 16:47:03 | sean-k-mooney | based on the commit messge i lied N=1000: 42.1 s (42.1 ms/node) — was 4 min 4 s (244 ms/node) | |
| 16:47:12 | sean-k-mooney | so it was 4 minutes even without any network overhead | |
| 16:47:31 | sean-k-mooney | i will also note the bigest boteel neck on the 1000 node case was in placmenet | |
| 16:47:42 | sean-k-mooney | in the jsonschema validation | |
| 16:47:57 | sean-k-mooney | but that partly an artifact fo me using the palcement fixture | |
| 16:48:07 | dansmith | I think we can also put #squarepegroundhole on some of this | |
| 16:49:02 | sean-k-mooney | anyway i just wanted to share one of the experiemts i did. im not sure if we coudl eveolve this simulator into a gerneicly useful perf tool | |
| 16:49:30 | sean-k-mooney | dansmith: you had asked about a schduer simultor in the past so this might be a foundation to build that if we had time | |
| 16:49:50 | Uggla | dansmith #squarepegroundhole what's that ? | |
| 16:49:52 | dansmith | I guess maybe I'd like to have JayF (a mere human) look over this for ironic gotchas before we look too deep at the other things | |
| 16:49:54 | dansmith | sean-k-mooney: yep, for sure | |
| 16:50:10 | JayF | the biggest issue is that it's dangerous to heavily lean on a cache | |
| 16:50:19 | dansmith | Uggla: ironic as a driver in nova is a square peg in a round hole (doesn't fit very well and has lots of ironic-specific issues like this) | |
| 16:50:33 | JayF | because it completely breaks use cases where "node completes cleaning -> immediate re-deploy" which while it's not a very openstack-y use case, is very common | |
| 16:50:43 | sean-k-mooney | Uggla: the reall issues is the compute agent was desgisn to have 1 pre compute node. the ironic driver has 1 agent for 1000s | |
| 16:50:50 | dansmith | this ^ | |
| 16:50:59 | JayF | we'd need ironic to close that loop on more of an event-driven basis vs a poll to fix that concern before relying more heabily on cache | |
| 16:51:17 | JayF | admittedly my initial email was written assuming "ironic updates more stuff about ironic nodes" may be the path, I didn't expect sean-k-mooney to find low-hanging-fruit like he has | |
| 16:51:25 | sean-k-mooney | JayF: ya i saw your coment on the cach. i think we shoudl actully start lookign at riping out the hasring stuff | |
| 16:51:41 | sean-k-mooney | but also have a wider converation on this in general | |
| 16:52:02 | JayF | even just if we put in knobs to let operators make the tradeoff, it might help, idk | |
| 16:52:14 | JayF | I'll ensure I review those before the cross-project session | |
| 16:52:26 | JayF | if a sync between just us two to refine before PTG is valuable, we can do that too | |