Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2025-09-12
10:02:52 sean-k-mooney we could do it with perf but that might be overkill
10:03:07 gibi I guess somewhere in the /proc it is recorded
10:03:19 gibi as top knows it
10:03:29 sean-k-mooney top woudl also prtint it. yes its aviabel in proc, we can turn on atop in ci
10:03:43 sean-k-mooney i t was added not too long ago but i have not seen job using it
10:04:11 sean-k-mooney im just not sure if it will show cumlitive cpu tiem
10:04:36 gibi I will check I see podman login --username gibi --password
10:04:38 gibi ahh not that
10:04:46 gibi I see https://github.com/openstack/devstack/commit/1aac81ee881534276fd7d6540ed356a85d064a13
10:06:12 sean-k-mooney yep so i was looking at that patch a few times when looking at odd gate failerus and debated if it woudl help but i have not tried it yet
10:06:18 gibi it should be enabled by defautl but I don't see the logs in the run so I need to dig
10:06:48 sean-k-mooney i dont think its on by defaualt
10:07:11 sean-k-mooney https://github.com/openstack/devstack/commit/1aac81ee881534276fd7d6540ed356a85d064a13#diff-3fedb5d16b14e2fe731944ed0819eabefe91f42d456377e5ce6f0a1874232269R1097
10:07:16 sean-k-mooney shoudl card against it
10:07:31 sean-k-mooney we startar the base devstack josb with disable all services
10:07:42 sean-k-mooney but even without that i dont see this patch addign it to the default service list
10:08:13 sean-k-mooney gibi: but it shoudl jsut require added `enable_service atop`
10:08:57 sean-k-mooney or in ci i guess thats `atop: true`
10:10:17 sean-k-mooney gibi: oh i see you addign post-run: playbooks/nova-rally-fake-virt-threading/post.yaml and i guess that also runs rally a second time correct
10:11:05 gibi yes, it calls rally task restart that trigger rally to re-run the previous task
10:11:41 gibi https://review.opendev.org/c/openstack/nova/+/960130/16/roles/rerun-rally/tasks/main.yaml#11
10:12:02 gibi it would be nice to re-use the rally roles but it is not stored in the rally-openstack repo in a re-usable way
10:12:56 gibi also they hardcode a bunch of things that does not help reuse like the report generation does not handle multiple rally runs
10:13:28 gibi food even :)
10:14:36 sean-k-mooney enjoy. while we should do more testing this is all very promising congrats
10:54:50 opendevreview Merged openstack/nova master: Update master for stable/2025.2 https://review.opendev.org/c/openstack/nova/+/960745
10:55:03 opendevreview Merged openstack/nova master: reno: Update master for unmaintained/2023.1 https://review.opendev.org/c/openstack/nova/+/935117
10:55:10 opendevreview Merged openstack/placement master: Update master for stable/2025.2 https://review.opendev.org/c/openstack/placement/+/960509
10:55:31 opendevreview Merged openstack/nova stable/2025.2: Update .gitreview for stable/2025.2 https://review.opendev.org/c/openstack/nova/+/960743
10:55:42 opendevreview Merged openstack/nova stable/2025.2: Update TOX_CONSTRAINTS_FILE for stable/2025.2 https://review.opendev.org/c/openstack/nova/+/960744
12:22:05 opendevreview Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130
12:40:46 opendevreview Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130
13:30:49 opendevreview Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130
13:55:24 dansmith gibi: just trying to sanity check some of the stats there..
13:56:19 dansmith it seems like for the majority of the run everything is idle all the time, with the exception of a short thing at the front where we have idle_workers < max_workers, and I wonder if that's just because we're starting up
13:56:45 opendevreview Julien LE JEUNE proposed openstack/nova master: nova-condutor puts instance in error state https://review.opendev.org/c/openstack/nova/+/901655
13:57:02 dansmith I went looking for a comparison of how busy the workers were between the two.. like if the threaded mode was able to keep up with 5 workers...but only just, or if they both were taxed, but keeping up
13:57:36 dansmith just wondering if maybe that stats log line is not giving us accurate results (I assume the wall time is, and that looks good of course)
13:58:09 gibi dansmith: I think given the limited number of 5 computes the scheduler can keep up with the requests as the scheduling them is fairly easy
13:59:49 gibi I'm trying to crank up both the number of computes and the number of parallel VM boot requests to see if we can make the scheduler more busy
14:03:02 dansmith okay yeah, my point is just that while the numbers seem great, feels like maybe the load is too low to really measure a difference
14:04:27 sean-k-mooney gibi: you could create a custom zull nodeset in the job with up to 5 nodes
14:05:03 sean-k-mooney that the node limit in our tenant. i also suppoect at least on the subnodes you coudl push to 10-20 fake agents
14:05:29 sean-k-mooney so you might be able to similate somethign in the 85 compute range
14:06:19 gibi dansmith: I agree with your observation and I will try to crank up the numbers
14:06:46 gibi sean-k-mooney: yeah I just doubled the compute agents to 10
14:07:16 sean-k-mooney im just looking at the mem low point to see how close you were and the swap usage
14:07:22 gibi sean-k-mooney: also I found that the memory_tracker collects cpu time data as well per process so I can pull RSS and time there
14:07:40 sean-k-mooney to see if i can find anythign to indeicatre how many more we might eb able to run
14:07:50 sean-k-mooney we can also requst vms with 16GB of ram now
14:08:01 sean-k-mooney via a custom node set
14:08:20 sean-k-mooney gibi: oh i looked brifely but must have missed that before
14:08:22 gibi soo many options:)
14:09:00 sean-k-mooney oh is time process time?
14:09:24 gibi as far as I see yes
14:09:25 sean-k-mooney as in wall clock time of the acticve (user) time of the process
14:09:54 sean-k-mooney memory_tracker low_point: 1900960
14:09:59 gibi it is cputime from ps
14:10:10 gibi https://github.com/openstack/devstack/blob/f6d8dab0e885b8de8c0f44388d538da7d4f9b7ec/tools/memory_tracker.sh#L88C66-L88C73
14:10:20 sean-k-mooney so the low point was almost 2G fo aviabel ram
14:11:10 opendevreview Julien LE JEUNE proposed openstack/nova master: nova-condutor puts instance in error state https://review.opendev.org/c/openstack/nova/+/901655
14:11:33 sean-k-mooney teh resident memory of nova-comptue seams to be around 165mb so if we round up to say 200MB
14:12:15 sean-k-mooney we can proably run 20-25 fake agent on the contoler and i would guess 50+ on the 8G subnodes based on ram
14:13:15 sean-k-mooney my question woudl then be what those the cpu load look like but you proably can simulate a 200 node cluster if needed
14:14:02 sean-k-mooney if you were to try that i woudl proably do 4 dedicatec comptue and disable the nova-comptues on the contoler entirly
14:15:43 sean-k-mooney gibi interesting i wonder if we shoudl make the colelcted parmaters configurable via an devstack environment var
14:16:25 gibi soo many options:)
14:16:38 sean-k-mooney ill stop suggesting them
14:18:29 sean-k-mooney your current patch has times 400 and concurrency 40 + 10 fake computes
14:18:37 sean-k-mooney and you turned on atop
14:18:46 sean-k-mooney so lets see what that shows
14:19:31 sean-k-mooney ... post failure
14:19:39 sean-k-mooney rally: error: unrecognized arguments: -n 2 | xargs rally task trends --out /opt/stack/.rally/results/trends.html --tasks
14:20:31 opendevreview Sylvain Bauza proposed openstack/nova master: Support multiple allocations for vGPUs https://review.opendev.org/c/openstack/nova/+/845757
14:22:31 gibi ahh probably ansible command vs shell. I use pipes
14:27:08 sean-k-mooney ah right proably
14:27:28 sean-k-mooney ya your using command
14:27:33 sean-k-mooney so you need shell for that to work
14:29:24 opendevreview Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130
15:06:05 opendevreview Florian proposed openstack/nova master: Add check for PCIe devices attach limit for volume and ports https://review.opendev.org/c/openstack/nova/+/955584
15:11:04 gibi sean-k-mooney: dansmith: 20 fake fits in one node https://zuul.opendev.org/t/openstack/build/d865972fcf374ebb813496cec36be473/logs memory low point is around 700MB
15:21:24 sean-k-mooney gibi: that is for a contoller right dedicated comptue could run more. mind if i create a seperete patch just to play with that aspect and a custom nodeset
15:22:25 sean-k-mooney i dont want to modify your but i ws thinkign of puting one on top and tweakign the nodeset def and subnodes to see if we coudl get to the 100-200 fake node range
16:07:09 sean-k-mooney @gibi with the increase comptues and iteration and concurrance
16:07:21 sean-k-mooney the tests look like tehy took about the same amoutn of tiem
16:07:46 sean-k-mooney well no
16:08:02 sean-k-mooney it went form around 207 to 270 seconds
16:11:21 sean-k-mooney the median time is shockingly consitent across all 4 runs
16:13:24 sean-k-mooney the max time doubles as we doubled the number of instances and the number of hosts.
16:13:51 sean-k-mooney that i do find a littel interstign as i was expectign that to cancle out espcially since you doubled the concurrency as well
16:14:59 sean-k-mooney gibi: you could consider upping the max confurrent build per host form the default 10 to see it that reduces that but its seams to be linear
16:15:32 sean-k-mooney so if you cahnge that form 10 to say 100 i wonder if that woudl affect anything
16:26:06 zigo Got this building Nova Flamingo: https://paste.opendev.org/show/bpVRJbZbVFoY5mieellm/
16:26:10 zigo Any idea what's going on?
16:27:28 zigo Oh, should be my patch... :P
16:28:27 sean-k-mooney we have been removing the disperate imports of eventlet from the difent moduels and centralising them
16:28:50 zigo It's my fault, no worries.
16:29:50 zigo I had a patch adding greenthread.sleep(0), from past fix in Epoxy, removed it, should be ok this time.
16:33:46 sean-k-mooney gibi: https://gist.github.com/SeanMooney/27bb304653173a393c72d4cf4eca98e2 i got gemini to so some basic analasy fo the raw memory tracker data
16:37:32 sean-k-mooney i actuly need to check if i gave it the right data

Earlier   Later