Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2025-09-12
13:30:49 opendevreview Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130
13:55:24 dansmith gibi: just trying to sanity check some of the stats there..
13:56:19 dansmith it seems like for the majority of the run everything is idle all the time, with the exception of a short thing at the front where we have idle_workers < max_workers, and I wonder if that's just because we're starting up
13:56:45 opendevreview Julien LE JEUNE proposed openstack/nova master: nova-condutor puts instance in error state https://review.opendev.org/c/openstack/nova/+/901655
13:57:02 dansmith I went looking for a comparison of how busy the workers were between the two.. like if the threaded mode was able to keep up with 5 workers...but only just, or if they both were taxed, but keeping up
13:57:36 dansmith just wondering if maybe that stats log line is not giving us accurate results (I assume the wall time is, and that looks good of course)
13:58:09 gibi dansmith: I think given the limited number of 5 computes the scheduler can keep up with the requests as the scheduling them is fairly easy
13:59:49 gibi I'm trying to crank up both the number of computes and the number of parallel VM boot requests to see if we can make the scheduler more busy
14:03:02 dansmith okay yeah, my point is just that while the numbers seem great, feels like maybe the load is too low to really measure a difference
14:04:27 sean-k-mooney gibi: you could create a custom zull nodeset in the job with up to 5 nodes
14:05:03 sean-k-mooney that the node limit in our tenant. i also suppoect at least on the subnodes you coudl push to 10-20 fake agents
14:05:29 sean-k-mooney so you might be able to similate somethign in the 85 compute range
14:06:19 gibi dansmith: I agree with your observation and I will try to crank up the numbers
14:06:46 gibi sean-k-mooney: yeah I just doubled the compute agents to 10
14:07:16 sean-k-mooney im just looking at the mem low point to see how close you were and the swap usage
14:07:22 gibi sean-k-mooney: also I found that the memory_tracker collects cpu time data as well per process so I can pull RSS and time there
14:07:40 sean-k-mooney to see if i can find anythign to indeicatre how many more we might eb able to run
14:07:50 sean-k-mooney we can also requst vms with 16GB of ram now
14:08:01 sean-k-mooney via a custom node set
14:08:20 sean-k-mooney gibi: oh i looked brifely but must have missed that before
14:08:22 gibi soo many options:)
14:09:00 sean-k-mooney oh is time process time?
14:09:24 gibi as far as I see yes
14:09:25 sean-k-mooney as in wall clock time of the acticve (user) time of the process
14:09:54 sean-k-mooney memory_tracker low_point: 1900960
14:09:59 gibi it is cputime from ps
14:10:10 gibi https://github.com/openstack/devstack/blob/f6d8dab0e885b8de8c0f44388d538da7d4f9b7ec/tools/memory_tracker.sh#L88C66-L88C73
14:10:20 sean-k-mooney so the low point was almost 2G fo aviabel ram
14:11:10 opendevreview Julien LE JEUNE proposed openstack/nova master: nova-condutor puts instance in error state https://review.opendev.org/c/openstack/nova/+/901655
14:11:33 sean-k-mooney teh resident memory of nova-comptue seams to be around 165mb so if we round up to say 200MB
14:12:15 sean-k-mooney we can proably run 20-25 fake agent on the contoler and i would guess 50+ on the 8G subnodes based on ram
14:13:15 sean-k-mooney my question woudl then be what those the cpu load look like but you proably can simulate a 200 node cluster if needed
14:14:02 sean-k-mooney if you were to try that i woudl proably do 4 dedicatec comptue and disable the nova-comptues on the contoler entirly
14:15:43 sean-k-mooney gibi interesting i wonder if we shoudl make the colelcted parmaters configurable via an devstack environment var
14:16:25 gibi soo many options:)
14:16:38 sean-k-mooney ill stop suggesting them
14:18:29 sean-k-mooney your current patch has times 400 and concurrency 40 + 10 fake computes
14:18:37 sean-k-mooney and you turned on atop
14:18:46 sean-k-mooney so lets see what that shows
14:19:31 sean-k-mooney ... post failure
14:19:39 sean-k-mooney rally: error: unrecognized arguments: -n 2 | xargs rally task trends --out /opt/stack/.rally/results/trends.html --tasks
14:20:31 opendevreview Sylvain Bauza proposed openstack/nova master: Support multiple allocations for vGPUs https://review.opendev.org/c/openstack/nova/+/845757
14:22:31 gibi ahh probably ansible command vs shell. I use pipes
14:27:08 sean-k-mooney ah right proably
14:27:28 sean-k-mooney ya your using command
14:27:33 sean-k-mooney so you need shell for that to work
14:29:24 opendevreview Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130
15:06:05 opendevreview Florian proposed openstack/nova master: Add check for PCIe devices attach limit for volume and ports https://review.opendev.org/c/openstack/nova/+/955584
15:11:04 gibi sean-k-mooney: dansmith: 20 fake fits in one node https://zuul.opendev.org/t/openstack/build/d865972fcf374ebb813496cec36be473/logs memory low point is around 700MB
15:21:24 sean-k-mooney gibi: that is for a contoller right dedicated comptue could run more. mind if i create a seperete patch just to play with that aspect and a custom nodeset
15:22:25 sean-k-mooney i dont want to modify your but i ws thinkign of puting one on top and tweakign the nodeset def and subnodes to see if we coudl get to the 100-200 fake node range
16:07:09 sean-k-mooney @gibi with the increase comptues and iteration and concurrance
16:07:21 sean-k-mooney the tests look like tehy took about the same amoutn of tiem
16:07:46 sean-k-mooney well no
16:08:02 sean-k-mooney it went form around 207 to 270 seconds
16:11:21 sean-k-mooney the median time is shockingly consitent across all 4 runs
16:13:24 sean-k-mooney the max time doubles as we doubled the number of instances and the number of hosts.
16:13:51 sean-k-mooney that i do find a littel interstign as i was expectign that to cancle out espcially since you doubled the concurrency as well
16:14:59 sean-k-mooney gibi: you could consider upping the max confurrent build per host form the default 10 to see it that reduces that but its seams to be linear
16:15:32 sean-k-mooney so if you cahnge that form 10 to say 100 i wonder if that woudl affect anything
16:26:06 zigo Got this building Nova Flamingo: https://paste.opendev.org/show/bpVRJbZbVFoY5mieellm/
16:26:10 zigo Any idea what's going on?
16:27:28 zigo Oh, should be my patch... :P
16:28:27 sean-k-mooney we have been removing the disperate imports of eventlet from the difent moduels and centralising them
16:28:50 zigo It's my fault, no worries.
16:29:50 zigo I had a patch adding greenthread.sleep(0), from past fix in Epoxy, removed it, should be ok this time.
16:33:46 sean-k-mooney gibi: https://gist.github.com/SeanMooney/27bb304653173a393c72d4cf4eca98e2 i got gemini to so some basic analasy fo the raw memory tracker data
16:37:32 sean-k-mooney i actuly need to check if i gave it the right data
16:39:00 opendevreview Tobias Urdin proposed openstack/nova master: Prevent leaking host info when HostMappingNotFound https://review.opendev.org/c/openstack/nova/+/959296
16:39:07 opendevreview Tobias Urdin proposed openstack/nova master: Prevent leaking host info when HostMappingNotFound https://review.opendev.org/c/openstack/nova/+/959296
16:39:14 sean-k-mooney the memory tracker data seasm to end at Sep 12 14:44:52
16:39:26 sean-k-mooney were as the second rally run starts at 2025-09-12T14:48:54
16:39:49 sean-k-mooney so i dont think we actully have good data for that
18:17:34 opendevreview Merged openstack/nova stable/2025.1: Fix 'nova-manage image_property set' command https://review.opendev.org/c/openstack/nova/+/958834
21:14:57 sean-k-mooney gibi: i gave up on gmini and got claude to write a script to process the atop data by converting it to json an then procesing that
21:15:06 sean-k-mooney this is the inial output https://paste.opendev.org/show/brIoumAiGGb9GHx2eXzv/
21:15:58 sean-k-mooney im still not sure tha tis correct either but ill push it somewhere on monday after i have checked if it sane
#openstack-nova - 2025-09-13
20:55:21 opendevreview Thomas Goirand proposed openstack/nova-specs master: Restrict booting from volume with disk https://review.opendev.org/c/openstack/nova-specs/+/960513
#openstack-nova - 2025-09-14
10:41:31 ivveh hello folks, i have read https://docs.openstack.org/nova/latest/admin/manage-shares.html and it says that: "Due to current virtiofs implementation in Qemu / libvirt, the following Nova features are blocked for VMs with shares attached" and I have also seen post where this limitation blaim is put on KVM/QEMU (that it can't attach a filesystem to a running VM. Assuming the vm has memfd&shared mem enabled, this is not true and yo
10:42:05 ivveh just looking for clarification, if everyone is misinterpreting this statement or what is going on?
10:43:25 ivveh because there is nothing hindering from KVM/QEMU side to execute "virsh attach-device <instance> some.xml --live --config" which contains xml config for a filesystem with a socket coming from virtiofsd
10:43:48 ivveh (while the instance is running).. you can also detach-device as well
10:44:45 ivveh and since this was implemented not so long ago and this has worked for quite some time in qemu, im just wondering.. is it pointing perhaps to a very old version of QEMU/KVM?
#openstack-nova - 2025-09-15
07:37:35 opendevreview Julien LE JEUNE proposed openstack/nova master: Adds regression test for bug LP#2044235 https://review.opendev.org/c/openstack/nova/+/960349
08:21:59 opendevreview Julien LE JEUNE proposed openstack/nova master: Adds regression test for bug LP#2044235 https://review.opendev.org/c/openstack/nova/+/960349
09:01:27 opendevreview Julien LE JEUNE proposed openstack/nova master: nova-conductor puts instance in error state https://review.opendev.org/c/openstack/nova/+/901655
09:47:46 opendevreview Sylvain Bauza proposed openstack/nova master: Support multiple allocations for vGPUs https://review.opendev.org/c/openstack/nova/+/845757
10:40:16 sean-k-mooney ivveh so the current state is as follows, when we were adding the feature to nova orginally the qemu feature to supprot attach was not complete so it was put out of scope of the nova feature wit the intent to support atach in the future. the nova support for this took longer then intended and since then libvirt and qemu have released supprot for it but nova has not been
10:40:18 sean-k-mooney updated to supprot it
10:41:26 sean-k-mooney ivveh so the tl;dr is it used ot be a qemu limiation and not its a nova one as we have not updated to use the new qemu fatures. for partcial reasons however the limitation si still in effefct for openstack but its something we would like to eventually adress.
10:49:35 opendevreview Rajesh Tailor proposed openstack/nova master: Fix string format specifier https://review.opendev.org/c/openstack/nova/+/961211
11:31:24 ratailor sean-k-mooney, gibi bauzas could you please review clean backports, one more +2 is required. https://review.opendev.org/q/project:openstack/nova+branch:stable/2024.2+owner:ratailor@redhat.com
11:48:07 sean-k-mooney gibi's on pto until wednesday i belvie but i can take a look at the latter two now. it looks like elod and gibi have already approved the first 2
11:48:24 sean-k-mooney oh elod already didi it
11:49:57 sean-k-mooney since elod has appvoed the 2024.2 version ill quickly pre review the 2024.1 backports
11:51:06 jlejeune hello sean-k-mooney, did you have time to check my regression test: https://review.opendev.org/c/openstack/nova/+/960349 ?
11:52:53 sean-k-mooney i saw that you change to a plain Excption you could also have used a TestEXcption but my main concern with that is you likely can just catch every excption in the main code.
11:53:55 sean-k-mooney this might be fine https://review.opendev.org/c/openstack/nova/+/901655/8/nova/compute/manager.py#9095 because its in pre live migraiton
11:54:29 sean-k-mooney but the futher we get into the migraiton tbe more clean up that is required and the more careful and specific we need to be with what we catch
11:55:33 sean-k-mooney you are at least converting the generic error to a specific on in that expcption handeler which is good

Earlier   Later