| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-09-11 | |||
| 19:37:24 | opendevreview | Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130 | |
| 20:02:00 | opendevreview | Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130 | |
| #openstack-nova - 2025-09-12 | |||
| 06:59:04 | opendevreview | Lajos Katona proposed openstack/nova master: Use SDK for Neutron networks https://review.opendev.org/c/openstack/nova/+/928022 | |
| 07:05:55 | jlejeune | sean-k-mooney: hello, yeah I know that catch 'Exception' is not necessarily a good idea, but in that case if I only put the try/except statement around the 'check_can_live_migrate_source' call, it will catch every exceptions which may happen during that specific rpc call | |
| 07:09:19 | ratailor | elodilles, could you please review when you have time https://review.opendev.org/c/openstack/nova/+/958834 and https://review.opendev.org/q/owner:ratailor@redhat.com+branch:stable/2024.2 | |
| 08:17:48 | opendevreview | Julien LE JEUNE proposed openstack/nova master: nova-condutor puts instance in error state https://review.opendev.org/c/openstack/nova/+/901655 | |
| 08:37:32 | elodilles | ratailor: ACK, added them to my TODO list | |
| 08:41:29 | opendevreview | Merged openstack/placement stable/2025.2: Update .gitreview for stable/2025.2 https://review.opendev.org/c/openstack/placement/+/960507 | |
| 08:41:31 | opendevreview | Merged openstack/placement stable/2025.2: Update TOX_CONSTRAINTS_FILE for stable/2025.2 https://review.opendev.org/c/openstack/placement/+/960508 | |
| 08:54:02 | opendevreview | OpenStack Release Bot proposed openstack/nova stable/2025.2: Update .gitreview for stable/2025.2 https://review.opendev.org/c/openstack/nova/+/960743 | |
| 08:54:07 | opendevreview | OpenStack Release Bot proposed openstack/nova stable/2025.2: Update TOX_CONSTRAINTS_FILE for stable/2025.2 https://review.opendev.org/c/openstack/nova/+/960744 | |
| 08:54:12 | opendevreview | OpenStack Release Bot proposed openstack/nova master: Update master for stable/2025.2 https://review.opendev.org/c/openstack/nova/+/960745 | |
| 09:01:30 | gibi | sean-k-mooney: dansmith: I have the first comparable rally results between eventlet and native threading. See my top level comment in https://review.opendev.org/c/openstack/nova/+/960130/16#message-e6a461d92172aea570e8f18405d47ee00b9c300a It links to both rally results with timing and I manually pulled out the pool usage and the memory usage of the scheduler (I have the intention to script some of it | |
| 09:01:36 | gibi | in CI) | |
| 09:49:23 | ratailor | elodilles, Thanks! | |
| 09:52:52 | opendevreview | Taketani Ryo proposed openstack/nova-specs master: Add a spec for 2026.1 for libvirt launching Arm CCA instances https://review.opendev.org/c/openstack/nova-specs/+/960777 | |
| 09:54:36 | sean-k-mooney | gibi do you happen ot know if they ran on the same provider | |
| 09:54:59 | sean-k-mooney | gibi: i.e. are the host vms comparible | |
| 09:58:06 | gibi | the two rally execution happen within the same CI job on the same CI worker. So it is always comparable :) | |
| 09:59:02 | gibi | the base job runs rally then in the post-run I reconfigure to native threading and run rally again | |
| 09:59:58 | sean-k-mooney | yep just reading the comment on the patch | |
| 10:00:17 | sean-k-mooney | so that is good because it elimiates the biggest variabel in our ci | |
| 10:00:26 | sean-k-mooney | run to run hardware deltas | |
| 10:00:37 | sean-k-mooney | looking atht eh results this is very promising | |
| 10:01:25 | sean-k-mooney | gibi: the only metric i woul have liket to see that i have not seen yet is the total cpu time of the nova process | |
| 10:01:48 | sean-k-mooney | i.e. is there any delter on cpu load with eventlet vs threading | |
| 10:02:18 | gibi | sean-k-mooney: how can we collect that? | |
| 10:02:25 | sean-k-mooney | in theory there shoudl not be since the eventlet shcdluer vs kernel shclduer selecting the next thread shoudl eb more or less a wash | |
| 10:02:52 | sean-k-mooney | we could do it with perf but that might be overkill | |
| 10:03:07 | gibi | I guess somewhere in the /proc it is recorded | |
| 10:03:19 | gibi | as top knows it | |
| 10:03:29 | sean-k-mooney | top woudl also prtint it. yes its aviabel in proc, we can turn on atop in ci | |
| 10:03:43 | sean-k-mooney | i t was added not too long ago but i have not seen job using it | |
| 10:04:11 | sean-k-mooney | im just not sure if it will show cumlitive cpu tiem | |
| 10:04:36 | gibi | I will check I see podman login --username gibi --password | |
| 10:04:38 | gibi | ahh not that | |
| 10:04:46 | gibi | I see https://github.com/openstack/devstack/commit/1aac81ee881534276fd7d6540ed356a85d064a13 | |
| 10:06:12 | sean-k-mooney | yep so i was looking at that patch a few times when looking at odd gate failerus and debated if it woudl help but i have not tried it yet | |
| 10:06:18 | gibi | it should be enabled by defautl but I don't see the logs in the run so I need to dig | |
| 10:06:48 | sean-k-mooney | i dont think its on by defaualt | |
| 10:07:11 | sean-k-mooney | https://github.com/openstack/devstack/commit/1aac81ee881534276fd7d6540ed356a85d064a13#diff-3fedb5d16b14e2fe731944ed0819eabefe91f42d456377e5ce6f0a1874232269R1097 | |
| 10:07:16 | sean-k-mooney | shoudl card against it | |
| 10:07:31 | sean-k-mooney | we startar the base devstack josb with disable all services | |
| 10:07:42 | sean-k-mooney | but even without that i dont see this patch addign it to the default service list | |
| 10:08:13 | sean-k-mooney | gibi: but it shoudl jsut require added `enable_service atop` | |
| 10:08:57 | sean-k-mooney | or in ci i guess thats `atop: true` | |
| 10:10:17 | sean-k-mooney | gibi: oh i see you addign post-run: playbooks/nova-rally-fake-virt-threading/post.yaml and i guess that also runs rally a second time correct | |
| 10:11:05 | gibi | yes, it calls rally task restart that trigger rally to re-run the previous task | |
| 10:11:41 | gibi | https://review.opendev.org/c/openstack/nova/+/960130/16/roles/rerun-rally/tasks/main.yaml#11 | |
| 10:12:02 | gibi | it would be nice to re-use the rally roles but it is not stored in the rally-openstack repo in a re-usable way | |
| 10:12:56 | gibi | also they hardcode a bunch of things that does not help reuse like the report generation does not handle multiple rally runs | |
| 10:13:28 | gibi | food even :) | |
| 10:14:36 | sean-k-mooney | enjoy. while we should do more testing this is all very promising congrats | |
| 10:54:50 | opendevreview | Merged openstack/nova master: Update master for stable/2025.2 https://review.opendev.org/c/openstack/nova/+/960745 | |
| 10:55:03 | opendevreview | Merged openstack/nova master: reno: Update master for unmaintained/2023.1 https://review.opendev.org/c/openstack/nova/+/935117 | |
| 10:55:10 | opendevreview | Merged openstack/placement master: Update master for stable/2025.2 https://review.opendev.org/c/openstack/placement/+/960509 | |
| 10:55:31 | opendevreview | Merged openstack/nova stable/2025.2: Update .gitreview for stable/2025.2 https://review.opendev.org/c/openstack/nova/+/960743 | |
| 10:55:42 | opendevreview | Merged openstack/nova stable/2025.2: Update TOX_CONSTRAINTS_FILE for stable/2025.2 https://review.opendev.org/c/openstack/nova/+/960744 | |
| 12:22:05 | opendevreview | Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130 | |
| 12:40:46 | opendevreview | Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130 | |
| 13:30:49 | opendevreview | Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130 | |
| 13:55:24 | dansmith | gibi: just trying to sanity check some of the stats there.. | |
| 13:56:19 | dansmith | it seems like for the majority of the run everything is idle all the time, with the exception of a short thing at the front where we have idle_workers < max_workers, and I wonder if that's just because we're starting up | |
| 13:56:45 | opendevreview | Julien LE JEUNE proposed openstack/nova master: nova-condutor puts instance in error state https://review.opendev.org/c/openstack/nova/+/901655 | |
| 13:57:02 | dansmith | I went looking for a comparison of how busy the workers were between the two.. like if the threaded mode was able to keep up with 5 workers...but only just, or if they both were taxed, but keeping up | |
| 13:57:36 | dansmith | just wondering if maybe that stats log line is not giving us accurate results (I assume the wall time is, and that looks good of course) | |
| 13:58:09 | gibi | dansmith: I think given the limited number of 5 computes the scheduler can keep up with the requests as the scheduling them is fairly easy | |
| 13:59:49 | gibi | I'm trying to crank up both the number of computes and the number of parallel VM boot requests to see if we can make the scheduler more busy | |
| 14:03:02 | dansmith | okay yeah, my point is just that while the numbers seem great, feels like maybe the load is too low to really measure a difference | |
| 14:04:27 | sean-k-mooney | gibi: you could create a custom zull nodeset in the job with up to 5 nodes | |
| 14:05:03 | sean-k-mooney | that the node limit in our tenant. i also suppoect at least on the subnodes you coudl push to 10-20 fake agents | |
| 14:05:29 | sean-k-mooney | so you might be able to similate somethign in the 85 compute range | |
| 14:06:19 | gibi | dansmith: I agree with your observation and I will try to crank up the numbers | |
| 14:06:46 | gibi | sean-k-mooney: yeah I just doubled the compute agents to 10 | |
| 14:07:16 | sean-k-mooney | im just looking at the mem low point to see how close you were and the swap usage | |
| 14:07:22 | gibi | sean-k-mooney: also I found that the memory_tracker collects cpu time data as well per process so I can pull RSS and time there | |
| 14:07:40 | sean-k-mooney | to see if i can find anythign to indeicatre how many more we might eb able to run | |
| 14:07:50 | sean-k-mooney | we can also requst vms with 16GB of ram now | |
| 14:08:01 | sean-k-mooney | via a custom node set | |
| 14:08:20 | sean-k-mooney | gibi: oh i looked brifely but must have missed that before | |
| 14:08:22 | gibi | soo many options:) | |
| 14:09:00 | sean-k-mooney | oh is time process time? | |
| 14:09:24 | gibi | as far as I see yes | |
| 14:09:25 | sean-k-mooney | as in wall clock time of the acticve (user) time of the process | |
| 14:09:54 | sean-k-mooney | memory_tracker low_point: 1900960 | |
| 14:09:59 | gibi | it is cputime from ps | |
| 14:10:10 | gibi | https://github.com/openstack/devstack/blob/f6d8dab0e885b8de8c0f44388d538da7d4f9b7ec/tools/memory_tracker.sh#L88C66-L88C73 | |
| 14:10:20 | sean-k-mooney | so the low point was almost 2G fo aviabel ram | |
| 14:11:10 | opendevreview | Julien LE JEUNE proposed openstack/nova master: nova-condutor puts instance in error state https://review.opendev.org/c/openstack/nova/+/901655 | |
| 14:11:33 | sean-k-mooney | teh resident memory of nova-comptue seams to be around 165mb so if we round up to say 200MB | |
| 14:12:15 | sean-k-mooney | we can proably run 20-25 fake agent on the contoler and i would guess 50+ on the 8G subnodes based on ram | |
| 14:13:15 | sean-k-mooney | my question woudl then be what those the cpu load look like but you proably can simulate a 200 node cluster if needed | |
| 14:14:02 | sean-k-mooney | if you were to try that i woudl proably do 4 dedicatec comptue and disable the nova-comptues on the contoler entirly | |
| 14:15:43 | sean-k-mooney | gibi interesting i wonder if we shoudl make the colelcted parmaters configurable via an devstack environment var | |
| 14:16:25 | gibi | soo many options:) | |
| 14:16:38 | sean-k-mooney | ill stop suggesting them | |
| 14:18:29 | sean-k-mooney | your current patch has times 400 and concurrency 40 + 10 fake computes | |
| 14:18:37 | sean-k-mooney | and you turned on atop | |
| 14:18:46 | sean-k-mooney | so lets see what that shows | |
| 14:19:31 | sean-k-mooney | ... post failure | |