| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-09-11 | |||
| 14:35:26 | jlejeune | ^^ | |
| 14:35:46 | sean-k-mooney | so to your question atbout messagingTimeout it might be because of how we mock our messaage bus using the olso in memory oslo.messaging driver | |
| 14:35:58 | jlejeune | exactly | |
| 14:36:33 | sean-k-mooney | we sometimes need to disable case as call to get this to work as well | |
| 14:37:11 | jlejeune | I didn't find any example of that, do you have some in mind ? | |
| 14:37:54 | sean-k-mooney | no but i think we have or had example of causing live migration to fail in other tests | |
| 14:38:53 | sean-k-mooney | im not sure if your familar with the CAST_AS_CALL fixture as the name suggest it turn rpc casts which are fire and forget into a sync call that run in place and returns only when its complete | |
| 14:39:10 | sean-k-mooney | i dont think that is the problem here | |
| 14:39:31 | sean-k-mooney | but it can cause slightly diffent rpc behavior in teh regress test and sometime we have to turn it off | |
| 14:40:32 | jlejeune | yes I saw that flag and I tried to turn it off, but without any success | |
| 14:52:00 | sean-k-mooney | so we do have some regression tests for live migration roolback https://github.com/openstack/nova/blob/759e03c35d91dac4a04237d58493665c01952c9f/nova/tests/functional/regressions/test_bug_1848343.py#L105 | |
| 14:53:35 | opendevreview | Julien LE JEUNE proposed openstack/nova master: Adds regression test for bug LP#2044235 https://review.opendev.org/c/openstack/nova/+/960349 | |
| 14:55:44 | sean-k-mooney | im wonderign if there is a bette way to similate the failrue | |
| 14:58:03 | sean-k-mooney | ah you now just raising a generic excption | |
| 15:01:19 | jlejeune | yes because I can't raise the MessagingTimeout exception, if I do so, it's caught by the secheduler when it tries to find a destination host | |
| 15:02:54 | jlejeune | but after thinking about the issue, for my point of view, it's not very a big deal to catch all exceptions coming from that rpc call during the pre live migration check, what do you think about it ? | |
| 15:09:43 | opendevreview | Julien LE JEUNE proposed openstack/nova master: Adds regression test for bug LP#2044235 https://review.opendev.org/c/openstack/nova/+/960349 | |
| 16:17:48 | opendevreview | Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130 | |
| 16:32:31 | sean-k-mooney | jlejeune: am it depend we do have some logic that may care but lets look at it in the fix patch | |
| 16:33:01 | sean-k-mooney | jlejeune: we try to avoid `except Exception` in gernal | |
| 16:33:10 | sean-k-mooney | because it can hide effort | |
| 16:33:21 | sean-k-mooney | *errors | |
| 16:52:27 | opendevreview | Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130 | |
| 18:27:00 | opendevreview | Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130 | |
| 18:47:00 | opendevreview | Lajos Katona proposed openstack/nova master: Use SDK for Neutron networks https://review.opendev.org/c/openstack/nova/+/928022 | |
| 19:00:15 | opendevreview | Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130 | |
| 19:37:24 | opendevreview | Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130 | |
| 20:02:00 | opendevreview | Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130 | |
| #openstack-nova - 2025-09-12 | |||
| 06:59:04 | opendevreview | Lajos Katona proposed openstack/nova master: Use SDK for Neutron networks https://review.opendev.org/c/openstack/nova/+/928022 | |
| 07:05:55 | jlejeune | sean-k-mooney: hello, yeah I know that catch 'Exception' is not necessarily a good idea, but in that case if I only put the try/except statement around the 'check_can_live_migrate_source' call, it will catch every exceptions which may happen during that specific rpc call | |
| 07:09:19 | ratailor | elodilles, could you please review when you have time https://review.opendev.org/c/openstack/nova/+/958834 and https://review.opendev.org/q/owner:ratailor@redhat.com+branch:stable/2024.2 | |
| 08:17:48 | opendevreview | Julien LE JEUNE proposed openstack/nova master: nova-condutor puts instance in error state https://review.opendev.org/c/openstack/nova/+/901655 | |
| 08:37:32 | elodilles | ratailor: ACK, added them to my TODO list | |
| 08:41:29 | opendevreview | Merged openstack/placement stable/2025.2: Update .gitreview for stable/2025.2 https://review.opendev.org/c/openstack/placement/+/960507 | |
| 08:41:31 | opendevreview | Merged openstack/placement stable/2025.2: Update TOX_CONSTRAINTS_FILE for stable/2025.2 https://review.opendev.org/c/openstack/placement/+/960508 | |
| 08:54:02 | opendevreview | OpenStack Release Bot proposed openstack/nova stable/2025.2: Update .gitreview for stable/2025.2 https://review.opendev.org/c/openstack/nova/+/960743 | |
| 08:54:07 | opendevreview | OpenStack Release Bot proposed openstack/nova stable/2025.2: Update TOX_CONSTRAINTS_FILE for stable/2025.2 https://review.opendev.org/c/openstack/nova/+/960744 | |
| 08:54:12 | opendevreview | OpenStack Release Bot proposed openstack/nova master: Update master for stable/2025.2 https://review.opendev.org/c/openstack/nova/+/960745 | |
| 09:01:30 | gibi | sean-k-mooney: dansmith: I have the first comparable rally results between eventlet and native threading. See my top level comment in https://review.opendev.org/c/openstack/nova/+/960130/16#message-e6a461d92172aea570e8f18405d47ee00b9c300a It links to both rally results with timing and I manually pulled out the pool usage and the memory usage of the scheduler (I have the intention to script some of it | |
| 09:01:36 | gibi | in CI) | |
| 09:49:23 | ratailor | elodilles, Thanks! | |
| 09:52:52 | opendevreview | Taketani Ryo proposed openstack/nova-specs master: Add a spec for 2026.1 for libvirt launching Arm CCA instances https://review.opendev.org/c/openstack/nova-specs/+/960777 | |
| 09:54:36 | sean-k-mooney | gibi do you happen ot know if they ran on the same provider | |
| 09:54:59 | sean-k-mooney | gibi: i.e. are the host vms comparible | |
| 09:58:06 | gibi | the two rally execution happen within the same CI job on the same CI worker. So it is always comparable :) | |
| 09:59:02 | gibi | the base job runs rally then in the post-run I reconfigure to native threading and run rally again | |
| 09:59:58 | sean-k-mooney | yep just reading the comment on the patch | |
| 10:00:17 | sean-k-mooney | so that is good because it elimiates the biggest variabel in our ci | |
| 10:00:26 | sean-k-mooney | run to run hardware deltas | |
| 10:00:37 | sean-k-mooney | looking atht eh results this is very promising | |
| 10:01:25 | sean-k-mooney | gibi: the only metric i woul have liket to see that i have not seen yet is the total cpu time of the nova process | |
| 10:01:48 | sean-k-mooney | i.e. is there any delter on cpu load with eventlet vs threading | |
| 10:02:18 | gibi | sean-k-mooney: how can we collect that? | |
| 10:02:25 | sean-k-mooney | in theory there shoudl not be since the eventlet shcdluer vs kernel shclduer selecting the next thread shoudl eb more or less a wash | |
| 10:02:52 | sean-k-mooney | we could do it with perf but that might be overkill | |
| 10:03:07 | gibi | I guess somewhere in the /proc it is recorded | |
| 10:03:19 | gibi | as top knows it | |
| 10:03:29 | sean-k-mooney | top woudl also prtint it. yes its aviabel in proc, we can turn on atop in ci | |
| 10:03:43 | sean-k-mooney | i t was added not too long ago but i have not seen job using it | |
| 10:04:11 | sean-k-mooney | im just not sure if it will show cumlitive cpu tiem | |
| 10:04:36 | gibi | I will check I see podman login --username gibi --password | |
| 10:04:38 | gibi | ahh not that | |
| 10:04:46 | gibi | I see https://github.com/openstack/devstack/commit/1aac81ee881534276fd7d6540ed356a85d064a13 | |
| 10:06:12 | sean-k-mooney | yep so i was looking at that patch a few times when looking at odd gate failerus and debated if it woudl help but i have not tried it yet | |
| 10:06:18 | gibi | it should be enabled by defautl but I don't see the logs in the run so I need to dig | |
| 10:06:48 | sean-k-mooney | i dont think its on by defaualt | |
| 10:07:11 | sean-k-mooney | https://github.com/openstack/devstack/commit/1aac81ee881534276fd7d6540ed356a85d064a13#diff-3fedb5d16b14e2fe731944ed0819eabefe91f42d456377e5ce6f0a1874232269R1097 | |
| 10:07:16 | sean-k-mooney | shoudl card against it | |
| 10:07:31 | sean-k-mooney | we startar the base devstack josb with disable all services | |
| 10:07:42 | sean-k-mooney | but even without that i dont see this patch addign it to the default service list | |
| 10:08:13 | sean-k-mooney | gibi: but it shoudl jsut require added `enable_service atop` | |
| 10:08:57 | sean-k-mooney | or in ci i guess thats `atop: true` | |
| 10:10:17 | sean-k-mooney | gibi: oh i see you addign post-run: playbooks/nova-rally-fake-virt-threading/post.yaml and i guess that also runs rally a second time correct | |
| 10:11:05 | gibi | yes, it calls rally task restart that trigger rally to re-run the previous task | |
| 10:11:41 | gibi | https://review.opendev.org/c/openstack/nova/+/960130/16/roles/rerun-rally/tasks/main.yaml#11 | |
| 10:12:02 | gibi | it would be nice to re-use the rally roles but it is not stored in the rally-openstack repo in a re-usable way | |
| 10:12:56 | gibi | also they hardcode a bunch of things that does not help reuse like the report generation does not handle multiple rally runs | |
| 10:13:28 | gibi | food even :) | |
| 10:14:36 | sean-k-mooney | enjoy. while we should do more testing this is all very promising congrats | |
| 10:54:50 | opendevreview | Merged openstack/nova master: Update master for stable/2025.2 https://review.opendev.org/c/openstack/nova/+/960745 | |
| 10:55:03 | opendevreview | Merged openstack/nova master: reno: Update master for unmaintained/2023.1 https://review.opendev.org/c/openstack/nova/+/935117 | |
| 10:55:10 | opendevreview | Merged openstack/placement master: Update master for stable/2025.2 https://review.opendev.org/c/openstack/placement/+/960509 | |
| 10:55:31 | opendevreview | Merged openstack/nova stable/2025.2: Update .gitreview for stable/2025.2 https://review.opendev.org/c/openstack/nova/+/960743 | |
| 10:55:42 | opendevreview | Merged openstack/nova stable/2025.2: Update TOX_CONSTRAINTS_FILE for stable/2025.2 https://review.opendev.org/c/openstack/nova/+/960744 | |
| 12:22:05 | opendevreview | Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130 | |
| 12:40:46 | opendevreview | Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130 | |
| 13:30:49 | opendevreview | Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130 | |
| 13:55:24 | dansmith | gibi: just trying to sanity check some of the stats there.. | |
| 13:56:19 | dansmith | it seems like for the majority of the run everything is idle all the time, with the exception of a short thing at the front where we have idle_workers < max_workers, and I wonder if that's just because we're starting up | |
| 13:56:45 | opendevreview | Julien LE JEUNE proposed openstack/nova master: nova-condutor puts instance in error state https://review.opendev.org/c/openstack/nova/+/901655 | |
| 13:57:02 | dansmith | I went looking for a comparison of how busy the workers were between the two.. like if the threaded mode was able to keep up with 5 workers...but only just, or if they both were taxed, but keeping up | |
| 13:57:36 | dansmith | just wondering if maybe that stats log line is not giving us accurate results (I assume the wall time is, and that looks good of course) | |
| 13:58:09 | gibi | dansmith: I think given the limited number of 5 computes the scheduler can keep up with the requests as the scheduling them is fairly easy | |
| 13:59:49 | gibi | I'm trying to crank up both the number of computes and the number of parallel VM boot requests to see if we can make the scheduler more busy | |
| 14:03:02 | dansmith | okay yeah, my point is just that while the numbers seem great, feels like maybe the load is too low to really measure a difference | |
| 14:04:27 | sean-k-mooney | gibi: you could create a custom zull nodeset in the job with up to 5 nodes | |
| 14:05:03 | sean-k-mooney | that the node limit in our tenant. i also suppoect at least on the subnodes you coudl push to 10-20 fake agents | |
| 14:05:29 | sean-k-mooney | so you might be able to similate somethign in the 85 compute range | |
| 14:06:19 | gibi | dansmith: I agree with your observation and I will try to crank up the numbers | |
| 14:06:46 | gibi | sean-k-mooney: yeah I just doubled the compute agents to 10 | |