Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2025-09-11
14:31:36 sean-k-mooney jlejeune: sorry i am in a few converstaion at once. i see you pushed the tst so i can try and reveiw that patch and play with it loccally
14:33:00 jlejeune sean-k-mooney: no worry, I saw that you're in discussion with zigo
14:33:26 sean-k-mooney and on slack and oslo and watcher channel
14:33:42 jlejeune :)
14:34:09 jlejeune you're such a busy man :)
14:34:33 sean-k-mooney that or i have the attention span fo a magpie
14:35:26 jlejeune ^^
14:35:46 sean-k-mooney so to your question atbout messagingTimeout it might be because of how we mock our messaage bus using the olso in memory oslo.messaging driver
14:35:58 jlejeune exactly
14:36:33 sean-k-mooney we sometimes need to disable case as call to get this to work as well
14:37:11 jlejeune I didn't find any example of that, do you have some in mind ?
14:37:54 sean-k-mooney no but i think we have or had example of causing live migration to fail in other tests
14:38:53 sean-k-mooney im not sure if your familar with the CAST_AS_CALL fixture as the name suggest it turn rpc casts which are fire and forget into a sync call that run in place and returns only when its complete
14:39:10 sean-k-mooney i dont think that is the problem here
14:39:31 sean-k-mooney but it can cause slightly diffent rpc behavior in teh regress test and sometime we have to turn it off
14:40:32 jlejeune yes I saw that flag and I tried to turn it off, but without any success
14:52:00 sean-k-mooney so we do have some regression tests for live migration roolback https://github.com/openstack/nova/blob/759e03c35d91dac4a04237d58493665c01952c9f/nova/tests/functional/regressions/test_bug_1848343.py#L105
14:53:35 opendevreview Julien LE JEUNE proposed openstack/nova master: Adds regression test for bug LP#2044235 https://review.opendev.org/c/openstack/nova/+/960349
14:55:44 sean-k-mooney im wonderign if there is a bette way to similate the failrue
14:58:03 sean-k-mooney ah you now just raising a generic excption
15:01:19 jlejeune yes because I can't raise the MessagingTimeout exception, if I do so, it's caught by the secheduler when it tries to find a destination host
15:02:54 jlejeune but after thinking about the issue, for my point of view, it's not very a big deal to catch all exceptions coming from that rpc call during the pre live migration check, what do you think about it ?
15:09:43 opendevreview Julien LE JEUNE proposed openstack/nova master: Adds regression test for bug LP#2044235 https://review.opendev.org/c/openstack/nova/+/960349
16:17:48 opendevreview Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130
16:32:31 sean-k-mooney jlejeune: am it depend we do have some logic that may care but lets look at it in the fix patch
16:33:01 sean-k-mooney jlejeune: we try to avoid `except Exception` in gernal
16:33:10 sean-k-mooney because it can hide effort
16:33:21 sean-k-mooney *errors
16:52:27 opendevreview Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130
18:27:00 opendevreview Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130
18:47:00 opendevreview Lajos Katona proposed openstack/nova master: Use SDK for Neutron networks https://review.opendev.org/c/openstack/nova/+/928022
19:00:15 opendevreview Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130
19:37:24 opendevreview Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130
20:02:00 opendevreview Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130
#openstack-nova - 2025-09-12
06:59:04 opendevreview Lajos Katona proposed openstack/nova master: Use SDK for Neutron networks https://review.opendev.org/c/openstack/nova/+/928022
07:05:55 jlejeune sean-k-mooney: hello, yeah I know that catch 'Exception' is not necessarily a good idea, but in that case if I only put the try/except statement around the 'check_can_live_migrate_source' call, it will catch every exceptions which may happen during that specific rpc call
07:09:19 ratailor elodilles, could you please review when you have time https://review.opendev.org/c/openstack/nova/+/958834 and https://review.opendev.org/q/owner:ratailor@redhat.com+branch:stable/2024.2
08:17:48 opendevreview Julien LE JEUNE proposed openstack/nova master: nova-condutor puts instance in error state https://review.opendev.org/c/openstack/nova/+/901655
08:37:32 elodilles ratailor: ACK, added them to my TODO list
08:41:29 opendevreview Merged openstack/placement stable/2025.2: Update .gitreview for stable/2025.2 https://review.opendev.org/c/openstack/placement/+/960507
08:41:31 opendevreview Merged openstack/placement stable/2025.2: Update TOX_CONSTRAINTS_FILE for stable/2025.2 https://review.opendev.org/c/openstack/placement/+/960508
08:54:02 opendevreview OpenStack Release Bot proposed openstack/nova stable/2025.2: Update .gitreview for stable/2025.2 https://review.opendev.org/c/openstack/nova/+/960743
08:54:07 opendevreview OpenStack Release Bot proposed openstack/nova stable/2025.2: Update TOX_CONSTRAINTS_FILE for stable/2025.2 https://review.opendev.org/c/openstack/nova/+/960744
08:54:12 opendevreview OpenStack Release Bot proposed openstack/nova master: Update master for stable/2025.2 https://review.opendev.org/c/openstack/nova/+/960745
09:01:30 gibi sean-k-mooney: dansmith: I have the first comparable rally results between eventlet and native threading. See my top level comment in https://review.opendev.org/c/openstack/nova/+/960130/16#message-e6a461d92172aea570e8f18405d47ee00b9c300a It links to both rally results with timing and I manually pulled out the pool usage and the memory usage of the scheduler (I have the intention to script some of it
09:01:36 gibi in CI)
09:49:23 ratailor elodilles, Thanks!
09:52:52 opendevreview Taketani Ryo proposed openstack/nova-specs master: Add a spec for 2026.1 for libvirt launching Arm CCA instances https://review.opendev.org/c/openstack/nova-specs/+/960777
09:54:36 sean-k-mooney gibi do you happen ot know if they ran on the same provider
09:54:59 sean-k-mooney gibi: i.e. are the host vms comparible
09:58:06 gibi the two rally execution happen within the same CI job on the same CI worker. So it is always comparable :)
09:59:02 gibi the base job runs rally then in the post-run I reconfigure to native threading and run rally again
09:59:58 sean-k-mooney yep just reading the comment on the patch
10:00:17 sean-k-mooney so that is good because it elimiates the biggest variabel in our ci
10:00:26 sean-k-mooney run to run hardware deltas
10:00:37 sean-k-mooney looking atht eh results this is very promising
10:01:25 sean-k-mooney gibi: the only metric i woul have liket to see that i have not seen yet is the total cpu time of the nova process
10:01:48 sean-k-mooney i.e. is there any delter on cpu load with eventlet vs threading
10:02:18 gibi sean-k-mooney: how can we collect that?
10:02:25 sean-k-mooney in theory there shoudl not be since the eventlet shcdluer vs kernel shclduer selecting the next thread shoudl eb more or less a wash
10:02:52 sean-k-mooney we could do it with perf but that might be overkill
10:03:07 gibi I guess somewhere in the /proc it is recorded
10:03:19 gibi as top knows it
10:03:29 sean-k-mooney top woudl also prtint it. yes its aviabel in proc, we can turn on atop in ci
10:03:43 sean-k-mooney i t was added not too long ago but i have not seen job using it
10:04:11 sean-k-mooney im just not sure if it will show cumlitive cpu tiem
10:04:36 gibi I will check I see podman login --username gibi --password
10:04:38 gibi ahh not that
10:04:46 gibi I see https://github.com/openstack/devstack/commit/1aac81ee881534276fd7d6540ed356a85d064a13
10:06:12 sean-k-mooney yep so i was looking at that patch a few times when looking at odd gate failerus and debated if it woudl help but i have not tried it yet
10:06:18 gibi it should be enabled by defautl but I don't see the logs in the run so I need to dig
10:06:48 sean-k-mooney i dont think its on by defaualt
10:07:11 sean-k-mooney https://github.com/openstack/devstack/commit/1aac81ee881534276fd7d6540ed356a85d064a13#diff-3fedb5d16b14e2fe731944ed0819eabefe91f42d456377e5ce6f0a1874232269R1097
10:07:16 sean-k-mooney shoudl card against it
10:07:31 sean-k-mooney we startar the base devstack josb with disable all services
10:07:42 sean-k-mooney but even without that i dont see this patch addign it to the default service list
10:08:13 sean-k-mooney gibi: but it shoudl jsut require added `enable_service atop`
10:08:57 sean-k-mooney or in ci i guess thats `atop: true`
10:10:17 sean-k-mooney gibi: oh i see you addign post-run: playbooks/nova-rally-fake-virt-threading/post.yaml and i guess that also runs rally a second time correct
10:11:05 gibi yes, it calls rally task restart that trigger rally to re-run the previous task
10:11:41 gibi https://review.opendev.org/c/openstack/nova/+/960130/16/roles/rerun-rally/tasks/main.yaml#11
10:12:02 gibi it would be nice to re-use the rally roles but it is not stored in the rally-openstack repo in a re-usable way
10:12:56 gibi also they hardcode a bunch of things that does not help reuse like the report generation does not handle multiple rally runs
10:13:28 gibi food even :)
10:14:36 sean-k-mooney enjoy. while we should do more testing this is all very promising congrats
10:54:50 opendevreview Merged openstack/nova master: Update master for stable/2025.2 https://review.opendev.org/c/openstack/nova/+/960745
10:55:03 opendevreview Merged openstack/nova master: reno: Update master for unmaintained/2023.1 https://review.opendev.org/c/openstack/nova/+/935117
10:55:10 opendevreview Merged openstack/placement master: Update master for stable/2025.2 https://review.opendev.org/c/openstack/placement/+/960509
10:55:31 opendevreview Merged openstack/nova stable/2025.2: Update .gitreview for stable/2025.2 https://review.opendev.org/c/openstack/nova/+/960743
10:55:42 opendevreview Merged openstack/nova stable/2025.2: Update TOX_CONSTRAINTS_FILE for stable/2025.2 https://review.opendev.org/c/openstack/nova/+/960744
12:22:05 opendevreview Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130
12:40:46 opendevreview Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130
13:30:49 opendevreview Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130
13:55:24 dansmith gibi: just trying to sanity check some of the stats there..
13:56:19 dansmith it seems like for the majority of the run everything is idle all the time, with the exception of a short thing at the front where we have idle_workers < max_workers, and I wonder if that's just because we're starting up
13:56:45 opendevreview Julien LE JEUNE proposed openstack/nova master: nova-condutor puts instance in error state https://review.opendev.org/c/openstack/nova/+/901655
13:57:02 dansmith I went looking for a comparison of how busy the workers were between the two.. like if the threaded mode was able to keep up with 5 workers...but only just, or if they both were taxed, but keeping up
13:57:36 dansmith just wondering if maybe that stats log line is not giving us accurate results (I assume the wall time is, and that looks good of course)
13:58:09 gibi dansmith: I think given the limited number of 5 computes the scheduler can keep up with the requests as the scheduling them is fairly easy
13:59:49 gibi I'm trying to crank up both the number of computes and the number of parallel VM boot requests to see if we can make the scheduler more busy

Earlier   Later