Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2026-02-19
13:58:47 sean-k-mooney looking at it breifly i woudl be ok with proceeding in that direction too
13:59:22 sean-k-mooney that obviously not a rigours code review but directionally i think the trade offs you are makign make sense
14:01:14 gibi yeah if I would have 2 weeks I would go all in on the wrapper
14:01:33 gibi but if I want to be realistic and want nova-compute with threading in G then the split approach is less risky
14:11:29 opendevreview Balazs Gibizer proposed openstack/nova master: [compute]Use single long task executor https://review.opendev.org/c/openstack/nova/+/977251
14:11:30 opendevreview Balazs Gibizer proposed openstack/nova master: Run nova-compute in native threading mode https://review.opendev.org/c/openstack/nova/+/965467
14:42:53 dansmith gibi: you're saying you don't think it's reasonable to lift the live migration limit to match what is probably the higher build limit?
14:43:45 dansmith live migrations are controlled by the admins, so I guess I don't see that as a fatal compromise for the time being,
14:44:59 dansmith but even if you go with the semaphore approach, I think you can still handle cancelation if the tasks each immediately check to see if they've been canceled after they acquire the semaphore and exit if so, no? the operation will appear to be a bit sticky until the (or a) current one finishes, but I imagine that's happening today...
15:35:20 Uggla FYI: https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/thread/VTLDDSXUHPKON3WOKNMATFL7ERHFPCYB/
15:36:05 sean-k-mooney Uggla: my understandign was the bug scrub was previously ment ot happen in teh channel after the meeting
15:36:17 sean-k-mooney not part of it
15:36:38 sean-k-mooney but sure. for watcher we do bug scrubing in the meeting each week and it works fine
15:36:55 gmaan bauzas: this is also ready for your review https://review.opendev.org/c/openstack/nova/+/975586
15:37:20 gmaan gibi: bauzas: can either of you +w this one, I thought it was merged https://review.opendev.org/c/openstack/nova/+/975242/4
15:38:27 gmaan gibi: dansmith sean-k-mooney: for single executor, adding semaphore for lm makes it comples than the approach gibi mentioned. I also thought the same. to have separate executor for lm and shared for all other long running tasks
15:38:28 Uggla bauzas, gibi, elodilles, gmaan see 16:35 msg.
15:39:39 gmaan Uggla: yeah, I saw the email too. this cycle, gate is giving the best feeling of FF
15:39:42 dansmith gmaan: the only problem there is that 20 live migrations waiting on semaphores will consume all the builder threads, but the point is to limit the activity a bit, so I think that should be okay
15:40:14 Uggla sean-k-mooney, I would like to have a sync point during the meeting to restart it. Because I was lazy on that topic... :(
15:41:26 opendevreview Bence Romsics proposed openstack/nova master: WIP Functional reproducer for #2051685 https://review.opendev.org/c/openstack/nova/+/977331
15:42:34 gmaan dansmith: but as we recommend only 1 live migration at a time (currently), by common executor, will 10 (or say 5) parallel live migration be successful and on time ? I am wondering on that side too
15:43:14 opendevreview Bence Romsics proposed openstack/nova master: WIP Functional reproducer for #2051685 https://review.opendev.org/c/openstack/nova/+/977331
15:43:50 dansmith gmaan: you can limit parallel migrations manually by not issuing more than one of them at a time, but a semaphore (set to 1) for live migrations will limit them to one at a time... it's just that if you start 10 in parallel, 9 will consume threads from the pool that will prevent builds from happening too
15:44:28 gmaan ohk
15:45:49 dansmith the suggestion of putting them both into one pool and requiring them to be the same was a compromise to avoid the complexity that gibi seemed to not think was achievable in a short period of time.. that compromise is, of course, a compromise and has some restrictions
15:46:14 dansmith if we're not okay with those, then we could just punt and try to do the complex thing first
15:46:29 dansmith I tend to want an incremental approach where not everything will be perfect in the first round
15:49:42 gmaan live migration are doable by manager user also not just admin, but still 10 as default shoudl be ok and if operator make it lower then they know the limitation. if needed we can make it 20 as default ?
15:50:27 dansmith I thought the default for the limit was 10 and we _recommended_ they set it to one?
15:53:16 gmaan k, one will be issue i think
16:12:28 sean-k-mooney the default for the live migtion is 1 and we recomemnd it to be one
16:12:49 sean-k-mooney but for the long live pool 10 is proably a reasonable default
16:13:10 gmaan I am saying with single limit for executor, build is default to 10 so that will go on live migration also
16:16:09 sean-k-mooney i woudl prefer to still havea way to limit live migration with a semephro if possibel as i dont like the idea of changign the default by proxy to 10
16:17:03 sean-k-mooney but i dont think that is a conflciting request
16:17:34 sean-k-mooney we jsut keep the existign cofnig option and semepor for live migraton
16:30:10 opendevreview Koya Watanabe proposed openstack/nova-specs master: Repropose instance-metadata-tag-protection https://review.opendev.org/c/openstack/nova-specs/+/977339
16:34:20 opendevreview Koya Watanabe proposed openstack/nova-specs master: Repropose newly instance metadata/tag protection feature https://review.opendev.org/c/openstack/nova-specs/+/977339
16:55:29 sean-k-mooney gibi: melwitt will ye have time to look at https://review.opendev.org/c/openstack/nova/+/975859/4 and https://review.opendev.org/c/openstack/nova/+/975872/5 next week? its a bug fix so it is not as time sensitive as features so if not that is ok
16:55:34 gibi sorry I had to step away. I don't think 10 concurrent live migration is a good thing, as well as limiting the concurrent builds to 1. The semaphore approach makes it complicated to cancel queued live migrations waiting on the semaphore (=> complexity), but also 10 lm waiting on the sempahore will block any new build requests as the sempahore is take on the worker from the Executor. I understand this
16:55:40 gibi was as compromise but I think this is too big of a compromise. As a sort term I'm now pushing for build and snapshot sharing an executor and live migration having its own (as today). Then after G I would do the per task type limit with an executor wrapper
16:56:39 gibi and sorry again, but I have to step away again :/ (but I will read back)
17:03:00 gmaan I am good that way for G and it is simple. semaphore approach still consume the executor so it does not really solve the operation limitation. If we are going to make it more smarter in future then this approach is ok for now.
17:06:48 sean-k-mooney this approch beign gibis current patch with 1 pool for live migration and another for the rest of the backgorund task and no semaphor?
17:07:12 sean-k-mooney im ok with that for this release as well jsut making sure that we are talking about th esame thing
17:08:29 opendevreview Ghanshyam Maan proposed openstack/nova master: Add manager graceful shutdown, timeout, and wait https://review.opendev.org/c/openstack/nova/+/975586
17:11:22 gmaan bauzas: gibi thanks for review on graceful shutdown changes, I fixed the bauzas comment in this change itself instead of followup because anyways I need to change the other changes in that series.
17:11:44 gmaan this one https://review.opendev.org/c/openstack/nova/+/975586
17:36:36 opendevreview Merged openstack/nova master: Add 2nd RPC server for compute service https://review.opendev.org/c/openstack/nova/+/975242
18:09:09 opendevreview melanie witt proposed openstack/nova master: TPM: fixups for live migration of `host` secret security https://review.opendev.org/c/openstack/nova/+/976316
18:09:09 opendevreview melanie witt proposed openstack/nova master: TPM: support live migration of `deployment` secret security https://review.opendev.org/c/openstack/nova/+/925771
18:09:10 opendevreview melanie witt proposed openstack/nova master: TPM: bump service version to enable live migration https://review.opendev.org/c/openstack/nova/+/975724
18:09:10 opendevreview melanie witt proposed openstack/nova master: TPM: test live migration between hosts with different security https://review.opendev.org/c/openstack/nova/+/952629
18:09:11 opendevreview melanie witt proposed openstack/nova master: TPM: add late check for supported TPM secret security https://review.opendev.org/c/openstack/nova/+/956975
18:09:11 opendevreview melanie witt proposed openstack/nova master: TPM: enable conversion of secret security modes via resize https://review.opendev.org/c/openstack/nova/+/962052
18:09:12 opendevreview melanie witt proposed openstack/nova master: DNM vtpm tempest https://review.opendev.org/c/openstack/nova/+/957477
18:57:16 gmaan gibi: In my graceful shutdown change, i saw test_submit_second_while_delaying_first failing in threading job AssertionError: 1.997275639999998 not greater than 2.0
18:57:17 gmaan https://1069208da190f941bbcb-6faf22591116ac424591f44dbeb2cb9b.ssl.cf1.rackcdn.com/openstack/0cc7f4aa239e491493e73f88b45c8986/testr_results.html
18:57:45 gmaan not sure you or sean-k-mooney talked about it but just to let you know that this is happening in more places
18:58:24 sean-k-mooney gmaan: melwitt mentioned it a day or two ago i think
18:58:31 gmaan ohk
18:58:51 sean-k-mooney i didnt look at it in too much detail but we likely neeed to mock time slightly diffently
18:59:18 sean-k-mooney that or use the assert almsot equesl th9ing for floats
18:59:55 sean-k-mooney its defnitly a semi flaky test but i hav enot see it fail much
19:00:17 gmaan k, I just saw it in my change but not anywhere else
19:01:32 gmaan its for delay task StaticallyDelayingCancellableTaskExecutorWrapper with delay of 2 so task should not finish before 2 if we check float then it will fail too
19:03:02 sean-k-mooney well sleep shodl not be less then the ammount but i have not looked at the code
19:03:10 gmaan or maybe it is just matter of next line captired the monotomic time after task is submitted. maybe we can capture time before task is submitted
19:03:26 gmaan let me propose the change and see if that make sense
19:03:33 sean-k-mooney ack
19:04:20 sean-k-mooney im just finishing up but ill be around for a few more mins so if you push it before i wrap for the weekend ill review it quickly
19:04:58 gmaan k, give me few min
19:09:39 sean-k-mooney we could just assert >=1.9
19:09:49 sean-k-mooney but ya its the order of https://github.com/openstack/nova/blob/7a303bc1e28e9426f2f6d9898a18edda34bb8dd9/nova/tests/unit/test_utils.py#L2166-L2167
19:10:15 sean-k-mooney its already submited ot teh executor at that point when we recored teh teim
19:13:24 opendevreview Ghanshyam Maan proposed openstack/nova master: Fix the flasky test test_submit_second_while_delaying_first https://review.opendev.org/c/openstack/nova/+/977356
19:13:46 gmaan sean-k-mooney: melwitt gibi ^^
19:14:09 sean-k-mooney that is exactly the fix i was expecting so +2 :)
19:14:16 gmaan thanks
19:14:45 gmaan and it seems py310 job also green
19:15:09 sean-k-mooney already or locally
19:15:33 gmaan already, i can see it is passing in my change
19:15:34 sean-k-mooney locally i gues because it has not run yet in the gate
19:15:50 sean-k-mooney https://zuul.openstack.org/status?change=977356
19:15:51 gmaan in ohter change i mean which is still in gate
19:16:03 sean-k-mooney oh ok
19:16:44 sean-k-mooney is the grenade issue fixed out of interest
19:17:03 sean-k-mooney ModuleNotFoundError: No module named 'typing_extensions'
19:17:05 gmaan which one? is there new one
19:17:12 sean-k-mooney that failing in watcher
19:17:28 gmaan oh, i can see grenade job also passing in my nova change
19:17:45 gmaan I think it was same in py310 also, i did not dig into it but same error
19:18:12 sean-k-mooney oh ok i tought i saw folks taling about this in one of the irc channle
19:18:27 sean-k-mooney i think its a testtools issue and we ewre waiting on the new release
19:18:32 sean-k-mooney which happend eailer today
19:18:46 gmaan yeah
19:19:47 sean-k-mooney ok well that wil either pass or not but should be resovled one way or the ther by monday
19:20:03 gmaan this is merged so all green https://review.opendev.org/c/openstack/nova/+/975242/4
19:20:16 gmaan you can recheck maybe

Earlier   Later