Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2026-02-19
10:45:57 sean-k-mooney i need to tweak my old patch that we reverted for the basic testing of cpu_share_set and repo-spe that for nova-alt-config
10:51:19 opendevreview ribaudr proposed openstack/nova master: FUP Add HW_PCI_LIVE_MIGRATABLE trait to PCI resource providers https://review.opendev.org/c/openstack/nova/+/977310
11:13:41 sean-k-mooney lajoskatona: gibi tested and approved it will hopefully merge later today and unblock https://review.opendev.org/c/openstack/whitebox-tempest-plugin/+/975225
11:14:20 sean-k-mooney it has also been tested trasitivly via that change
11:17:47 lajoskatona sean-k-mooney, gibi: thanks
11:24:01 priteau Lots of jobs hitting post_failure in 2025.1 :( ModuleNotFoundError: No module named 'typing_extensions'
11:25:15 priteau That's the testtools issue
11:26:00 priteau Is there a patch for this already?
11:30:55 gibi priteau: as far as I know fix landed in testtools and we are waiting for a new release from testtools to get it
11:35:52 gibi priteau: https://github.com/testing-cabal/testtools/pull/570
11:36:26 priteau https://pypi.org/project/testtools/
11:36:29 priteau Released: 2 minutes ago
11:39:12 priteau But we would need a bump of upper-constraints?
11:39:35 priteau 2024.1 has testtools===2.7.1
11:40:48 tkajinam priteau, no
11:41:00 tkajinam priteau, the failing task was common for all branches and doesn't use u-c
11:41:06 tkajinam s/was/is/
11:43:25 priteau OK
11:43:30 priteau So recheck should be enough?
11:52:15 frickler likely we will need to build new images, as IIUC the broken venv is preconfigured there
12:02:38 opendevreview ribaudr proposed openstack/nova master: FUP Add HW_PCI_LIVE_MIGRATABLE trait to PCI resource providers https://review.opendev.org/c/openstack/nova/+/977310
12:25:01 tkajinam frickler, seems so, looking at the failure still appearing
12:52:28 opendevreview ribaudr proposed openstack/nova master: FUP Add HW_PCI_LIVE_MIGRATABLE trait to PCI resource providers https://review.opendev.org/c/openstack/nova/+/977310
12:56:26 priteau frickler: ah, this is why I couldn't see a fresh installation of testtools
13:08:31 gibi dansmith: gmaan: after I implemented the singel long task executor I realized that we have a logical problem https://review.opendev.org/c/openstack/nova/+/977251/1#message-ff7b23fb4a19eee42289ff428220461ae3df2da8 The recommended limit for the concurrent live migration is wildly different from the number of parallel builds
13:09:25 gibi I don't think deplolyers will accept this approach to have 1 lm - 1 build or 10 lm - 10 build, config.
13:09:47 gibi we recommend a single lm but that is not a useful value for concurrent builds
13:10:22 gibi this experiment shows me that we might not able to avoid the complexity in https://review.opendev.org/c/openstack/nova/+/975924/1
13:12:38 sean-k-mooney gibi: ya those need to be split
13:13:05 sean-k-mooney we could still sue a semapor for teh ratelimiting
13:13:20 sean-k-mooney and just have the pool limit be seperate
13:13:33 sean-k-mooney we can warn if the semephor config option exceed the pool size
13:13:43 sean-k-mooney but i think that an ok compromise
13:33:37 opendevreview ribaudr proposed openstack/nova master: FUP Add HW_PCI_LIVE_MIGRATABLE trait to PCI resource providers https://review.opendev.org/c/openstack/nova/+/977310
13:34:59 gibi sean-k-mooney: we cannot really take the semaphore approach for live migration as we rely on cancellability of live migrations. If I use the shared executor and take a semaphore within the lm task then lm task waiting for the semaphore becomes non cancellable
13:35:23 gibi on lm tasks waiting in an executor queue is cancellable
13:35:51 sean-k-mooney it depend on how we do cancelation i guess
13:35:59 gibi we cancel futures
13:36:16 sean-k-mooney right but live migation supprot direct cancelation via the api as well
13:36:25 gibi sure we can re-engineer the lm abort logic but then that is complexity
13:36:26 sean-k-mooney so even if its in progess we can actuly cancel it
13:36:44 sean-k-mooney ya so for now i guess we coudl keep those executor seperate
13:37:02 sean-k-mooney but i dont think we want a seprate executor per config optionright
13:37:11 sean-k-mooney at elast not unless the idel at 0
13:37:42 gibi ahh there is different abort modes of cancellation of lm, but we basically loose the abort mode for lms that are not yet running due to the sempahore (limit'l
13:37:47 gibi (limit)
13:38:05 sean-k-mooney yes you can force compelte or abort
13:38:20 gibi if the lm is really executing the we abort via the driver
13:38:21 sean-k-mooney sorry i have not had time to properly load context
13:38:51 sean-k-mooney you are stating https://review.opendev.org/c/openstack/nova/+/975924/1 wont work right
13:39:01 sean-k-mooney or are you saying it will and we cant avoid the complexity
13:39:40 gibi I'm saying I think we probably need the complexity from https://review.opendev.org/c/openstack/nova/+/975924/1 as a shared executor in https://review.opendev.org/c/openstack/nova/+/977251/1#message-ff7b23fb4a19eee42289ff428220461ae3df2da8 does not work well
13:39:53 sean-k-mooney ack
13:40:28 sean-k-mooney i have not reviewd iether patch yet hence why im asking for your gut feeling on which one is more likely to work
13:41:30 gibi having a shared executor for build and snapshot might work, but we would need a separate for lm. Or we need the complexity from the wrapper that implements limits per task type.
13:41:54 sean-k-mooney i dont hate the idea of TaskTypeLimiterExecutorWrapper
13:42:14 sean-k-mooney i actully think we will want to have a task/proirty aware executro in the future
13:42:36 sean-k-mooney i think that is nessisary complexity rather then overengeinging given our usecases
13:42:53 sean-k-mooney but i ahve only really read the doc strings at this point
13:46:51 sean-k-mooney gibi: honestly i dont think https://review.opendev.org/c/openstack/nova/+/975924/1/nova/utils.py is that complex and i woudl personlly extend it to add a priorty filed to the task or type stuct or both
13:47:42 sean-k-mooney proirtyon type woudl mean all tasks fo this type has a default priorty we regestry when we registrer the type
13:48:22 sean-k-mooney that woudl allow use to still work properly if the concurnace on teh executor is less then the limtis for all the indiviual types
13:48:56 sean-k-mooney we can technially do that in your version as well as proposed
13:49:34 sean-k-mooney but obvioulsy we might want to express some prefence beyond arivle time in when the next task si enquened
13:50:12 sean-k-mooney gibi: if this is don correctly by the way this can just live en futureist
13:50:23 sean-k-mooney but im ok with the idea of building this out in nova first
13:53:39 gibi yeah
13:58:06 gibi given the closeness of FF I'm trying to make a way forward with a split approach. build and snapshot in a shared executor, and lm having its own executor.
13:58:22 gibi but I think the right approach is the TaskTypeLimiterExecutorWrapper
13:58:27 gibi we just don't have time
13:58:47 sean-k-mooney looking at it breifly i woudl be ok with proceeding in that direction too
13:59:22 sean-k-mooney that obviously not a rigours code review but directionally i think the trade offs you are makign make sense
14:01:14 gibi yeah if I would have 2 weeks I would go all in on the wrapper
14:01:33 gibi but if I want to be realistic and want nova-compute with threading in G then the split approach is less risky
14:11:29 opendevreview Balazs Gibizer proposed openstack/nova master: [compute]Use single long task executor https://review.opendev.org/c/openstack/nova/+/977251
14:11:30 opendevreview Balazs Gibizer proposed openstack/nova master: Run nova-compute in native threading mode https://review.opendev.org/c/openstack/nova/+/965467
14:42:53 dansmith gibi: you're saying you don't think it's reasonable to lift the live migration limit to match what is probably the higher build limit?
14:43:45 dansmith live migrations are controlled by the admins, so I guess I don't see that as a fatal compromise for the time being,
14:44:59 dansmith but even if you go with the semaphore approach, I think you can still handle cancelation if the tasks each immediately check to see if they've been canceled after they acquire the semaphore and exit if so, no? the operation will appear to be a bit sticky until the (or a) current one finishes, but I imagine that's happening today...
15:35:20 Uggla FYI: https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/thread/VTLDDSXUHPKON3WOKNMATFL7ERHFPCYB/
15:36:05 sean-k-mooney Uggla: my understandign was the bug scrub was previously ment ot happen in teh channel after the meeting
15:36:17 sean-k-mooney not part of it
15:36:38 sean-k-mooney but sure. for watcher we do bug scrubing in the meeting each week and it works fine
15:36:55 gmaan bauzas: this is also ready for your review https://review.opendev.org/c/openstack/nova/+/975586
15:37:20 gmaan gibi: bauzas: can either of you +w this one, I thought it was merged https://review.opendev.org/c/openstack/nova/+/975242/4
15:38:27 gmaan gibi: dansmith sean-k-mooney: for single executor, adding semaphore for lm makes it comples than the approach gibi mentioned. I also thought the same. to have separate executor for lm and shared for all other long running tasks
15:38:28 Uggla bauzas, gibi, elodilles, gmaan see 16:35 msg.
15:39:39 gmaan Uggla: yeah, I saw the email too. this cycle, gate is giving the best feeling of FF
15:39:42 dansmith gmaan: the only problem there is that 20 live migrations waiting on semaphores will consume all the builder threads, but the point is to limit the activity a bit, so I think that should be okay
15:40:14 Uggla sean-k-mooney, I would like to have a sync point during the meeting to restart it. Because I was lazy on that topic... :(
15:41:26 opendevreview Bence Romsics proposed openstack/nova master: WIP Functional reproducer for #2051685 https://review.opendev.org/c/openstack/nova/+/977331
15:42:34 gmaan dansmith: but as we recommend only 1 live migration at a time (currently), by common executor, will 10 (or say 5) parallel live migration be successful and on time ? I am wondering on that side too
15:43:14 opendevreview Bence Romsics proposed openstack/nova master: WIP Functional reproducer for #2051685 https://review.opendev.org/c/openstack/nova/+/977331
15:43:50 dansmith gmaan: you can limit parallel migrations manually by not issuing more than one of them at a time, but a semaphore (set to 1) for live migrations will limit them to one at a time... it's just that if you start 10 in parallel, 9 will consume threads from the pool that will prevent builds from happening too
15:44:28 gmaan ohk
15:45:49 dansmith the suggestion of putting them both into one pool and requiring them to be the same was a compromise to avoid the complexity that gibi seemed to not think was achievable in a short period of time.. that compromise is, of course, a compromise and has some restrictions
15:46:14 dansmith if we're not okay with those, then we could just punt and try to do the complex thing first
15:46:29 dansmith I tend to want an incremental approach where not everything will be perfect in the first round
15:49:42 gmaan live migration are doable by manager user also not just admin, but still 10 as default shoudl be ok and if operator make it lower then they know the limitation. if needed we can make it 20 as default ?
15:50:27 dansmith I thought the default for the limit was 10 and we _recommended_ they set it to one?

Earlier   Later