Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2026-02-19
11:52:15 frickler likely we will need to build new images, as IIUC the broken venv is preconfigured there
12:02:38 opendevreview ribaudr proposed openstack/nova master: FUP Add HW_PCI_LIVE_MIGRATABLE trait to PCI resource providers https://review.opendev.org/c/openstack/nova/+/977310
12:25:01 tkajinam frickler, seems so, looking at the failure still appearing
12:52:28 opendevreview ribaudr proposed openstack/nova master: FUP Add HW_PCI_LIVE_MIGRATABLE trait to PCI resource providers https://review.opendev.org/c/openstack/nova/+/977310
12:56:26 priteau frickler: ah, this is why I couldn't see a fresh installation of testtools
13:08:31 gibi dansmith: gmaan: after I implemented the singel long task executor I realized that we have a logical problem https://review.opendev.org/c/openstack/nova/+/977251/1#message-ff7b23fb4a19eee42289ff428220461ae3df2da8 The recommended limit for the concurrent live migration is wildly different from the number of parallel builds
13:09:25 gibi I don't think deplolyers will accept this approach to have 1 lm - 1 build or 10 lm - 10 build, config.
13:09:47 gibi we recommend a single lm but that is not a useful value for concurrent builds
13:10:22 gibi this experiment shows me that we might not able to avoid the complexity in https://review.opendev.org/c/openstack/nova/+/975924/1
13:12:38 sean-k-mooney gibi: ya those need to be split
13:13:05 sean-k-mooney we could still sue a semapor for teh ratelimiting
13:13:20 sean-k-mooney and just have the pool limit be seperate
13:13:33 sean-k-mooney we can warn if the semephor config option exceed the pool size
13:13:43 sean-k-mooney but i think that an ok compromise
13:33:37 opendevreview ribaudr proposed openstack/nova master: FUP Add HW_PCI_LIVE_MIGRATABLE trait to PCI resource providers https://review.opendev.org/c/openstack/nova/+/977310
13:34:59 gibi sean-k-mooney: we cannot really take the semaphore approach for live migration as we rely on cancellability of live migrations. If I use the shared executor and take a semaphore within the lm task then lm task waiting for the semaphore becomes non cancellable
13:35:23 gibi on lm tasks waiting in an executor queue is cancellable
13:35:51 sean-k-mooney it depend on how we do cancelation i guess
13:35:59 gibi we cancel futures
13:36:16 sean-k-mooney right but live migation supprot direct cancelation via the api as well
13:36:25 gibi sure we can re-engineer the lm abort logic but then that is complexity
13:36:26 sean-k-mooney so even if its in progess we can actuly cancel it
13:36:44 sean-k-mooney ya so for now i guess we coudl keep those executor seperate
13:37:02 sean-k-mooney but i dont think we want a seprate executor per config optionright
13:37:11 sean-k-mooney at elast not unless the idel at 0
13:37:42 gibi ahh there is different abort modes of cancellation of lm, but we basically loose the abort mode for lms that are not yet running due to the sempahore (limit'l
13:37:47 gibi (limit)
13:38:05 sean-k-mooney yes you can force compelte or abort
13:38:20 gibi if the lm is really executing the we abort via the driver
13:38:21 sean-k-mooney sorry i have not had time to properly load context
13:38:51 sean-k-mooney you are stating https://review.opendev.org/c/openstack/nova/+/975924/1 wont work right
13:39:01 sean-k-mooney or are you saying it will and we cant avoid the complexity
13:39:40 gibi I'm saying I think we probably need the complexity from https://review.opendev.org/c/openstack/nova/+/975924/1 as a shared executor in https://review.opendev.org/c/openstack/nova/+/977251/1#message-ff7b23fb4a19eee42289ff428220461ae3df2da8 does not work well
13:39:53 sean-k-mooney ack
13:40:28 sean-k-mooney i have not reviewd iether patch yet hence why im asking for your gut feeling on which one is more likely to work
13:41:30 gibi having a shared executor for build and snapshot might work, but we would need a separate for lm. Or we need the complexity from the wrapper that implements limits per task type.
13:41:54 sean-k-mooney i dont hate the idea of TaskTypeLimiterExecutorWrapper
13:42:14 sean-k-mooney i actully think we will want to have a task/proirty aware executro in the future
13:42:36 sean-k-mooney i think that is nessisary complexity rather then overengeinging given our usecases
13:42:53 sean-k-mooney but i ahve only really read the doc strings at this point
13:46:51 sean-k-mooney gibi: honestly i dont think https://review.opendev.org/c/openstack/nova/+/975924/1/nova/utils.py is that complex and i woudl personlly extend it to add a priorty filed to the task or type stuct or both
13:47:42 sean-k-mooney proirtyon type woudl mean all tasks fo this type has a default priorty we regestry when we registrer the type
13:48:22 sean-k-mooney that woudl allow use to still work properly if the concurnace on teh executor is less then the limtis for all the indiviual types
13:48:56 sean-k-mooney we can technially do that in your version as well as proposed
13:49:34 sean-k-mooney but obvioulsy we might want to express some prefence beyond arivle time in when the next task si enquened
13:50:12 sean-k-mooney gibi: if this is don correctly by the way this can just live en futureist
13:50:23 sean-k-mooney but im ok with the idea of building this out in nova first
13:53:39 gibi yeah
13:58:06 gibi given the closeness of FF I'm trying to make a way forward with a split approach. build and snapshot in a shared executor, and lm having its own executor.
13:58:22 gibi but I think the right approach is the TaskTypeLimiterExecutorWrapper
13:58:27 gibi we just don't have time
13:58:47 sean-k-mooney looking at it breifly i woudl be ok with proceeding in that direction too
13:59:22 sean-k-mooney that obviously not a rigours code review but directionally i think the trade offs you are makign make sense
14:01:14 gibi yeah if I would have 2 weeks I would go all in on the wrapper
14:01:33 gibi but if I want to be realistic and want nova-compute with threading in G then the split approach is less risky
14:11:29 opendevreview Balazs Gibizer proposed openstack/nova master: [compute]Use single long task executor https://review.opendev.org/c/openstack/nova/+/977251
14:11:30 opendevreview Balazs Gibizer proposed openstack/nova master: Run nova-compute in native threading mode https://review.opendev.org/c/openstack/nova/+/965467
14:42:53 dansmith gibi: you're saying you don't think it's reasonable to lift the live migration limit to match what is probably the higher build limit?
14:43:45 dansmith live migrations are controlled by the admins, so I guess I don't see that as a fatal compromise for the time being,
14:44:59 dansmith but even if you go with the semaphore approach, I think you can still handle cancelation if the tasks each immediately check to see if they've been canceled after they acquire the semaphore and exit if so, no? the operation will appear to be a bit sticky until the (or a) current one finishes, but I imagine that's happening today...
15:35:20 Uggla FYI: https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/thread/VTLDDSXUHPKON3WOKNMATFL7ERHFPCYB/
15:36:05 sean-k-mooney Uggla: my understandign was the bug scrub was previously ment ot happen in teh channel after the meeting
15:36:17 sean-k-mooney not part of it
15:36:38 sean-k-mooney but sure. for watcher we do bug scrubing in the meeting each week and it works fine
15:36:55 gmaan bauzas: this is also ready for your review https://review.opendev.org/c/openstack/nova/+/975586
15:37:20 gmaan gibi: bauzas: can either of you +w this one, I thought it was merged https://review.opendev.org/c/openstack/nova/+/975242/4
15:38:27 gmaan gibi: dansmith sean-k-mooney: for single executor, adding semaphore for lm makes it comples than the approach gibi mentioned. I also thought the same. to have separate executor for lm and shared for all other long running tasks
15:38:28 Uggla bauzas, gibi, elodilles, gmaan see 16:35 msg.
15:39:39 gmaan Uggla: yeah, I saw the email too. this cycle, gate is giving the best feeling of FF
15:39:42 dansmith gmaan: the only problem there is that 20 live migrations waiting on semaphores will consume all the builder threads, but the point is to limit the activity a bit, so I think that should be okay
15:40:14 Uggla sean-k-mooney, I would like to have a sync point during the meeting to restart it. Because I was lazy on that topic... :(
15:41:26 opendevreview Bence Romsics proposed openstack/nova master: WIP Functional reproducer for #2051685 https://review.opendev.org/c/openstack/nova/+/977331
15:42:34 gmaan dansmith: but as we recommend only 1 live migration at a time (currently), by common executor, will 10 (or say 5) parallel live migration be successful and on time ? I am wondering on that side too
15:43:14 opendevreview Bence Romsics proposed openstack/nova master: WIP Functional reproducer for #2051685 https://review.opendev.org/c/openstack/nova/+/977331
15:43:50 dansmith gmaan: you can limit parallel migrations manually by not issuing more than one of them at a time, but a semaphore (set to 1) for live migrations will limit them to one at a time... it's just that if you start 10 in parallel, 9 will consume threads from the pool that will prevent builds from happening too
15:44:28 gmaan ohk
15:45:49 dansmith the suggestion of putting them both into one pool and requiring them to be the same was a compromise to avoid the complexity that gibi seemed to not think was achievable in a short period of time.. that compromise is, of course, a compromise and has some restrictions
15:46:14 dansmith if we're not okay with those, then we could just punt and try to do the complex thing first
15:46:29 dansmith I tend to want an incremental approach where not everything will be perfect in the first round
15:49:42 gmaan live migration are doable by manager user also not just admin, but still 10 as default shoudl be ok and if operator make it lower then they know the limitation. if needed we can make it 20 as default ?
15:50:27 dansmith I thought the default for the limit was 10 and we _recommended_ they set it to one?
15:53:16 gmaan k, one will be issue i think
16:12:28 sean-k-mooney the default for the live migtion is 1 and we recomemnd it to be one
16:12:49 sean-k-mooney but for the long live pool 10 is proably a reasonable default
16:13:10 gmaan I am saying with single limit for executor, build is default to 10 so that will go on live migration also
16:16:09 sean-k-mooney i woudl prefer to still havea way to limit live migration with a semephro if possibel as i dont like the idea of changign the default by proxy to 10
16:17:03 sean-k-mooney but i dont think that is a conflciting request
16:17:34 sean-k-mooney we jsut keep the existign cofnig option and semepor for live migraton
16:30:10 opendevreview Koya Watanabe proposed openstack/nova-specs master: Repropose instance-metadata-tag-protection https://review.opendev.org/c/openstack/nova-specs/+/977339
16:34:20 opendevreview Koya Watanabe proposed openstack/nova-specs master: Repropose newly instance metadata/tag protection feature https://review.opendev.org/c/openstack/nova-specs/+/977339
16:55:29 sean-k-mooney gibi: melwitt will ye have time to look at https://review.opendev.org/c/openstack/nova/+/975859/4 and https://review.opendev.org/c/openstack/nova/+/975872/5 next week? its a bug fix so it is not as time sensitive as features so if not that is ok
16:55:34 gibi sorry I had to step away. I don't think 10 concurrent live migration is a good thing, as well as limiting the concurrent builds to 1. The semaphore approach makes it complicated to cancel queued live migrations waiting on the semaphore (=> complexity), but also 10 lm waiting on the sempahore will block any new build requests as the sempahore is take on the worker from the Executor. I understand this
16:55:40 gibi was as compromise but I think this is too big of a compromise. As a sort term I'm now pushing for build and snapshot sharing an executor and live migration having its own (as today). Then after G I would do the per task type limit with an executor wrapper
16:56:39 gibi and sorry again, but I have to step away again :/ (but I will read back)
17:03:00 gmaan I am good that way for G and it is simple. semaphore approach still consume the executor so it does not really solve the operation limitation. If we are going to make it more smarter in future then this approach is ok for now.
17:06:48 sean-k-mooney this approch beign gibis current patch with 1 pool for live migration and another for the rest of the backgorund task and no semaphor?
17:07:12 sean-k-mooney im ok with that for this release as well jsut making sure that we are talking about th esame thing
17:08:29 opendevreview Ghanshyam Maan proposed openstack/nova master: Add manager graceful shutdown, timeout, and wait https://review.opendev.org/c/openstack/nova/+/975586
17:11:22 gmaan bauzas: gibi thanks for review on graceful shutdown changes, I fixed the bauzas comment in this change itself instead of followup because anyways I need to change the other changes in that series.
17:11:44 gmaan this one https://review.opendev.org/c/openstack/nova/+/975586

Earlier   Later