Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2024-10-21
16:05:38 sean-k-mooney gibi: it was just going to be my next poc, warp the existing fucntion in awaitlet and dispatch ot asyncio on a seperat ethead
16:05:56 sean-k-mooney specificlay for the nova-api sacather gather usecause
16:06:08 dansmith I'm quite sure that doing that will not be a magic bullet unless we're actually running the asyncio event loop
16:06:32 sean-k-mooney right that woudl be idea run the asyncio eventloop in a sperate thread nad dispatch the async task into it
16:06:58 sean-k-mooney again this is just one of the many altertinitve i was condiering
16:07:10 sean-k-mooney im not conviced it is a good approch but its what was suggested in teh wiki
16:07:28 dansmith yeah, but unless we're the parent, I think that the thing that spawned us may receive the SIGALRM and not deliver it to our event loop as a result meaning it doesn't really work like we think if it's blocked in a C call
16:08:06 dansmith anyway, we should dive into a poc and confirm
16:08:13 sean-k-mooney https://wiki.openstack.org/wiki/Eventlet-removal#3._Timeout_and_Deadline_Management_2
16:09:06 gibi sean-k-mooney: for me more moving pieces sounds worse, e.g. awaitlet + asyncio + a worker thread running the eventloop is just too much complexity.
16:09:17 sean-k-mooney i did a poc of this before https://review.opendev.org/c/openstack/nova/+/904425/10/nova/context.py#422
16:09:28 sean-k-mooney with a diffent approch
16:09:35 sean-k-mooney gibi: ack which is valid
16:10:17 sean-k-mooney gibi: i dont think the event approch you suggest really works as we cant assume we can generially check the event within the fucntio we are dispatching
16:10:22 dansmith gibi: ++
16:10:30 sean-k-mooney gibi: sqlachmen does supprot a statement_timeout
16:10:43 sean-k-mooney which i think is going to be the answer for db queies
16:10:54 sean-k-mooney i think this is currently set per session or conenction
16:11:05 gibi sean-k-mooney: my point that we need to make assumption about the functions we run as a task or we need to prepare to abandon them from the gather thread
16:11:07 dansmith sean-k-mooney: "To handle signals the event loop must be run in the main thread."
16:11:25 sean-k-mooney dansmith: sure but we cant use sinals for this
16:11:40 dansmith I think that unless you're the main thread, you can't delegate things to executors and expect things like timeouts to preempt
16:11:45 sean-k-mooney when run under mod_wsig some of those are uncondtionally handeled by apache
16:12:05 sean-k-mooney so i dont think we can rely on sig_arlarm in general
16:12:36 dansmith right, but I think that's how asyncio handles preemptable timeouts right?
16:12:47 sean-k-mooney i do not think so
16:12:58 sean-k-mooney asycio also works on windows where it cant
16:13:18 dansmith sure, but it doesn't have to implement the primitives in the same way
16:13:26 johnthetubaguy What is the argument against brute forcing it with extra threads (for the scatter/gather) apache just says no? Or just worried about the system resources/latency of the extra thread?
16:13:57 sean-k-mooney johnthetubaguy: its not really the scather gahter that the concern
16:13:58 dansmith I've just run into some weirdness when trying to run asyncio compatible stuff from deep inside "regular" code and it seems like it was related to some of the caveats around the event loop being in the main thread
16:14:13 sean-k-mooney we coudl remove that but we currently use eventlet to timeout queiry that done return
16:14:14 gibi johnthetubaguy: threads are not cancellable so we cannot easily port the today's timeout handling
16:14:29 dansmith right ^
16:14:30 sean-k-mooney so i think the reall fix for that is the sql alchemey statement timeout
16:14:34 johnthetubaguy Ah… got you.
16:14:36 dansmith sean-k-mooney: eight
16:14:44 dansmith sean-k-mooney: *right
16:14:51 sean-k-mooney process can be killd but that not reaonble ot do in the api
16:15:39 gibi sean-k-mooney: sure, if all the blocking call can be made non blocking with a timeout then we can just push down our timeout handling to the bottom of the call stack
16:15:39 sean-k-mooney ok so i think ill pause lookign at other "timeout methods" that work for arbitry funcitons
16:15:57 sean-k-mooney and insted look at how we can pass timeouts to existing libs directly i.e. sqlachmey
16:16:31 gibi yeah, I think we should not chase the "timeut for arbitrary function" direction with native threads. But accept that we need to per task type based solution
16:16:44 dansmith agree
16:17:15 sean-k-mooney that is fine with me.
16:17:35 sean-k-mooney i was hoping to limit the scope fo the refactors but lets expore that instead then
16:18:22 sean-k-mooney to be clear changing form eventlet.timeout to native timeouts in sqlachemy shoudl work with or without eventlet monkypatching
16:18:43 dansmith yeah
16:18:54 sean-k-mooney dansmith: gibi: would you condier this in or out of scope for this cycle. perhasps we shoudl dicsuss in ptg session
16:19:10 sean-k-mooney im not sure if that is high or low risk
16:19:14 gibi sean-k-mooney: do we have a timeout today both on the gather greenthread and on the task greenthreads doing the work?
16:19:15 dansmith I think we could get an eventlet.Timeout or a sqla.timeout at any point so we should already be handling those
16:20:13 gibi sean-k-mooney: I consider the work in scope for E, landing it for E might not happen but if I want to learn the devils in the details I need to work on an example, and this is a good example.
16:20:24 sean-k-mooney ack
16:20:26 dansmith also, AFAIK eventlet will let the timeout happen to unblock the waiter, but the worker thread actually blocked remains there to be GC'd until it frees up,
16:20:44 dansmith so using real timeouts should be better anyway to unblock and return threads to the pool quicker
16:21:18 sean-k-mooney when the timeout fire i belvie the eventlet calls greenthread.throw on the worker greentrhead
16:21:35 dansmith right but not in the actual worker thread which is blocked on a C call
16:21:43 sean-k-mooney but to dans point it may very well block for ever or untill the os kills the tcp socket
16:21:46 dansmith there's no way to generically interrupt the latter
16:22:08 dansmith the greenthread.throw is only handled as an exception when the blocking call returns
16:22:15 sean-k-mooney right
16:22:35 sean-k-mooney so until the os returns an interupt that resuble the worker thread it wont die
16:23:04 dansmith right
16:23:21 dansmith that's why moving to real library timeouts will benefit real threads and the green threads approache
16:25:41 sean-k-mooney ack for now the syncronos ceph lib does not supprot that but i think there async (not asynio) function that return futres can, still need to confirm that
16:29:20 dansmith ack
16:34:11 sean-k-mooney they do supprot timouts on the connect method and the "cil command" wrappers
16:34:20 sean-k-mooney https://docs.ceph.com/en/reef/rados/api/python/#rados.Rados.connect https://docs.ceph.com/en/reef/rados/api/python/#rados.Rados.mon_command
16:37:31 bauzas gouthamr: hey, what exactly do you want to discuss about virtiofs during nova sessions ? https://etherpad.opendev.org/p/nova-2025.1-ptg#L231
16:37:42 opendevreview Artom Lifshitz proposed openstack/nova master: libvirt pwr mgmt: reproduce bug 2085122 https://review.opendev.org/c/openstack/nova/+/932922
16:37:43 opendevreview Artom Lifshitz proposed openstack/nova master: pwr mgmt: power up instances's CPUs upon init https://review.opendev.org/c/openstack/nova/+/932926
16:39:15 gouthamr bauzas: i'd like to know the current status of the development / focus of the team going into Epoxy
16:39:31 bauzas okay, noted
16:39:37 opendevreview Artom Lifshitz proposed openstack/nova master: libvirt pwr mgmt: reproduce bug 2085122 https://review.opendev.org/c/openstack/nova/+/932922
16:39:38 opendevreview Artom Lifshitz proposed openstack/nova master: pwr mgmt: power up instances's CPUs upon init https://review.opendev.org/c/openstack/nova/+/932926
16:39:49 bauzas I'll provide a raw agenday tomorrow morning European time
16:56:01 johnthetubaguy bauzas: I believe its MIG vGPU not working with detach volume any more, Caracal onwards? Came up in the scientific SIG
16:56:50 bauzas johnthetubaguy: I may have said that MIG isn't officially supported in Nova
16:57:05 bauzas because that's nvidia specific and can't be documented upstream
16:58:37 johnthetubaguy bauzas: could you link me to where we say that? Its in the PCI docs I guess?
16:59:32 johnthetubaguy (It might be vGPU not MIG directly, but I could well be confused)
17:01:29 sean-k-mooney the gap when is the write/read commends to add data to the volume
17:01:47 sean-k-mooney context of ^ is our current abllity to timeout ceph commands
17:02:23 sean-k-mooney johnthetubaguy: i see i tought we fixed that?
17:02:37 johnthetubaguy @bauzas https://review.opendev.org/c/openstack/nova/+/925037 Maybe it got fixed another way?
17:02:57 sean-k-mooney johnthetubaguy: there was a breakage casue by the alisa feature i think, i dont recall the full details i woudl have to find the review
17:03:07 bauzas johnthetubaguy: we don't exactly mention that nvidia bits are forbidden, but vgpu upstream docs somehow show it up https://docs.openstack.org/nova/latest/admin/virtual-gpu.html
17:04:17 bauzas oh ok, now I get the picture
17:04:33 sean-k-mooney johnthetubaguy: you dont happen to have the review do you. ill see if i can find it
17:05:15 sean-k-mooney johnthetubaguy: if i recall correctly we asked them to repoduce the failure in a fucntional test and then demonstrate there patch fixed it
17:05:21 sean-k-mooney but i dont think we ask for thid party ci
17:06:22 sean-k-mooney johnthetubaguy: was it related to https://review.opendev.org/c/openstack/nova/+/925037 https://bugs.launchpad.net/nova/+bug/1942345
17:06:55 bauzas guys, I need to drop for my sanity
17:07:19 bauzas we'll have long days of nova sessions so today is a hard stop
17:07:29 sean-k-mooney we fixed that with https://review.opendev.org/c/openstack/nova/+/806943
17:08:08 fungi i also linked it in the dalmation retrospective section of the ptg pad since bauzas asked
17:09:21 sean-k-mooney ack
17:10:33 fungi and yes, the proposer seemed to think he was being told 1. we don't support your use case, and 2. if you want this merged you need to set up your own ci to provide testing for it
17:10:36 johnthetubaguy To be clear, my motive is I know people using vGPU on Antelope, and we are about to SLURP them to Caracal, but no one can afford GPUs in their staging environments these days. So people raising the flag and saying it’s all broken make me worry, and want to know how I can help fix things. A chat in the retrospective sounds like a good idea, regardless.

Earlier   Later