Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2024-10-21
14:35:43 gibi ping me with the separate bug and I will quickly do the triage of it
14:37:21 jlejeune ok
14:55:02 jlejeune gibi: https://bugs.launchpad.net/nova/+bug/2085135
15:09:46 gibi jlejeune: thank
15:44:06 johnthetubaguy dansmith: are you thinking threads for each task being better for what nova-compute is generally doing? I hadn’t really thought about that. I could see that limiting the code changes, I suppose?
15:44:29 dansmith johnthetubaguy: I want us to at least consider that yeah
15:44:35 bauzas johnthetubaguy: that was something we were considering from the last PTG
15:45:29 dansmith johnthetubaguy: I suspect we have some places where we think we're threadsafe, but aren't
15:45:38 dansmith so we'll need to fix that up, which we should have fixed anyway
15:46:14 dansmith but I think much of nova (potentially conductor as well) could just move straight to real threads, ops can control how much parallelism they want (which they've asked for before) to scale performance, and then we can move on
15:47:17 johnthetubaguy Yeah, I had not thought of it like that. Its a good point.
15:49:06 clarkb just as a point of info zuul runs largely with real threads (I think the websocket lib we use may involve asyncio but I can't remember)
15:49:32 JayF ^ is what dtantsur has been pitching for IPA, although we've written no code in that direction
15:49:38 clarkb it works reasonably well and in theory will work better when we can drop the gil and sort out all of the places we missed extra lockign (but we already coordinate multiple processes with zookeeper so don't really need that scaling approach with dropping gil)
15:50:25 dtantsur Yep. Not all of our project needs to handle an extremely high concurrency.
15:50:40 dtantsur Especially Ironic, which has "I'm busy, come back later" as an inherent part of its API :)
15:51:20 johnthetubaguy Honestly, most of the bugs I find are where we attempt high concurrency badly, hard blocking on some of the API calls and tasks would totally make me operator self happier :)
15:51:25 opendevreview Artom Lifshitz proposed openstack/nova master: libvirt pwr mgmt: reproduce bug 2085122 https://review.opendev.org/c/openstack/nova/+/932922
16:05:12 sean-k-mooney gibi: im not actully sold on usign ayncio for the timout stuff by the way
16:05:38 sean-k-mooney gibi: it was just going to be my next poc, warp the existing fucntion in awaitlet and dispatch ot asyncio on a seperat ethead
16:05:56 sean-k-mooney specificlay for the nova-api sacather gather usecause
16:06:08 dansmith I'm quite sure that doing that will not be a magic bullet unless we're actually running the asyncio event loop
16:06:32 sean-k-mooney right that woudl be idea run the asyncio eventloop in a sperate thread nad dispatch the async task into it
16:06:58 sean-k-mooney again this is just one of the many altertinitve i was condiering
16:07:10 sean-k-mooney im not conviced it is a good approch but its what was suggested in teh wiki
16:07:28 dansmith yeah, but unless we're the parent, I think that the thing that spawned us may receive the SIGALRM and not deliver it to our event loop as a result meaning it doesn't really work like we think if it's blocked in a C call
16:08:06 dansmith anyway, we should dive into a poc and confirm
16:08:13 sean-k-mooney https://wiki.openstack.org/wiki/Eventlet-removal#3._Timeout_and_Deadline_Management_2
16:09:06 gibi sean-k-mooney: for me more moving pieces sounds worse, e.g. awaitlet + asyncio + a worker thread running the eventloop is just too much complexity.
16:09:17 sean-k-mooney i did a poc of this before https://review.opendev.org/c/openstack/nova/+/904425/10/nova/context.py#422
16:09:28 sean-k-mooney with a diffent approch
16:09:35 sean-k-mooney gibi: ack which is valid
16:10:17 sean-k-mooney gibi: i dont think the event approch you suggest really works as we cant assume we can generially check the event within the fucntio we are dispatching
16:10:22 dansmith gibi: ++
16:10:30 sean-k-mooney gibi: sqlachmen does supprot a statement_timeout
16:10:43 sean-k-mooney which i think is going to be the answer for db queies
16:10:54 sean-k-mooney i think this is currently set per session or conenction
16:11:05 gibi sean-k-mooney: my point that we need to make assumption about the functions we run as a task or we need to prepare to abandon them from the gather thread
16:11:07 dansmith sean-k-mooney: "To handle signals the event loop must be run in the main thread."
16:11:25 sean-k-mooney dansmith: sure but we cant use sinals for this
16:11:40 dansmith I think that unless you're the main thread, you can't delegate things to executors and expect things like timeouts to preempt
16:11:45 sean-k-mooney when run under mod_wsig some of those are uncondtionally handeled by apache
16:12:05 sean-k-mooney so i dont think we can rely on sig_arlarm in general
16:12:36 dansmith right, but I think that's how asyncio handles preemptable timeouts right?
16:12:47 sean-k-mooney i do not think so
16:12:58 sean-k-mooney asycio also works on windows where it cant
16:13:18 dansmith sure, but it doesn't have to implement the primitives in the same way
16:13:26 johnthetubaguy What is the argument against brute forcing it with extra threads (for the scatter/gather) apache just says no? Or just worried about the system resources/latency of the extra thread?
16:13:57 sean-k-mooney johnthetubaguy: its not really the scather gahter that the concern
16:13:58 dansmith I've just run into some weirdness when trying to run asyncio compatible stuff from deep inside "regular" code and it seems like it was related to some of the caveats around the event loop being in the main thread
16:14:13 sean-k-mooney we coudl remove that but we currently use eventlet to timeout queiry that done return
16:14:14 gibi johnthetubaguy: threads are not cancellable so we cannot easily port the today's timeout handling
16:14:29 dansmith right ^
16:14:30 sean-k-mooney so i think the reall fix for that is the sql alchemey statement timeout
16:14:34 johnthetubaguy Ah… got you.
16:14:36 dansmith sean-k-mooney: eight
16:14:44 dansmith sean-k-mooney: *right
16:14:51 sean-k-mooney process can be killd but that not reaonble ot do in the api
16:15:39 gibi sean-k-mooney: sure, if all the blocking call can be made non blocking with a timeout then we can just push down our timeout handling to the bottom of the call stack
16:15:39 sean-k-mooney ok so i think ill pause lookign at other "timeout methods" that work for arbitry funcitons
16:15:57 sean-k-mooney and insted look at how we can pass timeouts to existing libs directly i.e. sqlachmey
16:16:31 gibi yeah, I think we should not chase the "timeut for arbitrary function" direction with native threads. But accept that we need to per task type based solution
16:16:44 dansmith agree
16:17:15 sean-k-mooney that is fine with me.
16:17:35 sean-k-mooney i was hoping to limit the scope fo the refactors but lets expore that instead then
16:18:22 sean-k-mooney to be clear changing form eventlet.timeout to native timeouts in sqlachemy shoudl work with or without eventlet monkypatching
16:18:43 dansmith yeah
16:18:54 sean-k-mooney dansmith: gibi: would you condier this in or out of scope for this cycle. perhasps we shoudl dicsuss in ptg session
16:19:10 sean-k-mooney im not sure if that is high or low risk
16:19:14 gibi sean-k-mooney: do we have a timeout today both on the gather greenthread and on the task greenthreads doing the work?
16:19:15 dansmith I think we could get an eventlet.Timeout or a sqla.timeout at any point so we should already be handling those
16:20:13 gibi sean-k-mooney: I consider the work in scope for E, landing it for E might not happen but if I want to learn the devils in the details I need to work on an example, and this is a good example.
16:20:24 sean-k-mooney ack
16:20:26 dansmith also, AFAIK eventlet will let the timeout happen to unblock the waiter, but the worker thread actually blocked remains there to be GC'd until it frees up,
16:20:44 dansmith so using real timeouts should be better anyway to unblock and return threads to the pool quicker
16:21:18 sean-k-mooney when the timeout fire i belvie the eventlet calls greenthread.throw on the worker greentrhead
16:21:35 dansmith right but not in the actual worker thread which is blocked on a C call
16:21:43 sean-k-mooney but to dans point it may very well block for ever or untill the os kills the tcp socket
16:21:46 dansmith there's no way to generically interrupt the latter
16:22:08 dansmith the greenthread.throw is only handled as an exception when the blocking call returns
16:22:15 sean-k-mooney right
16:22:35 sean-k-mooney so until the os returns an interupt that resuble the worker thread it wont die
16:23:04 dansmith right
16:23:21 dansmith that's why moving to real library timeouts will benefit real threads and the green threads approache
16:25:41 sean-k-mooney ack for now the syncronos ceph lib does not supprot that but i think there async (not asynio) function that return futres can, still need to confirm that
16:29:20 dansmith ack
16:34:11 sean-k-mooney they do supprot timouts on the connect method and the "cil command" wrappers
16:34:20 sean-k-mooney https://docs.ceph.com/en/reef/rados/api/python/#rados.Rados.connect https://docs.ceph.com/en/reef/rados/api/python/#rados.Rados.mon_command
16:37:31 bauzas gouthamr: hey, what exactly do you want to discuss about virtiofs during nova sessions ? https://etherpad.opendev.org/p/nova-2025.1-ptg#L231
16:37:42 opendevreview Artom Lifshitz proposed openstack/nova master: libvirt pwr mgmt: reproduce bug 2085122 https://review.opendev.org/c/openstack/nova/+/932922
16:37:43 opendevreview Artom Lifshitz proposed openstack/nova master: pwr mgmt: power up instances's CPUs upon init https://review.opendev.org/c/openstack/nova/+/932926
16:39:15 gouthamr bauzas: i'd like to know the current status of the development / focus of the team going into Epoxy
16:39:31 bauzas okay, noted
16:39:37 opendevreview Artom Lifshitz proposed openstack/nova master: libvirt pwr mgmt: reproduce bug 2085122 https://review.opendev.org/c/openstack/nova/+/932922
16:39:38 opendevreview Artom Lifshitz proposed openstack/nova master: pwr mgmt: power up instances's CPUs upon init https://review.opendev.org/c/openstack/nova/+/932926
16:39:49 bauzas I'll provide a raw agenday tomorrow morning European time
16:56:01 johnthetubaguy bauzas: I believe its MIG vGPU not working with detach volume any more, Caracal onwards? Came up in the scientific SIG
16:56:50 bauzas johnthetubaguy: I may have said that MIG isn't officially supported in Nova
16:57:05 bauzas because that's nvidia specific and can't be documented upstream
16:58:37 johnthetubaguy bauzas: could you link me to where we say that? Its in the PCI docs I guess?

Earlier   Later