Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2024-03-25
15:17:00 mnaser i'd have to double check what is the earliest version of eventlet that supports this change though
15:17:11 mnaser i.e. how recent of an openstack cloud you'd need to test this out
15:17:29 dansmith yeah, might need the recent stuff from eventlet I think
15:17:55 dansmith I think we're expecting it will "just work" from the devstack perspective, but I'm worried that under load we'll see performance and/o deadlock type issues
15:18:04 dansmith but yeah, we can get a sniff test first for sure
15:19:30 mnaser it seems we'll need to bump eventlet version
15:19:42 mnaser https://github.com/eventlet/eventlet/commits/e1afe7b387a2452c47a98b596c9451f9d95f7bdc/eventlet/hubs/asyncio.py
15:20:04 mnaser seems like it was added since 0.35.2 and thats not even in bobcat so it might require a new env
15:21:05 mnaser rally seems like a good idea, i'd be happy to help stand up and env to run rally against since one of the concerns i'd have is well -- if you need to run caracal for this i dont know if i can find something that's running that with enough of a workload :)
15:21:22 dansmith okay
15:22:19 mnaser i assume the challenge is not going straight to asyncio is the fact there are a lot of eventlet based dependencies such as oslo messaging etc?
15:22:49 dansmith parse error
15:22:53 JayF That's the idea, we want to have a migration where openstack can run eventlet/asyncio side by side so we can migrate off eventlet (and onto asyncio, depending on the outcome of testing+the goal)
15:23:04 dansmith the challenge *with* going straight to asyncio/
15:23:30 dansmith mnaser: asyncio is a vastly different programming model from eventlet
15:23:36 clarkb asyncio is cooperative and eventlet pretends to not be
15:23:38 clarkb ya that
15:23:43 dansmith mnaser: it's not a small change, it's a rewrite everything sort of deal
15:24:04 mnaser ah, so ctrl+f and looking at the number of 'eventlet' references is not an indicator of how challenging it might end up being
15:24:11 dansmith what we're discussing is a hack to let eventlet use asyncio as its sort of thread-switching engine, but it's just one step
15:24:20 dansmith mnaser: not even remotely close to that no :)
15:24:23 sean-k-mooney mnaser: correct its not
15:24:32 dansmith mnaser: ctrl-f and search for "def.*" would be more accurate :)
15:25:21 dansmith switching to real threads would be more of what you're describing, since the model is the same
15:25:43 mnaser fair enough, so if this doesn't work out we'd be in a bit of trouble
15:25:44 dansmith to extend what clarkb said, eventlet is the programming model of real threads but with the cooperative sort of behavior of asyncio
15:26:03 mnaser as in, it'd have to be an all-or-nothing and that is not what anyoen wants
15:26:23 dansmith tbh, I think we're in a pickle (maybe not trouble) regardless
15:26:57 dansmith if this hub hack doesn't just magically work, then I think my recommendation for the path forward will likely change (and be different per project/service)
15:27:47 dansmith if it does, all we really gain is kinda getting a chunk of the bottom end of eventlet out of the way, but it doesn't change the fact that we still need all the monkeypatching and top end of eventlet going forward
15:33:48 JayF Yeah, we'll still have a lot of eventlet-implicit-depedencies to fix -- e.g. like using asyncio entrypoints for sqlalchemy
15:34:07 JayF I'm extremely concerned about how things will operate -- and if performance will be harmed -- in the middle of that part of migrating
15:34:29 JayF When we have some portion of the stack on asyncio and some using implicit greening via eventlet
15:35:26 dansmith the actual mixing isn't so much the problem as it is the invisibility of the non-async call stacks above the asyncio hub I think
15:35:27 dansmith however,
15:35:51 dansmith there's really no slow migration path for existing code.. you can't really inject asyncio stuff at the lower layer, you kinda have to start at the top
15:36:13 dansmith there are some hacks, like making a sync method start an asyncio event loop just to make one async call, etc, but it's pretty heavy and gross, IME
15:37:59 sean-k-mooney sorry was on a call
15:38:43 sean-k-mooney so the hub just means wo can mix asyncio native code and eventlet
15:39:03 sean-k-mooney but without monkeypatching everything it buys us nothing
15:39:12 dansmith yup
15:39:46 sean-k-mooney and to transtation nova to async io while supprotign both hubs we would need to wrap all code that is monkeypatch today behind facde apis
15:39:55 sean-k-mooney and dispatch to the relevent implemention
15:40:05 sean-k-mooney until we could fully drop supprot for the old model
15:40:12 sean-k-mooney which is a long and non trivial thing
15:40:22 dansmith right, which I think I covered above in the "
15:40:28 dansmith rewrite everything" description :)
15:40:37 sean-k-mooney yep more or less
15:41:22 dansmith while everything won't be rewritten, knowing where those seams should go will require inspection on the order of "rewrite everything" IMHO.. we'll have to effectively consider everything needing a rewrite and exclude things we can avoid async'ing
15:42:20 sean-k-mooney yep and it will require use to also pessimies some code paths by adding in the check for "which hub am i using"
15:42:29 dansmith anyway, I have to run to an appointment soon, and I'm sure we'll get more PTG discussion on this..I think we need to sniff-test the switch with devstack in order to earn a report from mnaser with a more realistic set of nodes and load
15:42:30 sean-k-mooney plus the indrection
15:43:14 sean-k-mooney i dont currently have a devstack to playwith but i can proably just hack in an export of the hub env var
15:43:24 sean-k-mooney just to see if we can even deploy any of the existing services with it
15:43:53 JayF I suspect upstream eventlet (which is mostly Herve and friends these days) would consider it a bug if not
15:43:55 sean-k-mooney doing the proper enablement will take more time
15:44:37 sean-k-mooney JayF: Sure but bug are definetly a thing so you know we need to try it an see
15:45:34 dansmith right the point is, mnaser would like to know if it even works for a small deployment before spinning up a test cluster and rally run
15:46:15 sean-k-mooney uc currently clamps to eventlet===0.35.1 so its in 0.35.2 based on the docs ill need to check if eventlet===0.35.1 has it or ill need to bump uc in the patch
15:46:45 sean-k-mooney ok it is in 0.35.1
15:47:23 sean-k-mooney i have 13 mins before my next meeting give me a sec an ill see if i can hack a dnm patch
15:48:06 dansmith I think you need the latest to get the dns fix IIRC
15:52:16 sean-k-mooney https://review.opendev.org/c/openstack/devstack/+/914108
15:52:28 sean-k-mooney so ^ is the minimal change i belive
15:52:57 sean-k-mooney it wont enabel it for cli tools like nova-mange
15:53:08 sean-k-mooney but they also dont need eventlet today even if they are patched
15:53:29 sean-k-mooney so we can see if that cannary explode or not and then do somethign better
18:12:01 sean-k-mooney dansmith: so https://review.opendev.org/c/openstack/devstack/+/914108 exploded in keystone manage which is not even ment o use eventlet
18:12:20 sean-k-mooney so thats fun.
18:12:41 sean-k-mooney | File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/green/thread.py", line 36, in get_ident
18:12:43 sean-k-mooney 2024-03-25 16:04:04.700 | AttributeError: 'NoneType' object has no attribute 'getcurrent'
18:14:50 dansmith well, this is why I think merging a resolution to do a thing before it is even proven to *run* is a bad idea
18:16:20 sean-k-mooney im pretty sure this si form oslo log or oslo context
18:16:25 sean-k-mooney just tryign to grab the compute id
18:17:24 sean-k-mooney * thread id
18:17:26 sean-k-mooney not compute id
18:17:49 sean-k-mooney but apparnetly just defining the env_var has a sidefect fo doing some kind of patching
18:17:55 sean-k-mooney which i woudl not expect
18:20:24 dansmith if it's not even able to do this without crashing, I suspect we're a long ways from being a drop-in replacement with no stability or performance implications
18:21:45 JayF The first step on the journey tho, no matter how long, is to try it, find the issue, fix it, and get that cycle going. I know hberaud and others are engaged and invested on making this work; it'll be nice to give them fresh data to use towards that goal.
18:28:55 dansmith sure, but whether or not the asyncio hub is a good idea depends on some tests, IMHO
18:50:17 gmann bauzas: sean-k-mooney: read logs for instance action event. I do not think it is API change and should not require microversion bump. there is no change in response fields and It is populating the response with data we have in DB.
18:50:55 sean-k-mooney gmann: ya that was my sense too.
18:51:11 sean-k-mooney we can see if anyone else disagree tomorrow or the ptg
18:51:12 gmann From interop perspective, I think that is what this API is. "there are always possibilities of empty action and events". so it is very easily discoverable items in list
18:51:38 gmann sean-k-mooney: yeah, I will add my opinion in etherpad too and we can discuss in PTG
19:29:14 bauzas gmann: thanks for the short answer, and I'm fine with it, I was opiniating on the same point
19:36:39 opendevreview Merged openstack/nova master: Update contributor guide for 2024.2 Dalmatian https://review.opendev.org/c/openstack/nova/+/913481
#openstack-nova - 2024-03-26
13:53:55 opendevreview Anton Kurbatov proposed openstack/nova master: Fixing the 500 HTTP code in the metadata service for incorrect queries https://review.opendev.org/c/openstack/nova/+/914249
15:14:30 opendevreview ribaudr proposed openstack/nova master: Fix: Live migrating to a host with cpu_shared_set configured will now update the VM's configuration accordingly. https://review.opendev.org/c/openstack/nova/+/903706
15:14:31 opendevreview ribaudr proposed openstack/nova master: Test live migration between hosts with differnet cpu_shared_sets https://review.opendev.org/c/openstack/nova/+/913744
16:00:40 bauzas #startmeeting nova
16:00:40 opendevmeet Meeting started Tue Mar 26 16:00:40 2024 UTC and is due to finish in 60 minutes. The chair is bauzas. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:40 opendevmeet Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:40 opendevmeet The meeting name has been set to 'nova'
16:00:49 Uggla o/
16:01:07 auniyal o/
16:01:13 bauzas sorry guys, I don't have the agenda prepared
16:01:15 grandchild o/
16:01:17 bauzas but let's start

Earlier   Later