| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-03-25 | |||
| 13:40:50 | sean-k-mooney | and since we hope to have at least 1 for the maniala shares | |
| 13:40:55 | sean-k-mooney | im not sure this is a concern | |
| 13:41:16 | bauzas | well, this is not a concern from me, rather an open question | |
| 13:41:19 | artom | sean-k-mooney, so, NUMA placement is deterministic, right? (We don't promise it, but de facto it is)... So if this is not happening 100% of the time, it means there's a race or something similarly *non-deterministic* | |
| 13:41:25 | sean-k-mooney | personally i think this is somethign that shoudl be backported once fixed but again we can caht about that | |
| 13:41:41 | bauzas | I wonder #1 whether we need a microversion #2 whether we can backport it | |
| 13:41:48 | artom | When you say "reproduced", you mean the logs for the fitting of network metadata not appearing? | |
| 13:42:03 | bauzas | what I'd prefer is that #1 no and #yes :) | |
| 13:42:05 | sean-k-mooney | artom: deteministic in the sense that if you had only one host and you started with it empty and booted vms in the same sequence you would expect the same result yes | |
| 13:42:32 | sean-k-mooney | artom: obviouly in reality you have mutiple concurrent requests | |
| 13:42:33 | bauzas | sorry, meant : MHO : #1 no and #2 yes (no microversion, and yeah we can backport) | |
| 13:42:42 | sean-k-mooney | so you cant really prediect it 100% | |
| 13:42:47 | bauzas | but let's ask gmann and other folks | |
| 13:42:54 | sean-k-mooney | artom: but for functioanl tests it shoudl be prediable | |
| 13:43:10 | artom | sean-k-mooney, but in the context of a functional test with one host and one (or two) instances, it's fully predictable | |
| 13:43:32 | sean-k-mooney | artom: it should be yes | |
| 13:43:56 | sean-k-mooney | give me a minute to finish up a few more things and if you want we can try pair programming a repoducer in a bit | |
| 13:44:11 | artom | So if I'm not reproducing it (barring the possibility of me doing something wrong with my test), it means the cause is something non-deterministic... | |
| 13:44:32 | sean-k-mooney | artom: no it could mean our test fixtures are maskign it | |
| 13:45:36 | sean-k-mooney | on firday i tought i knew how to repoduce it but i need to recall what that was. | |
| 15:06:08 | dansmith | mnaser: do you have any sort of preprod environment you could test the eventlet hub switch-to-asyncio for us? | |
| 15:06:34 | dansmith | I'm worried about performance and stability issues by just blindly making that change, and I don't think a small-node-count devstack tempest run is good enough to say it's reasonable | |
| 15:07:04 | mnaser | dansmith: lol i was literally reading yesterday about the switch and searching the number of refs of eventlet in codesearch.opendev.org yesterday lol | |
| 15:07:07 | dansmith | and/or some willing tenant that could take the hit in the name of science? | |
| 15:07:20 | dansmith | ah, excellent | |
| 15:07:29 | mnaser | i assume this hits several services | |
| 15:07:45 | mnaser | so not really a "run a few computes with the new code" for example | |
| 15:08:30 | dansmith | no, we'd need coverage for really any service running eventlet in any form, which to me, means even nova-api which doesn't run eventlet proper, but which still depends on some of the plumbing | |
| 15:08:51 | dansmith | a couple compute nodes would be a start, and could be enough to signal "yeah, no, not going to work out of the box" | |
| 15:09:02 | dansmith | necessary but not sufficient | |
| 15:09:33 | mnaser | how intrusive are the changes in the code right now (i guess i can go try and find the review) | |
| 15:09:50 | mnaser | or since its the step 1 in the transition docs its just a one liner switching to the asyncio hub | |
| 15:10:12 | dansmith | yeah, the hub one-liner is what I'm talking about here | |
| 15:10:23 | dansmith | maybe check the latest discussion from this morning on the patch for context | |
| 15:11:04 | dansmith | https://review.opendev.org/c/openstack/governance/+/902585 | |
| 15:13:11 | bauzas | for me, I still need to bounce on this governance patch | |
| 15:13:46 | dansmith | I don't know what that means | |
| 15:14:14 | mnaser | dansmith: i wouldn't mind putting this in somewhere and testing but i think for me if there isn't anyone driving the work after the exercise ends up for no reason ;p | |
| 15:14:22 | bauzas | I meant to say "I would want to review it" :) | |
| 15:14:55 | dansmith | mnaser: okay, I think the point is that if it "just works" we'd start making that our default | |
| 15:15:15 | dansmith | mnaser: this is probably going to be everyone's problem pretty soon here so I'm not sure I'm too worried about it not getting attention | |
| 15:15:35 | dansmith | it already exploded and got some immediate attention a few months ago, which is why this spec exists | |
| 15:16:43 | mnaser | dansmith: in that case, if it "just works" in devstack, i'll be happy to flip the switch over in some envs here and report on what we're seeing | |
| 15:17:00 | mnaser | i'd have to double check what is the earliest version of eventlet that supports this change though | |
| 15:17:11 | mnaser | i.e. how recent of an openstack cloud you'd need to test this out | |
| 15:17:29 | dansmith | yeah, might need the recent stuff from eventlet I think | |
| 15:17:55 | dansmith | I think we're expecting it will "just work" from the devstack perspective, but I'm worried that under load we'll see performance and/o deadlock type issues | |
| 15:18:04 | dansmith | but yeah, we can get a sniff test first for sure | |
| 15:19:30 | mnaser | it seems we'll need to bump eventlet version | |
| 15:19:42 | mnaser | https://github.com/eventlet/eventlet/commits/e1afe7b387a2452c47a98b596c9451f9d95f7bdc/eventlet/hubs/asyncio.py | |
| 15:20:04 | mnaser | seems like it was added since 0.35.2 and thats not even in bobcat so it might require a new env | |
| 15:21:05 | mnaser | rally seems like a good idea, i'd be happy to help stand up and env to run rally against since one of the concerns i'd have is well -- if you need to run caracal for this i dont know if i can find something that's running that with enough of a workload :) | |
| 15:21:22 | dansmith | okay | |
| 15:22:19 | mnaser | i assume the challenge is not going straight to asyncio is the fact there are a lot of eventlet based dependencies such as oslo messaging etc? | |
| 15:22:49 | dansmith | parse error | |
| 15:22:53 | JayF | That's the idea, we want to have a migration where openstack can run eventlet/asyncio side by side so we can migrate off eventlet (and onto asyncio, depending on the outcome of testing+the goal) | |
| 15:23:04 | dansmith | the challenge *with* going straight to asyncio/ | |
| 15:23:30 | dansmith | mnaser: asyncio is a vastly different programming model from eventlet | |
| 15:23:36 | clarkb | asyncio is cooperative and eventlet pretends to not be | |
| 15:23:38 | clarkb | ya that | |
| 15:23:43 | dansmith | mnaser: it's not a small change, it's a rewrite everything sort of deal | |
| 15:24:04 | mnaser | ah, so ctrl+f and looking at the number of 'eventlet' references is not an indicator of how challenging it might end up being | |
| 15:24:11 | dansmith | what we're discussing is a hack to let eventlet use asyncio as its sort of thread-switching engine, but it's just one step | |
| 15:24:20 | dansmith | mnaser: not even remotely close to that no :) | |
| 15:24:23 | sean-k-mooney | mnaser: correct its not | |
| 15:24:32 | dansmith | mnaser: ctrl-f and search for "def.*" would be more accurate :) | |
| 15:25:21 | dansmith | switching to real threads would be more of what you're describing, since the model is the same | |
| 15:25:43 | mnaser | fair enough, so if this doesn't work out we'd be in a bit of trouble | |
| 15:25:44 | dansmith | to extend what clarkb said, eventlet is the programming model of real threads but with the cooperative sort of behavior of asyncio | |
| 15:26:03 | mnaser | as in, it'd have to be an all-or-nothing and that is not what anyoen wants | |
| 15:26:23 | dansmith | tbh, I think we're in a pickle (maybe not trouble) regardless | |
| 15:26:57 | dansmith | if this hub hack doesn't just magically work, then I think my recommendation for the path forward will likely change (and be different per project/service) | |
| 15:27:47 | dansmith | if it does, all we really gain is kinda getting a chunk of the bottom end of eventlet out of the way, but it doesn't change the fact that we still need all the monkeypatching and top end of eventlet going forward | |
| 15:33:48 | JayF | Yeah, we'll still have a lot of eventlet-implicit-depedencies to fix -- e.g. like using asyncio entrypoints for sqlalchemy | |
| 15:34:07 | JayF | I'm extremely concerned about how things will operate -- and if performance will be harmed -- in the middle of that part of migrating | |
| 15:34:29 | JayF | When we have some portion of the stack on asyncio and some using implicit greening via eventlet | |
| 15:35:26 | dansmith | the actual mixing isn't so much the problem as it is the invisibility of the non-async call stacks above the asyncio hub I think | |
| 15:35:27 | dansmith | however, | |
| 15:35:51 | dansmith | there's really no slow migration path for existing code.. you can't really inject asyncio stuff at the lower layer, you kinda have to start at the top | |
| 15:36:13 | dansmith | there are some hacks, like making a sync method start an asyncio event loop just to make one async call, etc, but it's pretty heavy and gross, IME | |
| 15:37:59 | sean-k-mooney | sorry was on a call | |
| 15:38:43 | sean-k-mooney | so the hub just means wo can mix asyncio native code and eventlet | |
| 15:39:03 | sean-k-mooney | but without monkeypatching everything it buys us nothing | |
| 15:39:12 | dansmith | yup | |
| 15:39:46 | sean-k-mooney | and to transtation nova to async io while supprotign both hubs we would need to wrap all code that is monkeypatch today behind facde apis | |
| 15:39:55 | sean-k-mooney | and dispatch to the relevent implemention | |
| 15:40:05 | sean-k-mooney | until we could fully drop supprot for the old model | |
| 15:40:12 | sean-k-mooney | which is a long and non trivial thing | |
| 15:40:22 | dansmith | right, which I think I covered above in the " | |
| 15:40:28 | dansmith | rewrite everything" description :) | |
| 15:40:37 | sean-k-mooney | yep more or less | |
| 15:41:22 | dansmith | while everything won't be rewritten, knowing where those seams should go will require inspection on the order of "rewrite everything" IMHO.. we'll have to effectively consider everything needing a rewrite and exclude things we can avoid async'ing | |
| 15:42:20 | sean-k-mooney | yep and it will require use to also pessimies some code paths by adding in the check for "which hub am i using" | |
| 15:42:29 | dansmith | anyway, I have to run to an appointment soon, and I'm sure we'll get more PTG discussion on this..I think we need to sniff-test the switch with devstack in order to earn a report from mnaser with a more realistic set of nodes and load | |
| 15:42:30 | sean-k-mooney | plus the indrection | |
| 15:43:14 | sean-k-mooney | i dont currently have a devstack to playwith but i can proably just hack in an export of the hub env var | |
| 15:43:24 | sean-k-mooney | just to see if we can even deploy any of the existing services with it | |
| 15:43:53 | JayF | I suspect upstream eventlet (which is mostly Herve and friends these days) would consider it a bug if not | |
| 15:43:55 | sean-k-mooney | doing the proper enablement will take more time | |
| 15:44:37 | sean-k-mooney | JayF: Sure but bug are definetly a thing so you know we need to try it an see | |