| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-01-11 | |||
| 20:06:00 | melwitt | ok. at least for me if use_cow_images is not set to False, it will do qcow2 instead of raw despite images_type = raw or flat | |
| 20:06:18 | sean-k-mooney | i started with your raw job | |
| 20:06:32 | sean-k-mooney | but i might not have kept that in the local.conf when i finished editing it | |
| 20:07:07 | sean-k-mooney | ok ya | |
| 20:07:12 | sean-k-mooney | i removed that in my editing | |
| 20:07:22 | melwitt | ok. that's the only way I know of that what you're seeing could be happening. if you do have use_cow_images = False then something else is wrong that I haven't seen yet | |
| 20:07:26 | sean-k-mooney | so everything is working as expecte its using qcow for the vms because that is the default | |
| 20:07:54 | sean-k-mooney | no in my local.conf i don thave those set so we are getting the default | |
| 20:08:32 | melwitt | ok I see | |
| 20:09:25 | sean-k-mooney | so cool i wont have time to do much with this tomorrow but ill play around with it on monday and start reivewing the series | |
| 20:09:34 | sean-k-mooney | have you done any multi node testing. | |
| 20:09:38 | sean-k-mooney | i.e. cold/live migration | |
| 20:10:33 | sean-k-mooney | the current zuul jobs seam to be single node | |
| 20:11:39 | sean-k-mooney | on a related note i ported your scater gather patch on to my serise and i got a pretty clean zuul run https://review.opendev.org/c/openstack/nova/+/905285 | |
| 20:12:14 | sean-k-mooney | the main delta between what i have doen is how we monkeypatch | |
| 20:12:43 | melwitt | sean-k-mooney: hm looks like no. I did resize in place. I'll add a multinode job on that patch | |
| 20:13:09 | sean-k-mooney | ok cool ill add a second node and play with that locally too next week | |
| 20:14:38 | melwitt | sean-k-mooney: ok cool. I dunno if you saw I wrote up a comment on the scatter gather patch about how I am seeing requests getting "stuck" on nova-api, like it receives the request and does all the stuff but does not respond. I have seen this in other failures that are not related with my patch too but with the patch it seems to happen much more often | |
| 20:14:58 | sean-k-mooney | ya i did | |
| 20:15:13 | melwitt | oh ok | |
| 20:15:16 | sean-k-mooney | i wanted to see if i saw the same behvior | |
| 20:15:33 | sean-k-mooney | i was wondering if htat coudl be related to how/when monkey patching was done | |
| 20:15:39 | melwitt | ok yeah | |
| 20:15:58 | sean-k-mooney | but there is also this comment on my patch https://review.opendev.org/c/openstack/nova/+/905285/1/nova/compute/multi_cell_list.py | |
| 20:16:02 | sean-k-mooney | that might be related too | |
| 20:16:10 | sean-k-mooney | the timeout might need to be done in the try? | |
| 20:16:28 | sean-k-mooney | or rather the creation or the timer | |
| 20:16:32 | melwitt | that's good. really want to find what/why that's happening | |
| 20:16:42 | melwitt | yeah I was thinking about that too | |
| 20:17:14 | sean-k-mooney | im not familar enough with timers in python to say one way or another but i dont think it would hurt to include it in the try block | |
| 20:18:28 | sean-k-mooney | melwitt: another data point on the previosu patch that uses the standar lib queue https://review.opendev.org/c/openstack/nova/+/900681/8?tab=change-view-tab-header-zuul-results-summary i got a bunch of timeourts | |
| 20:18:35 | sean-k-mooney | but i have not had time to look at way | |
| 20:18:39 | sean-k-mooney | *why | |
| 20:18:49 | melwitt | yeah, agree. it would be safer to move it into the try, I was going to do that anyway. I think I followed this example initially https://eventlet.net/doc/modules/timeout.html#eventlet.timeout.Timeout | |
| 20:19:26 | melwitt | whoa that is a lot of timeouts indeed | |
| 20:21:56 | melwitt | sean-k-mooney: I wasn't sure how to interpret this when I looked at it too, does this mean the eventlet queue does something special to work with greenlets? "The eventlet.queue module implements multi-producer, multi-consumer queues that work across greenlets, with the API similar to the classes found in the standard Queue and multiprocessing modules." https://eventlet.net/doc/modules/queue.html | |
| 20:22:23 | melwitt | if so, that could be why | |
| 20:22:43 | sean-k-mooney | ya so you have done the queue chagne in the same patch that swap to futureist | |
| 20:22:49 | sean-k-mooney | i have them split out | |
| 20:22:55 | melwitt | right, I know :) | |
| 20:23:06 | sean-k-mooney | so it could be related | |
| 20:23:20 | melwitt | I'm saying if eventlet.queue is "special" and you're not using it, maybe queue won't work right. oh, but eventlet should be patching queue right? hm | |
| 20:23:43 | sean-k-mooney | ya i was assuming eventlet was patching it | |
| 20:23:45 | melwitt | if it's patching it I dunno why it wouldn't work 🤔 | |
| 20:23:45 | melwitt | if it's patching it I dunno why it wouldn't work 🤔 | |
| 20:23:48 | melwitt | yeah | |
| 20:24:12 | sean-k-mooney | ok i need to get dinner and feed frya. she is getting barky | |
| 20:24:16 | sean-k-mooney | caht to you tomrorow | |
| 20:24:23 | JayF | some eventlet queue usages have known bugs | |
| 20:24:33 | melwitt | haha k, talk to you tomorrow | |
| 20:24:35 | JayF | Let me read scrollback and see if you are in those categories, if so I can likely find the bugs | |
| 20:24:48 | JayF | itamarst has been digging them up left and right as part of the "fix eventlet" work | |
| 20:25:02 | JayF | He leaves me frightening issue links in my slack every now and then :D | |
| 20:26:37 | melwitt | JayF: it's the opposite, what we were chatting about. sean did an in-place swap from eventlet.queue to standard lib queue and got a lot of timed out jobs. which might not be related to the switch https://review.opendev.org/c/openstack/nova/+/900681/8?tab=change-view-tab-header-zuul-results-summary | |
| 20:26:48 | JayF | Yeah this tracks | |
| 20:26:55 | JayF | because I think there's an RLock used by stdlib queue | |
| 20:27:00 | JayF | which is still getting monkey patched in this case | |
| 20:27:01 | JayF | let me dig | |
| 20:27:28 | JayF | so my hypothesis is this: | |
| 20:27:45 | melwitt | earlier runs looked fine and I dunno if I see any difference in the code so I'm not sure | |
| 20:28:02 | JayF | by using a stdlib queue, you're exposing yourself to using threading.Nlock (N can be multiple things). Some of these, particularly Rlock, is bugged in inconsistent ways (I am still looking for the github issue) | |
| 20:28:31 | melwitt | I didn't look in detail yet | |
| 20:31:52 | melwitt | hm ok | |
| 20:32:26 | sean-k-mooney | hum ok not back for long what i might do instead is this | |
| 20:32:59 | sean-k-mooney | add a inderiction where we import a diffent queue based on if we are patched or not and cenntralise that in the utils module | |
| 20:33:27 | sean-k-mooney | so that we can drop the selection of which to use in one place if we wever fully remove eventlet | |
| 20:34:08 | melwitt | ack | |
| 20:34:09 | sean-k-mooney | that way if the code in question is not monkypatche it will use the std lib one and where it is it will use the eventlet one | |
| 20:34:44 | JayF | sean-k-mooney: is "monkeypatched" a boolean for you? You're not playing games with monkeypatching on/off for different modules, are you? | |
| 20:35:24 | sean-k-mooney | its gloabl | |
| 20:35:27 | JayF | sean-k-mooney: that I think is going to be an attractive pattern that may lead to more pain than it's worth, based on my understanding many of the eventlet bugs in python3.11/3.12 which are just ... unfixable by design are mostly at the edges of stdlib interacting with monkey patched bits of code | |
| 20:35:39 | JayF | OK good, hopefully that means you'll avoid the sharpest edges | |
| 20:35:41 | sean-k-mooney | so in my case it depens on what direcotry the console cript is defiend in | |
| 20:35:50 | sean-k-mooney | if its under nova.cmd its monky patched | |
| 20:35:56 | sean-k-mooney | if its under nova.bin its not | |
| 20:36:13 | JayF | but there's never a case, as conjectured above, where say, queue would be std lib but threading would be eventlet? | |
| 20:36:33 | sean-k-mooney | not unless i screwed up | |
| 20:36:51 | sean-k-mooney | so the nova api in my serise can be patch or unpatched | |
| 20:36:59 | sean-k-mooney | if run via the wsgi script its unpatched | |
| 20:37:11 | sean-k-mooney | if run via nova.cmd.nova-api it is | |
| 20:37:11 | JayF | okay cool, in that case my hypothesis is invalid and unlikely to be happening | |
| 20:37:20 | JayF | and I'd suggest potentially even enforcing that if possible | |
| 20:37:25 | sean-k-mooney | but within any binary its one of the otehr | |
| 20:38:07 | JayF | Do you mind a quick topic change? | |
| 20:38:21 | sean-k-mooney | sure but i need to leave in a few | |
| 20:38:25 | sean-k-mooney | whats on your mind | |
| 20:38:27 | JayF | Just curious for ironic-guest-metadata, would you mind doing the piece of that work of splitting out the libvirt code to the common compute driver? | |
| 20:38:44 | JayF | I'd prefer not operate in a portion of the nova codebase outside of the ironic driver codepath, just based on familiarity | |
| 20:39:00 | JayF | but I can if you are too busy helping save the world from eventlet (a project which I too am invested in) | |
| 20:39:18 | sean-k-mooney | hum i might be able to ill have to see if i have time but should nbot be terrible complex | |
| 20:39:42 | sean-k-mooney | ask me again next week and ill see if i can help | |
| 20:40:16 | sean-k-mooney | i need to make some progress on the healtcheck stuff but im hoping to spend most of the next 4-5 weeks until codefreeze mainaly on upstream nova stuff | |
| 20:41:19 | sean-k-mooney | JayF: the eventlet stuff is just somethign im looking at in my personal time | |
| 20:41:58 | sean-k-mooney | so im not goign to be spending that much time looking at it untill we have some more detailed disucssions in the next ptg | |
| 20:42:34 | sean-k-mooney | i know others have concerse over if moving to asyncio or realtreading is really the right move | |
| 20:48:20 | JayF | Honestly, I joke sometimes that I'm a sysadmin who pretends to be a developer, but what that means in reality is for stuff like "should we use asyncio or $other?" type discussions, I tend to let folks with stronger opinions and foundational knowledge battle it out :) | |
| 20:48:40 | JayF | my concern is that we use something well maintained, with reasonable patterns that anyone who works on openstack for the next decade will be able to follow | |
| 20:49:39 | JayF | so like, I have zero opinions about how we move; I have lots of opinions that moving is important, and doing it well is important (and as I've mentioned elsewhere, I've gotten a part time developer assigned to OpenStack upstream at GR-OSS, he is onboarding now and will be pointed at eventlet migrations for underresourced openstack projects once we have a migration path set) | |
| 22:22:54 | opendevreview | melanie witt proposed openstack/nova master: DNM test ephemeral encryption + resize: qcow2, raw, rbd https://review.opendev.org/c/openstack/nova/+/862416 | |
| 23:04:10 | opendevreview | melanie witt proposed openstack/nova master: DNM test ephemeral encryption + resize: qcow2, raw, rbd https://review.opendev.org/c/openstack/nova/+/862416 | |