| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-01-11 | |||
| 19:05:40 | wncslln | sean-k-mooney: thanks for the help | |
| 19:38:46 | wncslln | sean-k-mooney: i found the problem xD probably the image that im using has no uefi/gpt. i downloaded the ubuntu cloud image with support to uefi and worked fine | |
| 19:44:37 | sean-k-mooney | cool | |
| 19:44:54 | sean-k-mooney | ya that make sense a lot of older images wont | |
| 19:45:15 | sean-k-mooney | and if you created it form an iso inn a vm you need to make sure that initall vm is also set up for uefi | |
| 19:45:33 | sean-k-mooney | same for windows as the installers have diffent default based on how the vm was booted | |
| 19:46:31 | wncslln | the image with problem is a snapshot from a vm annndddd probably the old image does not have uefi | |
| 19:50:13 | opendevreview | Merged openstack/nova-specs master: List requested availability zones https://review.opendev.org/c/openstack/nova-specs/+/904368 | |
| 19:54:43 | sean-k-mooney | melwitt: https://paste.opendev.org/show/biqXsY6SO6ePMPosdwIW/ | |
| 20:00:32 | melwitt | sean-k-mooney: that's good right? haha | |
| 20:00:58 | sean-k-mooney | yep https://termbin.com/9tk3 | |
| 20:01:15 | sean-k-mooney | well sort of | |
| 20:01:35 | sean-k-mooney | i copied that config form your raw job | |
| 20:01:48 | sean-k-mooney | so its is using disk encyption | |
| 20:01:54 | sean-k-mooney | and i can log into the vm via the console | |
| 20:02:00 | sean-k-mooney | so it booted fine | |
| 20:02:18 | sean-k-mooney | however its still booting form a qcow with a raw backing file | |
| 20:02:26 | sean-k-mooney | so its not actully checking raw images properly | |
| 20:02:56 | sean-k-mooney | but ya it looks like everything is working correctly other then that. and i booted with both swap and ephmeral | |
| 20:03:28 | sean-k-mooney | im going to leave it there for tonight just wanted to let you knwo i ahve successfuly deployed our code and booted a vm with it | |
| 20:03:31 | melwitt | sean-k-mooney: you have set both use_cow_images = False and images_type = raw? bc if you don't set both it won't do raw | |
| 20:04:06 | sean-k-mooney | images_type=raw or flat | |
| 20:04:32 | melwitt | ok, thanks :) | |
| 20:05:39 | sean-k-mooney | actully im goign to check something | |
| 20:06:00 | melwitt | ok. at least for me if use_cow_images is not set to False, it will do qcow2 instead of raw despite images_type = raw or flat | |
| 20:06:18 | sean-k-mooney | i started with your raw job | |
| 20:06:32 | sean-k-mooney | but i might not have kept that in the local.conf when i finished editing it | |
| 20:07:07 | sean-k-mooney | ok ya | |
| 20:07:12 | sean-k-mooney | i removed that in my editing | |
| 20:07:22 | melwitt | ok. that's the only way I know of that what you're seeing could be happening. if you do have use_cow_images = False then something else is wrong that I haven't seen yet | |
| 20:07:26 | sean-k-mooney | so everything is working as expecte its using qcow for the vms because that is the default | |
| 20:07:54 | sean-k-mooney | no in my local.conf i don thave those set so we are getting the default | |
| 20:08:32 | melwitt | ok I see | |
| 20:09:25 | sean-k-mooney | so cool i wont have time to do much with this tomorrow but ill play around with it on monday and start reivewing the series | |
| 20:09:34 | sean-k-mooney | have you done any multi node testing. | |
| 20:09:38 | sean-k-mooney | i.e. cold/live migration | |
| 20:10:33 | sean-k-mooney | the current zuul jobs seam to be single node | |
| 20:11:39 | sean-k-mooney | on a related note i ported your scater gather patch on to my serise and i got a pretty clean zuul run https://review.opendev.org/c/openstack/nova/+/905285 | |
| 20:12:14 | sean-k-mooney | the main delta between what i have doen is how we monkeypatch | |
| 20:12:43 | melwitt | sean-k-mooney: hm looks like no. I did resize in place. I'll add a multinode job on that patch | |
| 20:13:09 | sean-k-mooney | ok cool ill add a second node and play with that locally too next week | |
| 20:14:38 | melwitt | sean-k-mooney: ok cool. I dunno if you saw I wrote up a comment on the scatter gather patch about how I am seeing requests getting "stuck" on nova-api, like it receives the request and does all the stuff but does not respond. I have seen this in other failures that are not related with my patch too but with the patch it seems to happen much more often | |
| 20:14:58 | sean-k-mooney | ya i did | |
| 20:15:13 | melwitt | oh ok | |
| 20:15:16 | sean-k-mooney | i wanted to see if i saw the same behvior | |
| 20:15:33 | sean-k-mooney | i was wondering if htat coudl be related to how/when monkey patching was done | |
| 20:15:39 | melwitt | ok yeah | |
| 20:15:58 | sean-k-mooney | but there is also this comment on my patch https://review.opendev.org/c/openstack/nova/+/905285/1/nova/compute/multi_cell_list.py | |
| 20:16:02 | sean-k-mooney | that might be related too | |
| 20:16:10 | sean-k-mooney | the timeout might need to be done in the try? | |
| 20:16:28 | sean-k-mooney | or rather the creation or the timer | |
| 20:16:32 | melwitt | that's good. really want to find what/why that's happening | |
| 20:16:42 | melwitt | yeah I was thinking about that too | |
| 20:17:14 | sean-k-mooney | im not familar enough with timers in python to say one way or another but i dont think it would hurt to include it in the try block | |
| 20:18:28 | sean-k-mooney | melwitt: another data point on the previosu patch that uses the standar lib queue https://review.opendev.org/c/openstack/nova/+/900681/8?tab=change-view-tab-header-zuul-results-summary i got a bunch of timeourts | |
| 20:18:35 | sean-k-mooney | but i have not had time to look at way | |
| 20:18:39 | sean-k-mooney | *why | |
| 20:18:49 | melwitt | yeah, agree. it would be safer to move it into the try, I was going to do that anyway. I think I followed this example initially https://eventlet.net/doc/modules/timeout.html#eventlet.timeout.Timeout | |
| 20:19:26 | melwitt | whoa that is a lot of timeouts indeed | |
| 20:21:56 | melwitt | sean-k-mooney: I wasn't sure how to interpret this when I looked at it too, does this mean the eventlet queue does something special to work with greenlets? "The eventlet.queue module implements multi-producer, multi-consumer queues that work across greenlets, with the API similar to the classes found in the standard Queue and multiprocessing modules." https://eventlet.net/doc/modules/queue.html | |
| 20:22:23 | melwitt | if so, that could be why | |
| 20:22:43 | sean-k-mooney | ya so you have done the queue chagne in the same patch that swap to futureist | |
| 20:22:49 | sean-k-mooney | i have them split out | |
| 20:22:55 | melwitt | right, I know :) | |
| 20:23:06 | sean-k-mooney | so it could be related | |
| 20:23:20 | melwitt | I'm saying if eventlet.queue is "special" and you're not using it, maybe queue won't work right. oh, but eventlet should be patching queue right? hm | |
| 20:23:43 | sean-k-mooney | ya i was assuming eventlet was patching it | |
| 20:23:45 | melwitt | if it's patching it I dunno why it wouldn't work 🤔 | |
| 20:23:45 | melwitt | if it's patching it I dunno why it wouldn't work 🤔 | |
| 20:23:48 | melwitt | yeah | |
| 20:24:12 | sean-k-mooney | ok i need to get dinner and feed frya. she is getting barky | |
| 20:24:16 | sean-k-mooney | caht to you tomrorow | |
| 20:24:23 | JayF | some eventlet queue usages have known bugs | |
| 20:24:33 | melwitt | haha k, talk to you tomorrow | |
| 20:24:35 | JayF | Let me read scrollback and see if you are in those categories, if so I can likely find the bugs | |
| 20:24:48 | JayF | itamarst has been digging them up left and right as part of the "fix eventlet" work | |
| 20:25:02 | JayF | He leaves me frightening issue links in my slack every now and then :D | |
| 20:26:37 | melwitt | JayF: it's the opposite, what we were chatting about. sean did an in-place swap from eventlet.queue to standard lib queue and got a lot of timed out jobs. which might not be related to the switch https://review.opendev.org/c/openstack/nova/+/900681/8?tab=change-view-tab-header-zuul-results-summary | |
| 20:26:48 | JayF | Yeah this tracks | |
| 20:26:55 | JayF | because I think there's an RLock used by stdlib queue | |
| 20:27:00 | JayF | which is still getting monkey patched in this case | |
| 20:27:01 | JayF | let me dig | |
| 20:27:28 | JayF | so my hypothesis is this: | |
| 20:27:45 | melwitt | earlier runs looked fine and I dunno if I see any difference in the code so I'm not sure | |
| 20:28:02 | JayF | by using a stdlib queue, you're exposing yourself to using threading.Nlock (N can be multiple things). Some of these, particularly Rlock, is bugged in inconsistent ways (I am still looking for the github issue) | |
| 20:28:31 | melwitt | I didn't look in detail yet | |
| 20:31:52 | melwitt | hm ok | |
| 20:32:26 | sean-k-mooney | hum ok not back for long what i might do instead is this | |
| 20:32:59 | sean-k-mooney | add a inderiction where we import a diffent queue based on if we are patched or not and cenntralise that in the utils module | |
| 20:33:27 | sean-k-mooney | so that we can drop the selection of which to use in one place if we wever fully remove eventlet | |
| 20:34:08 | melwitt | ack | |
| 20:34:09 | sean-k-mooney | that way if the code in question is not monkypatche it will use the std lib one and where it is it will use the eventlet one | |
| 20:34:44 | JayF | sean-k-mooney: is "monkeypatched" a boolean for you? You're not playing games with monkeypatching on/off for different modules, are you? | |
| 20:35:24 | sean-k-mooney | its gloabl | |
| 20:35:27 | JayF | sean-k-mooney: that I think is going to be an attractive pattern that may lead to more pain than it's worth, based on my understanding many of the eventlet bugs in python3.11/3.12 which are just ... unfixable by design are mostly at the edges of stdlib interacting with monkey patched bits of code | |
| 20:35:39 | JayF | OK good, hopefully that means you'll avoid the sharpest edges | |
| 20:35:41 | sean-k-mooney | so in my case it depens on what direcotry the console cript is defiend in | |
| 20:35:50 | sean-k-mooney | if its under nova.cmd its monky patched | |
| 20:35:56 | sean-k-mooney | if its under nova.bin its not | |
| 20:36:13 | JayF | but there's never a case, as conjectured above, where say, queue would be std lib but threading would be eventlet? | |