| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-01-11 | |||
| 20:16:42 | melwitt | yeah I was thinking about that too | |
| 20:17:14 | sean-k-mooney | im not familar enough with timers in python to say one way or another but i dont think it would hurt to include it in the try block | |
| 20:18:28 | sean-k-mooney | melwitt: another data point on the previosu patch that uses the standar lib queue https://review.opendev.org/c/openstack/nova/+/900681/8?tab=change-view-tab-header-zuul-results-summary i got a bunch of timeourts | |
| 20:18:35 | sean-k-mooney | but i have not had time to look at way | |
| 20:18:39 | sean-k-mooney | *why | |
| 20:18:49 | melwitt | yeah, agree. it would be safer to move it into the try, I was going to do that anyway. I think I followed this example initially https://eventlet.net/doc/modules/timeout.html#eventlet.timeout.Timeout | |
| 20:19:26 | melwitt | whoa that is a lot of timeouts indeed | |
| 20:21:56 | melwitt | sean-k-mooney: I wasn't sure how to interpret this when I looked at it too, does this mean the eventlet queue does something special to work with greenlets? "The eventlet.queue module implements multi-producer, multi-consumer queues that work across greenlets, with the API similar to the classes found in the standard Queue and multiprocessing modules." https://eventlet.net/doc/modules/queue.html | |
| 20:22:23 | melwitt | if so, that could be why | |
| 20:22:43 | sean-k-mooney | ya so you have done the queue chagne in the same patch that swap to futureist | |
| 20:22:49 | sean-k-mooney | i have them split out | |
| 20:22:55 | melwitt | right, I know :) | |
| 20:23:06 | sean-k-mooney | so it could be related | |
| 20:23:20 | melwitt | I'm saying if eventlet.queue is "special" and you're not using it, maybe queue won't work right. oh, but eventlet should be patching queue right? hm | |
| 20:23:43 | sean-k-mooney | ya i was assuming eventlet was patching it | |
| 20:23:45 | melwitt | if it's patching it I dunno why it wouldn't work 🤔 | |
| 20:23:45 | melwitt | if it's patching it I dunno why it wouldn't work 🤔 | |
| 20:23:48 | melwitt | yeah | |
| 20:24:12 | sean-k-mooney | ok i need to get dinner and feed frya. she is getting barky | |
| 20:24:16 | sean-k-mooney | caht to you tomrorow | |
| 20:24:23 | JayF | some eventlet queue usages have known bugs | |
| 20:24:33 | melwitt | haha k, talk to you tomorrow | |
| 20:24:35 | JayF | Let me read scrollback and see if you are in those categories, if so I can likely find the bugs | |
| 20:24:48 | JayF | itamarst has been digging them up left and right as part of the "fix eventlet" work | |
| 20:25:02 | JayF | He leaves me frightening issue links in my slack every now and then :D | |
| 20:26:37 | melwitt | JayF: it's the opposite, what we were chatting about. sean did an in-place swap from eventlet.queue to standard lib queue and got a lot of timed out jobs. which might not be related to the switch https://review.opendev.org/c/openstack/nova/+/900681/8?tab=change-view-tab-header-zuul-results-summary | |
| 20:26:48 | JayF | Yeah this tracks | |
| 20:26:55 | JayF | because I think there's an RLock used by stdlib queue | |
| 20:27:00 | JayF | which is still getting monkey patched in this case | |
| 20:27:01 | JayF | let me dig | |
| 20:27:28 | JayF | so my hypothesis is this: | |
| 20:27:45 | melwitt | earlier runs looked fine and I dunno if I see any difference in the code so I'm not sure | |
| 20:28:02 | JayF | by using a stdlib queue, you're exposing yourself to using threading.Nlock (N can be multiple things). Some of these, particularly Rlock, is bugged in inconsistent ways (I am still looking for the github issue) | |
| 20:28:31 | melwitt | I didn't look in detail yet | |
| 20:31:52 | melwitt | hm ok | |
| 20:32:26 | sean-k-mooney | hum ok not back for long what i might do instead is this | |
| 20:32:59 | sean-k-mooney | add a inderiction where we import a diffent queue based on if we are patched or not and cenntralise that in the utils module | |
| 20:33:27 | sean-k-mooney | so that we can drop the selection of which to use in one place if we wever fully remove eventlet | |
| 20:34:08 | melwitt | ack | |
| 20:34:09 | sean-k-mooney | that way if the code in question is not monkypatche it will use the std lib one and where it is it will use the eventlet one | |
| 20:34:44 | JayF | sean-k-mooney: is "monkeypatched" a boolean for you? You're not playing games with monkeypatching on/off for different modules, are you? | |
| 20:35:24 | sean-k-mooney | its gloabl | |
| 20:35:27 | JayF | sean-k-mooney: that I think is going to be an attractive pattern that may lead to more pain than it's worth, based on my understanding many of the eventlet bugs in python3.11/3.12 which are just ... unfixable by design are mostly at the edges of stdlib interacting with monkey patched bits of code | |
| 20:35:39 | JayF | OK good, hopefully that means you'll avoid the sharpest edges | |
| 20:35:41 | sean-k-mooney | so in my case it depens on what direcotry the console cript is defiend in | |
| 20:35:50 | sean-k-mooney | if its under nova.cmd its monky patched | |
| 20:35:56 | sean-k-mooney | if its under nova.bin its not | |
| 20:36:13 | JayF | but there's never a case, as conjectured above, where say, queue would be std lib but threading would be eventlet? | |
| 20:36:33 | sean-k-mooney | not unless i screwed up | |
| 20:36:51 | sean-k-mooney | so the nova api in my serise can be patch or unpatched | |
| 20:36:59 | sean-k-mooney | if run via the wsgi script its unpatched | |
| 20:37:11 | sean-k-mooney | if run via nova.cmd.nova-api it is | |
| 20:37:11 | JayF | okay cool, in that case my hypothesis is invalid and unlikely to be happening | |
| 20:37:20 | JayF | and I'd suggest potentially even enforcing that if possible | |
| 20:37:25 | sean-k-mooney | but within any binary its one of the otehr | |
| 20:38:07 | JayF | Do you mind a quick topic change? | |
| 20:38:21 | sean-k-mooney | sure but i need to leave in a few | |
| 20:38:25 | sean-k-mooney | whats on your mind | |
| 20:38:27 | JayF | Just curious for ironic-guest-metadata, would you mind doing the piece of that work of splitting out the libvirt code to the common compute driver? | |
| 20:38:44 | JayF | I'd prefer not operate in a portion of the nova codebase outside of the ironic driver codepath, just based on familiarity | |
| 20:39:00 | JayF | but I can if you are too busy helping save the world from eventlet (a project which I too am invested in) | |
| 20:39:18 | sean-k-mooney | hum i might be able to ill have to see if i have time but should nbot be terrible complex | |
| 20:39:42 | sean-k-mooney | ask me again next week and ill see if i can help | |
| 20:40:16 | sean-k-mooney | i need to make some progress on the healtcheck stuff but im hoping to spend most of the next 4-5 weeks until codefreeze mainaly on upstream nova stuff | |
| 20:41:19 | sean-k-mooney | JayF: the eventlet stuff is just somethign im looking at in my personal time | |
| 20:41:58 | sean-k-mooney | so im not goign to be spending that much time looking at it untill we have some more detailed disucssions in the next ptg | |
| 20:42:34 | sean-k-mooney | i know others have concerse over if moving to asyncio or realtreading is really the right move | |
| 20:48:20 | JayF | Honestly, I joke sometimes that I'm a sysadmin who pretends to be a developer, but what that means in reality is for stuff like "should we use asyncio or $other?" type discussions, I tend to let folks with stronger opinions and foundational knowledge battle it out :) | |
| 20:48:40 | JayF | my concern is that we use something well maintained, with reasonable patterns that anyone who works on openstack for the next decade will be able to follow | |
| 20:49:39 | JayF | so like, I have zero opinions about how we move; I have lots of opinions that moving is important, and doing it well is important (and as I've mentioned elsewhere, I've gotten a part time developer assigned to OpenStack upstream at GR-OSS, he is onboarding now and will be pointed at eventlet migrations for underresourced openstack projects once we have a migration path set) | |
| 22:22:54 | opendevreview | melanie witt proposed openstack/nova master: DNM test ephemeral encryption + resize: qcow2, raw, rbd https://review.opendev.org/c/openstack/nova/+/862416 | |
| 23:04:10 | opendevreview | melanie witt proposed openstack/nova master: DNM test ephemeral encryption + resize: qcow2, raw, rbd https://review.opendev.org/c/openstack/nova/+/862416 | |
| #openstack-nova - 2024-01-12 | |||
| 03:28:51 | melwitt | sean-k-mooney: a heads up that live migration and cold migrate are broken ... I'm working on it | |
| 04:39:44 | opendevreview | sean mooney proposed openstack/nova master: [WIP] factor out driver instance metadata https://review.opendev.org/c/openstack/nova/+/905406 | |
| 04:46:42 | opendevreview | Amit Uniyal proposed openstack/nova master: Refactor vf profile for PCI device https://review.opendev.org/c/openstack/nova/+/905134 | |
| 04:54:09 | opendevreview | sean mooney proposed openstack/nova master: [WIP] factor out driver instance metadata https://review.opendev.org/c/openstack/nova/+/905406 | |
| 08:48:13 | opendevreview | Takashi Kajinami proposed openstack/nova master: Cleanup setup.py https://review.opendev.org/c/openstack/nova/+/905419 | |
| 09:32:46 | opendevreview | Takashi Kajinami proposed openstack/nova master: Remove deprecated [workarounds] enable_numa_live_migration https://review.opendev.org/c/openstack/nova/+/905426 | |
| 09:34:30 | opendevreview | Takashi Kajinami proposed openstack/nova master: Remove deprecated [workarounds] enable_numa_live_migration https://review.opendev.org/c/openstack/nova/+/905426 | |
| 10:46:48 | opendevreview | Sylvain Bauza proposed openstack/nova-specs master: add the blueprint link to the specs https://review.opendev.org/c/openstack/nova-specs/+/905342 | |
| 11:21:32 | sean-k-mooney | JayF: https://review.opendev.org/c/openstack/nova/+/905406 need more unit tests and i shoudl add a follow up to the libvirt dirver to start including the flavor/image properties in the libvirt xml but it should be enouch for you to start working on the ironic part | |
| 12:41:16 | opendevreview | Amit Uniyal proposed openstack/nova stable/2023.2: Refactor vf profile for PCI device https://review.opendev.org/c/openstack/nova/+/905135 | |
| 12:42:45 | opendevreview | Amit Uniyal proposed openstack/nova stable/2023.1: Refactor vf profile for PCI device https://review.opendev.org/c/openstack/nova/+/905137 | |
| 12:43:51 | opendevreview | Amit Uniyal proposed openstack/nova stable/zed: Refactor vf profile for PCI device https://review.opendev.org/c/openstack/nova/+/905138 | |
| 14:06:33 | opendevreview | Amit Uniyal proposed openstack/nova stable/yoga: Refactor vf profile for PCI device https://review.opendev.org/c/openstack/nova/+/905440 | |
| 14:16:42 | opendevreview | Amit Uniyal proposed openstack/nova stable/yoga: Refactor vf profile for PCI device https://review.opendev.org/c/openstack/nova/+/905440 | |
| 14:43:59 | opendevreview | Takashi Kajinami proposed openstack/nova master: Remove deprecated [workarounds] enable_numa_live_migration https://review.opendev.org/c/openstack/nova/+/905426 | |
| 22:44:31 | opendevreview | melanie witt proposed openstack/nova master: libvirt: Introduce support for rbd with LUKS https://review.opendev.org/c/openstack/nova/+/889912 | |
| 22:44:31 | opendevreview | melanie witt proposed openstack/nova master: WIP Suppport migration with ephemeral encryption https://review.opendev.org/c/openstack/nova/+/905512 | |
| #openstack-nova - 2024-01-13 | |||
| 05:26:44 | opendevreview | melanie witt proposed openstack/nova master: libvirt: Configure and teardown ephemeral encryption secrets https://review.opendev.org/c/openstack/nova/+/826754 | |
| 05:26:45 | opendevreview | melanie witt proposed openstack/nova master: imagebackend: Add support to libvirt_info for LUKS based encryption https://review.opendev.org/c/openstack/nova/+/826755 | |
| 05:26:45 | opendevreview | melanie witt proposed openstack/nova master: Add encryption support to convert_image https://review.opendev.org/c/openstack/nova/+/870934 | |
| 05:26:46 | opendevreview | melanie witt proposed openstack/nova master: Support create with ephemeral encryption for qcow2 https://review.opendev.org/c/openstack/nova/+/870932 | |
| 05:26:46 | opendevreview | melanie witt proposed openstack/nova master: Support resize with ephemeral encryption for qcow2 https://review.opendev.org/c/openstack/nova/+/870933 | |
| 05:26:47 | opendevreview | melanie witt proposed openstack/nova master: Add hw_ephemeral_encryption_secret_uuid image property https://review.opendev.org/c/openstack/nova/+/870935 | |
| 05:26:47 | opendevreview | melanie witt proposed openstack/nova master: Add encryption support to qemu-img rebase https://review.opendev.org/c/openstack/nova/+/870936 | |
| 05:26:48 | opendevreview | melanie witt proposed openstack/nova master: Support snapshot with ephemeral encryption for qcow2 https://review.opendev.org/c/openstack/nova/+/870937 | |
| 05:26:49 | opendevreview | melanie witt proposed openstack/nova master: Support rebuild and unshelve with ephemeral encryption https://review.opendev.org/c/openstack/nova/+/870939 | |
| 05:26:49 | opendevreview | melanie witt proposed openstack/nova master: Support rescue with ephemeral encryption https://review.opendev.org/c/openstack/nova/+/873675 | |
| 05:26:51 | opendevreview | melanie witt proposed openstack/nova master: Reject (resize|rebuild) API requests with conflicting encryption https://review.opendev.org/c/openstack/nova/+/904240 | |