| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-07-08 | |||
| 13:11:20 | gibi | hm, I don't get it if I turn off greendns | |
| 13:12:00 | sean-k-mooney | we coudl perhaps disable greendns in our func tests but i dont think a revert would be correct | |
| 13:12:28 | sean-k-mooney | we shoudl not be doing dns lookup in the func tests | |
| 13:13:36 | gibi | this is not about func test, there it is probably cosmetic. But I wonder if we reintroduced https://bugs.launchpad.net/nova/+bug/1808951 | |
| 13:13:57 | gibi | as the fix for that was to blacklist urllib3 | |
| 13:14:49 | gibi | now we see that urllib3 is imported before monkey patching | |
| 13:14:56 | gibi | so we might retrigger https://bugs.launchpad.net/nova/+bug/1808951 | |
| 13:15:08 | sean-k-mooney | well we now use eventliet directly in the nova-api | |
| 13:15:25 | sean-k-mooney | for scater gather so we cant skip monky patching in general there anymore | |
| 13:16:03 | gibi | I don't follow you how that is related | |
| 13:16:21 | sean-k-mooney | we know that disabling greendns can lead to the nova-api and other serivce had locking up on dns quieres | |
| 13:16:33 | sean-k-mooney | so if we revert the other patch we woudl break things | |
| 13:17:06 | gibi | OK, so if we disable greendns we break thing, but now that we enabled it we might break other things | |
| 13:17:29 | sean-k-mooney | we might but i dont think we will | |
| 13:17:58 | sean-k-mooney | have you found a code path where urllib3 is imported before monkeypatching? | |
| 13:18:10 | gibi | yes, it is the dns code in eventlet that imports it | |
| 13:18:41 | sean-k-mooney | so https://github.com/openstack/nova/blob/90c0c687a487601e009c72f60c88be92f6a55264/nova/monkey_patch.py#L30= | |
| 13:18:47 | sean-k-mooney | imports urllib3? | |
| 13:18:54 | gibi | indireclt yes | |
| 13:19:00 | gibi | indirectly | |
| 13:19:06 | sean-k-mooney | that sounds like an eventlet bug that we canot fix then | |
| 13:19:22 | sean-k-mooney | since we cant monkeypatch until after we import eventlet | |
| 13:20:27 | sean-k-mooney | so ya maybe we need to remove urllib3 form the problem list if eventlet have fixed the issue | |
| 13:20:34 | sean-k-mooney | if not then we shoudl file an eventlet bug | |
| 13:20:50 | sean-k-mooney | unless from nova import debugger | |
| 13:20:56 | sean-k-mooney | is pulling it in instead fo eventlet | |
| 13:21:20 | sean-k-mooney | not that i can see | |
| 13:21:24 | sean-k-mooney | https://github.com/openstack/nova/blob/90c0c687a487601e009c72f60c88be92f6a55264/nova/debugger.py | |
| 13:25:34 | gibi | hm I assuming that deployers had some workaround for our disable greendns as we had that disable for a long time | |
| 13:27:05 | gibi | so we might fixed a known and worked around problem with but re-introduced a break somewhere else. Still I feel like reverting the greendns patch would be safer while we figure out is urllib3 is safe | |
| 13:29:21 | gibi | anyhow updated the bug report linking to this discussion as I have now bandwidth to dig deeper into this now | |
| 13:32:09 | Uggla | gibi, sorry for that.... Uggla is a black cat that attract bug. | |
| 13:33:15 | gibi | Uggla: no worries. I like these kind of challenges :) | |
| 14:13:38 | sean-k-mooney | gibi: it was breaking our downstream customers | |
| 14:13:51 | sean-k-mooney | so we cant assume that | |
| 14:14:10 | sean-k-mooney | gibi: when i did the revert an renebaled greendns | |
| 14:14:28 | sean-k-mooney | i noted you could sitll diablie it by setting the env var | |
| 14:14:57 | sean-k-mooney | gibi: we were hardcodeing os.environ['EVENTLET_NO_GREENDNS'] = 'yes' | |
| 14:14:59 | sean-k-mooney | before | |
| 14:15:12 | sean-k-mooney | but you can just export that before you run nova-api | |
| 14:15:21 | sean-k-mooney | or set it in the wsgi env if you need too | |
| 14:15:25 | sean-k-mooney | but i dont think you should | |
| 14:18:59 | gibi | sean-k-mooney: so if somebody get hit by https://bugs.launchpad.net/nova/+bug/1808951 then they can set EVENTLET_NO_GREENDNS=yes in there api service script to get back the old behavior | |
| 14:19:14 | gibi | good point. I agree that is a viable workaround | |
| 14:19:20 | gibi | I will not that in https://bugs.launchpad.net/nova/+bug/1808951 | |
| 14:19:36 | sean-k-mooney | i just did in https://bugs.launchpad.net/nova/+bug/1981080/comments/2 | |
| 14:20:06 | sean-k-mooney | but yes you could not that in the orginal bug | |
| 14:20:39 | sean-k-mooney | gibi: do you happen to know what the orginal urllib3 bug is and if it was fixed | |
| 14:20:51 | sean-k-mooney | i have read over https://bugs.launchpad.net/nova/+bug/1808951 | |
| 14:21:08 | sean-k-mooney | and that references https://github.com/eventlet/eventlet/issues/371 | |
| 14:21:35 | opendevreview | Sergii Golovatiuk proposed openstack/nova master: Replace "db archive" with "db archive_deleted_raws" https://review.opendev.org/c/openstack/nova/+/847963 | |
| 14:21:42 | sean-k-mooney | that seams to have been fiex and broken | |
| 14:21:49 | sean-k-mooney | based on differnt python releases | |
| 14:22:23 | gibi | I think it is https://github.com/eventlet/eventlet/issues/371 as you noted | |
| 14:22:45 | gibi | at least https://github.com/eventlet/eventlet/issues/371#issuecomment-1047336652 reports that it still exists | |
| 14:22:52 | sean-k-mooney | https://github.com/eventlet/eventlet/issues/726 | |
| 14:23:05 | sean-k-mooney | so they have fixed some version fo this | |
| 14:23:21 | sean-k-mooney | gibi: did you actully se this cause issues | |
| 14:23:29 | sean-k-mooney | or did you just notice the warning | |
| 14:23:43 | sean-k-mooney | we use uwsgi in the devstack jobs | |
| 14:23:47 | sean-k-mooney | and i have not seen them fail | |
| 14:24:04 | sean-k-mooney | so just trying ot undersand if there is an actul functional issue | |
| 14:24:09 | sean-k-mooney | or just a warnign message in our test | |
| 14:24:33 | gibi | I only noticed the warning | |
| 14:24:50 | gibi | I did not see actuall failures | |
| 14:25:18 | sean-k-mooney | ok then i think we should just remove the warnign unless someoen reports an error | |
| 14:25:26 | sean-k-mooney | since it appears to work properly | |
| 14:26:00 | sean-k-mooney | given it has been runnign upstream and in production for down stream custoemr for a whiel now | |
| 14:26:35 | sean-k-mooney | hum actully | |
| 14:26:49 | opendevreview | Amit Uniyal proposed openstack/nova master: Adds check, if admin has set compute service down https://review.opendev.org/c/openstack/nova/+/848886 | |
| 14:27:43 | sean-k-mooney | gibi: ok we hae no shipped it downstream yet | |
| 14:27:49 | sean-k-mooney | my backport is still pending | |
| 14:27:59 | sean-k-mooney | it has been runin in the gate though for a hile | |
| 14:28:04 | sean-k-mooney | *while | |
| 14:30:28 | gibi | at least we are not setting EVENTLET_NO_GREENDNS so we should see it failing in the gate | |
| 14:30:42 | gibi | *not setting in devstack | |
| 14:30:52 | sean-k-mooney | yep | |
| 14:31:00 | sean-k-mooney | and since we dont i think it still correct | |
| 14:33:56 | gibi | do we test with ssl in the gate? | |
| 14:38:26 | sean-k-mooney | we use the tls_proxy | |
| 14:38:59 | sean-k-mooney | so we use apache to do tls termination using a prviate ca in front of uwsgi | |
| 14:39:34 | sean-k-mooney | i dont think uwsgi support tls directly or its a pain to configure which is why we do it that way | |
| 14:40:54 | gibi | then we probably don't trigger the ssl code that caused the original problem | |
| 14:41:35 | sean-k-mooney | maybe not sure if the api calls other service we would open an ssh clonenction | |
| 14:41:57 | sean-k-mooney | so i would expect the keystone middelware to trigger it no? | |
| 14:42:20 | sean-k-mooney | i dont thinkthe recursion was on the server side | |
| 14:43:17 | gibi | hm, yeah it was in keystonemiddleware | |
| 14:52:54 | Airamek | Hi! I know this isn't the place to ask for support, but #openstack is kinda dead. If anyone has a spare minute, could you look into this?(https://lists.openstack.org/pipermail/openstack-discuss/2022-July/029454.html). Thanks in advance! | |
| 14:59:10 | opendevreview | Balazs Gibizer proposed openstack/nova master: Add extra info about limitation of CellDatabase fixture https://review.opendev.org/c/openstack/nova/+/849123 | |
| 14:59:48 | gibi | Uggla: ^^ I added extra info the the lock escalation error so that next time we can track down the cause easier | |
| 15:07:19 | opendevreview | sean mooney proposed openstack/nova master: Adds check, if admin has set compute service down https://review.opendev.org/c/openstack/nova/+/848886 | |
| 15:07:19 | opendevreview | sean mooney proposed openstack/nova master: add regression test case for bug 1978983 https://review.opendev.org/c/openstack/nova/+/849104 | |
| 15:53:43 | gibi | I'm done for the week. Have a nice weekend folks! | |
| 16:06:08 | opendevreview | Alexey Stupnikov proposed openstack/nova master: Remove deleted projects from flavor access list https://review.opendev.org/c/openstack/nova/+/849131 | |
| 16:17:01 | opendevreview | Rajesh Tailor proposed openstack/placement master: Fix typos https://review.opendev.org/c/openstack/placement/+/848636 | |
| 17:03:44 | opendevreview | Sylvain Bauza proposed openstack/nova master: WIP: api: Drop generating a keypair and add special chars to naming https://review.opendev.org/c/openstack/nova/+/849133 | |
| #openstack-nova - 2022-07-09 | |||
| 05:37:54 | opendevreview | Rajesh Tailor proposed openstack/nova master: Remove handling of 'u' prefix https://review.opendev.org/c/openstack/nova/+/849177 | |
| 15:39:22 | opendevreview | Takashi Kajinami proposed openstack/nova stable/yoga: Retry attachment delete API call for 504 Gateway Timeout https://review.opendev.org/c/openstack/nova/+/849212 | |
| 15:43:35 | opendevreview | Takashi Kajinami proposed openstack/nova master: libvirt: Add new option to enforce multipath volume connections https://review.opendev.org/c/openstack/nova/+/845660 | |
| 16:21:04 | opendevreview | Takashi Kajinami proposed openstack/nova master: libvirt: Add new option to enforce multipath volume connections https://review.opendev.org/c/openstack/nova/+/845660 | |