| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-07-08 | |||
| 12:29:20 | bauzas | but I don't want to clockdown if I'm plugged | |
| 12:30:58 | opendevreview | Merged openstack/nova master: Retry attachment delete API call for 504 Gateway Timeout https://review.opendev.org/c/openstack/nova/+/845543 | |
| 12:33:15 | sean-k-mooney | bauzas: i use tlp | |
| 12:33:33 | sean-k-mooney | i dont actully use power save on battery but i configure it to limit the max frequency | |
| 12:33:55 | sean-k-mooney | but you can certenly set the profile to power save too if you prefer | |
| 12:34:23 | sean-k-mooney | https://paste.opendev.org/show/bPVClbF8MnuFjrhBE2z3/ | |
| 12:34:27 | sean-k-mooney | that is my tlp config | |
| 12:34:54 | sean-k-mooney | bauzas: so you could change line 6 two powersaver | |
| 12:35:03 | sean-k-mooney | or liek i do on line 11 | |
| 12:35:16 | sean-k-mooney | limite the cpu frequency | |
| 12:36:17 | sean-k-mooney | bauzas: tlp also has a tlp-stats tool which is kind of nice | |
| 12:37:22 | sean-k-mooney | you can see that i still have 90% capasity on my battery after 4 years | |
| 12:37:25 | sean-k-mooney | https://paste.opendev.org/show/bNtQTQ0T2Tx4Bd2nFyLu/ | |
| 12:37:37 | sean-k-mooney | partly becasue i only let it chagne to 80% | |
| 12:38:39 | sean-k-mooney | so i went with tlp since i coudl use it to do more then save power it can also extend battry life by contoling change and i cloud have a little more contol over all | |
| 12:39:37 | sean-k-mooney | https://termbin.com/ic06r is the full set of stats it has | |
| 12:39:47 | sean-k-mooney | and you can twek many of the paramters if you care about it | |
| 12:49:55 | opendevreview | Amit Uniyal proposed openstack/nova master: Adds check, if admin has set compute service down https://review.opendev.org/c/openstack/nova/+/848886 | |
| 12:49:56 | opendevreview | Amit Uniyal proposed openstack/nova master: add regression test case for bug 1978983 https://review.opendev.org/c/openstack/nova/+/849104 | |
| 12:50:58 | bauzas | ok, eventually fixed my issue, upgraded the firware | |
| 12:51:02 | gibi | sean-k-mooney: I'm wondering that we reintroduced a an issue by reenabling greendns. See my fresh bugreport https://bugs.launchpad.net/nova/+bug/1981080 | |
| 12:51:15 | bauzas | when the lid is closed, you can't upgrade your firmware, bizarre but ok | |
| 12:51:28 | bauzas | so this one was PEBKAC | |
| 12:51:57 | bauzas | I'll need to use to change the profile based on the docking info | |
| 12:52:04 | bauzas | need to see* | |
| 13:11:00 | sean-k-mooney | gibi: i was still getting that waring before that | |
| 13:11:20 | gibi | hm, I don't get it if I turn off greendns | |
| 13:12:00 | sean-k-mooney | we coudl perhaps disable greendns in our func tests but i dont think a revert would be correct | |
| 13:12:28 | sean-k-mooney | we shoudl not be doing dns lookup in the func tests | |
| 13:13:36 | gibi | this is not about func test, there it is probably cosmetic. But I wonder if we reintroduced https://bugs.launchpad.net/nova/+bug/1808951 | |
| 13:13:57 | gibi | as the fix for that was to blacklist urllib3 | |
| 13:14:49 | gibi | now we see that urllib3 is imported before monkey patching | |
| 13:14:56 | gibi | so we might retrigger https://bugs.launchpad.net/nova/+bug/1808951 | |
| 13:15:08 | sean-k-mooney | well we now use eventliet directly in the nova-api | |
| 13:15:25 | sean-k-mooney | for scater gather so we cant skip monky patching in general there anymore | |
| 13:16:03 | gibi | I don't follow you how that is related | |
| 13:16:21 | sean-k-mooney | we know that disabling greendns can lead to the nova-api and other serivce had locking up on dns quieres | |
| 13:16:33 | sean-k-mooney | so if we revert the other patch we woudl break things | |
| 13:17:06 | gibi | OK, so if we disable greendns we break thing, but now that we enabled it we might break other things | |
| 13:17:29 | sean-k-mooney | we might but i dont think we will | |
| 13:17:58 | sean-k-mooney | have you found a code path where urllib3 is imported before monkeypatching? | |
| 13:18:10 | gibi | yes, it is the dns code in eventlet that imports it | |
| 13:18:41 | sean-k-mooney | so https://github.com/openstack/nova/blob/90c0c687a487601e009c72f60c88be92f6a55264/nova/monkey_patch.py#L30= | |
| 13:18:47 | sean-k-mooney | imports urllib3? | |
| 13:18:54 | gibi | indireclt yes | |
| 13:19:00 | gibi | indirectly | |
| 13:19:06 | sean-k-mooney | that sounds like an eventlet bug that we canot fix then | |
| 13:19:22 | sean-k-mooney | since we cant monkeypatch until after we import eventlet | |
| 13:20:27 | sean-k-mooney | so ya maybe we need to remove urllib3 form the problem list if eventlet have fixed the issue | |
| 13:20:34 | sean-k-mooney | if not then we shoudl file an eventlet bug | |
| 13:20:50 | sean-k-mooney | unless from nova import debugger | |
| 13:20:56 | sean-k-mooney | is pulling it in instead fo eventlet | |
| 13:21:20 | sean-k-mooney | not that i can see | |
| 13:21:24 | sean-k-mooney | https://github.com/openstack/nova/blob/90c0c687a487601e009c72f60c88be92f6a55264/nova/debugger.py | |
| 13:25:34 | gibi | hm I assuming that deployers had some workaround for our disable greendns as we had that disable for a long time | |
| 13:27:05 | gibi | so we might fixed a known and worked around problem with but re-introduced a break somewhere else. Still I feel like reverting the greendns patch would be safer while we figure out is urllib3 is safe | |
| 13:29:21 | gibi | anyhow updated the bug report linking to this discussion as I have now bandwidth to dig deeper into this now | |
| 13:32:09 | Uggla | gibi, sorry for that.... Uggla is a black cat that attract bug. | |
| 13:33:15 | gibi | Uggla: no worries. I like these kind of challenges :) | |
| 14:13:38 | sean-k-mooney | gibi: it was breaking our downstream customers | |
| 14:13:51 | sean-k-mooney | so we cant assume that | |
| 14:14:10 | sean-k-mooney | gibi: when i did the revert an renebaled greendns | |
| 14:14:28 | sean-k-mooney | i noted you could sitll diablie it by setting the env var | |
| 14:14:57 | sean-k-mooney | gibi: we were hardcodeing os.environ['EVENTLET_NO_GREENDNS'] = 'yes' | |
| 14:14:59 | sean-k-mooney | before | |
| 14:15:12 | sean-k-mooney | but you can just export that before you run nova-api | |
| 14:15:21 | sean-k-mooney | or set it in the wsgi env if you need too | |
| 14:15:25 | sean-k-mooney | but i dont think you should | |
| 14:18:59 | gibi | sean-k-mooney: so if somebody get hit by https://bugs.launchpad.net/nova/+bug/1808951 then they can set EVENTLET_NO_GREENDNS=yes in there api service script to get back the old behavior | |
| 14:19:14 | gibi | good point. I agree that is a viable workaround | |
| 14:19:20 | gibi | I will not that in https://bugs.launchpad.net/nova/+bug/1808951 | |
| 14:19:36 | sean-k-mooney | i just did in https://bugs.launchpad.net/nova/+bug/1981080/comments/2 | |
| 14:20:06 | sean-k-mooney | but yes you could not that in the orginal bug | |
| 14:20:39 | sean-k-mooney | gibi: do you happen to know what the orginal urllib3 bug is and if it was fixed | |
| 14:20:51 | sean-k-mooney | i have read over https://bugs.launchpad.net/nova/+bug/1808951 | |
| 14:21:08 | sean-k-mooney | and that references https://github.com/eventlet/eventlet/issues/371 | |
| 14:21:35 | opendevreview | Sergii Golovatiuk proposed openstack/nova master: Replace "db archive" with "db archive_deleted_raws" https://review.opendev.org/c/openstack/nova/+/847963 | |
| 14:21:42 | sean-k-mooney | that seams to have been fiex and broken | |
| 14:21:49 | sean-k-mooney | based on differnt python releases | |
| 14:22:23 | gibi | I think it is https://github.com/eventlet/eventlet/issues/371 as you noted | |
| 14:22:45 | gibi | at least https://github.com/eventlet/eventlet/issues/371#issuecomment-1047336652 reports that it still exists | |
| 14:22:52 | sean-k-mooney | https://github.com/eventlet/eventlet/issues/726 | |
| 14:23:05 | sean-k-mooney | so they have fixed some version fo this | |
| 14:23:21 | sean-k-mooney | gibi: did you actully se this cause issues | |
| 14:23:29 | sean-k-mooney | or did you just notice the warning | |
| 14:23:43 | sean-k-mooney | we use uwsgi in the devstack jobs | |
| 14:23:47 | sean-k-mooney | and i have not seen them fail | |
| 14:24:04 | sean-k-mooney | so just trying ot undersand if there is an actul functional issue | |
| 14:24:09 | sean-k-mooney | or just a warnign message in our test | |
| 14:24:33 | gibi | I only noticed the warning | |
| 14:24:50 | gibi | I did not see actuall failures | |
| 14:25:18 | sean-k-mooney | ok then i think we should just remove the warnign unless someoen reports an error | |
| 14:25:26 | sean-k-mooney | since it appears to work properly | |
| 14:26:00 | sean-k-mooney | given it has been runnign upstream and in production for down stream custoemr for a whiel now | |
| 14:26:35 | sean-k-mooney | hum actully | |
| 14:26:49 | opendevreview | Amit Uniyal proposed openstack/nova master: Adds check, if admin has set compute service down https://review.opendev.org/c/openstack/nova/+/848886 | |
| 14:27:43 | sean-k-mooney | gibi: ok we hae no shipped it downstream yet | |
| 14:27:49 | sean-k-mooney | my backport is still pending | |
| 14:27:59 | sean-k-mooney | it has been runin in the gate though for a hile | |
| 14:28:04 | sean-k-mooney | *while | |