| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-07-08 | |||
| 10:32:09 | bauzas | but... lunch | |
| 10:32:11 | sean-k-mooney | yep its one of the ones i tried | |
| 10:32:37 | sean-k-mooney | it didnt work well for me when i use the power-profiles-daemon | |
| 10:33:44 | sean-k-mooney | i think my issue was mainly fan noise and it was not really goign down form max cpu | |
| 10:34:15 | sean-k-mooney | like idling on ac was full non turbo/bost clock speeed | |
| 10:40:15 | opendevreview | Merged openstack/python-novaclient master: Imported Translations from Zanata https://review.opendev.org/c/openstack/python-novaclient/+/848804 | |
| 12:01:55 | bauzas | found the guilty https://fedoraproject.org/wiki/Changes/Power_Profiles_Daemon | |
| 12:02:13 | bauzas | "The "performance" mode is only available on select systems and is implemented by different "drivers" based on the system or systems it targets. " | |
| 12:10:21 | bauzas | that's maybe me but I thought we were having different power management profiles whether the laptop was docked or not, previously | |
| 12:10:32 | bauzas | (in Gnome, I mean) | |
| 12:20:16 | bauzas | https://forums.lenovo.com/t5/Other-Linux-Discussions/Lenovo-L14-Gen1-AMD-Lap-Mode-Throttling/m-p/5154197 | |
| 12:20:51 | bauzas | but my surface is steady... | |
| 12:25:21 | sean-k-mooney | bauzas: so ya i would jsut remove that and go with one of the many other tools | |
| 12:25:35 | bauzas | checked, I'm not in lap mode | |
| 12:25:52 | bauzas | but there are a ton of issues with PPD with Thinkpads https://gitlab.freedesktop.org/hadess/power-profiles-daemon/-/issues | |
| 12:26:40 | bauzas | and that's the issue I hit when changing the PPD profile to 'perf' https://gitlab.freedesktop.org/hadess/power-profiles-daemon/-/issues/72 | |
| 12:28:32 | bauzas | sean-k-mooney: any power management tool you'd recommend that could set the power based on charging vs. discharging ? | |
| 12:28:59 | bauzas | powersave seems reasonable to me if I'm running on battery | |
| 12:29:20 | bauzas | but I don't want to clockdown if I'm plugged | |
| 12:30:58 | opendevreview | Merged openstack/nova master: Retry attachment delete API call for 504 Gateway Timeout https://review.opendev.org/c/openstack/nova/+/845543 | |
| 12:33:15 | sean-k-mooney | bauzas: i use tlp | |
| 12:33:33 | sean-k-mooney | i dont actully use power save on battery but i configure it to limit the max frequency | |
| 12:33:55 | sean-k-mooney | but you can certenly set the profile to power save too if you prefer | |
| 12:34:23 | sean-k-mooney | https://paste.opendev.org/show/bPVClbF8MnuFjrhBE2z3/ | |
| 12:34:27 | sean-k-mooney | that is my tlp config | |
| 12:34:54 | sean-k-mooney | bauzas: so you could change line 6 two powersaver | |
| 12:35:03 | sean-k-mooney | or liek i do on line 11 | |
| 12:35:16 | sean-k-mooney | limite the cpu frequency | |
| 12:36:17 | sean-k-mooney | bauzas: tlp also has a tlp-stats tool which is kind of nice | |
| 12:37:22 | sean-k-mooney | you can see that i still have 90% capasity on my battery after 4 years | |
| 12:37:25 | sean-k-mooney | https://paste.opendev.org/show/bNtQTQ0T2Tx4Bd2nFyLu/ | |
| 12:37:37 | sean-k-mooney | partly becasue i only let it chagne to 80% | |
| 12:38:39 | sean-k-mooney | so i went with tlp since i coudl use it to do more then save power it can also extend battry life by contoling change and i cloud have a little more contol over all | |
| 12:39:37 | sean-k-mooney | https://termbin.com/ic06r is the full set of stats it has | |
| 12:39:47 | sean-k-mooney | and you can twek many of the paramters if you care about it | |
| 12:49:55 | opendevreview | Amit Uniyal proposed openstack/nova master: Adds check, if admin has set compute service down https://review.opendev.org/c/openstack/nova/+/848886 | |
| 12:49:56 | opendevreview | Amit Uniyal proposed openstack/nova master: add regression test case for bug 1978983 https://review.opendev.org/c/openstack/nova/+/849104 | |
| 12:50:58 | bauzas | ok, eventually fixed my issue, upgraded the firware | |
| 12:51:02 | gibi | sean-k-mooney: I'm wondering that we reintroduced a an issue by reenabling greendns. See my fresh bugreport https://bugs.launchpad.net/nova/+bug/1981080 | |
| 12:51:15 | bauzas | when the lid is closed, you can't upgrade your firmware, bizarre but ok | |
| 12:51:28 | bauzas | so this one was PEBKAC | |
| 12:51:57 | bauzas | I'll need to use to change the profile based on the docking info | |
| 12:52:04 | bauzas | need to see* | |
| 13:11:00 | sean-k-mooney | gibi: i was still getting that waring before that | |
| 13:11:20 | gibi | hm, I don't get it if I turn off greendns | |
| 13:12:00 | sean-k-mooney | we coudl perhaps disable greendns in our func tests but i dont think a revert would be correct | |
| 13:12:28 | sean-k-mooney | we shoudl not be doing dns lookup in the func tests | |
| 13:13:36 | gibi | this is not about func test, there it is probably cosmetic. But I wonder if we reintroduced https://bugs.launchpad.net/nova/+bug/1808951 | |
| 13:13:57 | gibi | as the fix for that was to blacklist urllib3 | |
| 13:14:49 | gibi | now we see that urllib3 is imported before monkey patching | |
| 13:14:56 | gibi | so we might retrigger https://bugs.launchpad.net/nova/+bug/1808951 | |
| 13:15:08 | sean-k-mooney | well we now use eventliet directly in the nova-api | |
| 13:15:25 | sean-k-mooney | for scater gather so we cant skip monky patching in general there anymore | |
| 13:16:03 | gibi | I don't follow you how that is related | |
| 13:16:21 | sean-k-mooney | we know that disabling greendns can lead to the nova-api and other serivce had locking up on dns quieres | |
| 13:16:33 | sean-k-mooney | so if we revert the other patch we woudl break things | |
| 13:17:06 | gibi | OK, so if we disable greendns we break thing, but now that we enabled it we might break other things | |
| 13:17:29 | sean-k-mooney | we might but i dont think we will | |
| 13:17:58 | sean-k-mooney | have you found a code path where urllib3 is imported before monkeypatching? | |
| 13:18:10 | gibi | yes, it is the dns code in eventlet that imports it | |
| 13:18:41 | sean-k-mooney | so https://github.com/openstack/nova/blob/90c0c687a487601e009c72f60c88be92f6a55264/nova/monkey_patch.py#L30= | |
| 13:18:47 | sean-k-mooney | imports urllib3? | |
| 13:18:54 | gibi | indireclt yes | |
| 13:19:00 | gibi | indirectly | |
| 13:19:06 | sean-k-mooney | that sounds like an eventlet bug that we canot fix then | |
| 13:19:22 | sean-k-mooney | since we cant monkeypatch until after we import eventlet | |
| 13:20:27 | sean-k-mooney | so ya maybe we need to remove urllib3 form the problem list if eventlet have fixed the issue | |
| 13:20:34 | sean-k-mooney | if not then we shoudl file an eventlet bug | |
| 13:20:50 | sean-k-mooney | unless from nova import debugger | |
| 13:20:56 | sean-k-mooney | is pulling it in instead fo eventlet | |
| 13:21:20 | sean-k-mooney | not that i can see | |
| 13:21:24 | sean-k-mooney | https://github.com/openstack/nova/blob/90c0c687a487601e009c72f60c88be92f6a55264/nova/debugger.py | |
| 13:25:34 | gibi | hm I assuming that deployers had some workaround for our disable greendns as we had that disable for a long time | |
| 13:27:05 | gibi | so we might fixed a known and worked around problem with but re-introduced a break somewhere else. Still I feel like reverting the greendns patch would be safer while we figure out is urllib3 is safe | |
| 13:29:21 | gibi | anyhow updated the bug report linking to this discussion as I have now bandwidth to dig deeper into this now | |
| 13:32:09 | Uggla | gibi, sorry for that.... Uggla is a black cat that attract bug. | |
| 13:33:15 | gibi | Uggla: no worries. I like these kind of challenges :) | |
| 14:13:38 | sean-k-mooney | gibi: it was breaking our downstream customers | |
| 14:13:51 | sean-k-mooney | so we cant assume that | |
| 14:14:10 | sean-k-mooney | gibi: when i did the revert an renebaled greendns | |
| 14:14:28 | sean-k-mooney | i noted you could sitll diablie it by setting the env var | |
| 14:14:57 | sean-k-mooney | gibi: we were hardcodeing os.environ['EVENTLET_NO_GREENDNS'] = 'yes' | |
| 14:14:59 | sean-k-mooney | before | |
| 14:15:12 | sean-k-mooney | but you can just export that before you run nova-api | |
| 14:15:21 | sean-k-mooney | or set it in the wsgi env if you need too | |
| 14:15:25 | sean-k-mooney | but i dont think you should | |
| 14:18:59 | gibi | sean-k-mooney: so if somebody get hit by https://bugs.launchpad.net/nova/+bug/1808951 then they can set EVENTLET_NO_GREENDNS=yes in there api service script to get back the old behavior | |
| 14:19:14 | gibi | good point. I agree that is a viable workaround | |
| 14:19:20 | gibi | I will not that in https://bugs.launchpad.net/nova/+bug/1808951 | |
| 14:19:36 | sean-k-mooney | i just did in https://bugs.launchpad.net/nova/+bug/1981080/comments/2 | |
| 14:20:06 | sean-k-mooney | but yes you could not that in the orginal bug | |
| 14:20:39 | sean-k-mooney | gibi: do you happen to know what the orginal urllib3 bug is and if it was fixed | |
| 14:20:51 | sean-k-mooney | i have read over https://bugs.launchpad.net/nova/+bug/1808951 | |
| 14:21:08 | sean-k-mooney | and that references https://github.com/eventlet/eventlet/issues/371 | |
| 14:21:35 | opendevreview | Sergii Golovatiuk proposed openstack/nova master: Replace "db archive" with "db archive_deleted_raws" https://review.opendev.org/c/openstack/nova/+/847963 | |
| 14:21:42 | sean-k-mooney | that seams to have been fiex and broken | |
| 14:21:49 | sean-k-mooney | based on differnt python releases | |
| 14:22:23 | gibi | I think it is https://github.com/eventlet/eventlet/issues/371 as you noted | |
| 14:22:45 | gibi | at least https://github.com/eventlet/eventlet/issues/371#issuecomment-1047336652 reports that it still exists | |
| 14:22:52 | sean-k-mooney | https://github.com/eventlet/eventlet/issues/726 | |