| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-03-03 | |||
| 14:39:15 | dansmith | 18 other things in the gate right now | |
| 14:39:30 | gibi | bauzas: I can shepherd the patch during Saturday and a bit on Sunday as well. | |
| 14:39:35 | dansmith | so it'll be a bit if it re-runs, but it's also not a critical patch | |
| 14:40:03 | sean-k-mooney | post_failure form nova next. unfortunet | |
| 14:40:04 | bauzas | dansmith: I don't disagree | |
| 14:40:21 | bauzas | but it will be a bit of a pain to backport the revert if we go | |
| 14:40:49 | bauzas | if the release team says they're OK with releasing on Monday, then meh, we gonna try this weekend | |
| 14:41:00 | bauzas | gibi: last time you were way luckier than me | |
| 14:42:48 | sean-k-mooney | we technially didnt run out of memory but it got pretty clsoe memory_tracker low_point: 730 | |
| 14:43:01 | sean-k-mooney | * memory_tracker low_point: 7308 | |
| 14:43:16 | dansmith | sean-k-mooney: has nova-next been OOMing? | |
| 14:43:26 | sean-k-mooney | MemAvailable: 9152 kB | |
| 14:43:51 | sean-k-mooney | i think its been surviing because of swap | |
| 14:44:02 | sean-k-mooney | Mar 03 13:53:10.114492 np0033355853 memory_tracker.sh[131948]: SwapTotal: 4194300 kB | |
| 14:44:04 | sean-k-mooney | Mar 03 13:53:10.114492 np0033355853 memory_tracker.sh[131948]: SwapFree: 0 kB | |
| 14:44:18 | sean-k-mooney | https://zuul.opendev.org/t/openstack/build/ec34b5fa7a354e19a6919d167268cb8b/log/controller/logs/screen-memory_tracker.txt#3011 | |
| 14:44:59 | sean-k-mooney | dansmith: are you wondering if this is related to the mariadb tweaks ye did | |
| 14:45:31 | sean-k-mooney | keystone was giving 503s | |
| 14:45:33 | dansmith | sean-k-mooney: those tweaks are disabled by default in devstack right now | |
| 14:45:51 | dansmith | I'm just saying if we're memory constrained on that job, we might want to enable those tweaks | |
| 14:45:57 | sean-k-mooney | and when i see that it often because of the db/service getting oom killed | |
| 14:46:02 | dansmith | it seem to have done well for the ceph one | |
| 14:46:03 | sean-k-mooney | yep | |
| 14:46:23 | sean-k-mooney | im just looking to see if i can confim that in the logs | |
| 14:46:45 | sean-k-mooney | but that is why i was checkign the memory tracker i think we are runnign very close to out of memory if we have not hit it | |
| 14:47:11 | dansmith | on the ceph job my tweaks dropped mysql to half of what it was using (~800m to ~400m) | |
| 14:47:23 | sean-k-mooney | ar 03 13:53:09 np0033355853 kernel: sshd invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0 | |
| 14:47:34 | bauzas | https://zuul.opendev.org/t/openstack/build/ec34b5fa7a354e19a6919d167268cb8b | |
| 14:47:34 | dansmith | but, less memory usage could impair performance and make other things worse of course | |
| 14:47:48 | sean-k-mooney | so yes we are | |
| 14:47:50 | sean-k-mooney | https://zuul.opendev.org/t/openstack/build/ec34b5fa7a354e19a6919d167268cb8b/log/controller/logs/syslog.txt#5871 | |
| 14:47:56 | bauzas | we had two problems | |
| 14:48:01 | bauzas | a unresponsive API | |
| 14:48:11 | bauzas | and some leaked allocs | |
| 14:48:42 | sean-k-mooney | the api issue are because mysql got killed | |
| 14:48:45 | sean-k-mooney | Mar 03 13:53:09 np0033355853 kernel: Out of memory: Killed process 47910 (mysqld) total-vm:5223564kB, anon-rss:328112kB, file-rss:0kB, shmem-rss:0kB, UID:116 pgtables:2648kB oom_score_adj:0 | |
| 14:49:04 | dansmith | yeah that's usually how it works | |
| 14:49:16 | dansmith | mysql is killed and then we stop being able to talk to keystone (et al) | |
| 14:49:36 | sean-k-mooney | yep | |
| 14:50:10 | sean-k-mooney | so 1 we shoudl enabel that devstack feature for nova-next 2 we shoudl consider doing it by default | |
| 14:50:26 | sean-k-mooney | assuming it does not regress over all job time too much | |
| 14:50:40 | dansmith | we were going to do it by default after everyone is branched, to see if it is reasonable across the board, but not to break anyone before release | |
| 14:50:41 | sean-k-mooney | * default in jobs | |
| 14:50:45 | sean-k-mooney | not sure about defaulting in devstack | |
| 14:50:56 | sean-k-mooney | ack | |
| 14:51:00 | dansmith | we have it enabled in some jobs, so we should do that for -next if it can't make it worse | |
| 14:51:06 | dansmith | and then maybe we'll get the defaulting in a month or so | |
| 14:51:25 | dansmith | I can post a patch for -next | |
| 14:51:37 | sean-k-mooney | sounds good | |
| 14:51:48 | bauzas | cool | |
| 14:52:05 | sean-k-mooney | if we are swappign that hard its going to really slow down the job too | |
| 14:52:19 | sean-k-mooney | is the memory reduction much overall | |
| 14:53:05 | dansmith | like I said above, about 800m to 400m rss for mysql | |
| 14:53:21 | bauzas | that's not big | |
| 14:53:43 | bauzas | mysqld seems to be a canary | |
| 14:54:10 | sean-k-mooney | 400m is a lot when we only have 8192mb of ram in the vms | |
| 14:54:31 | opendevreview | Dan Smith proposed openstack/nova master: Make nova-next reduce mysql memory https://review.opendev.org/c/openstack/nova/+/876391 | |
| 14:54:42 | dansmith | yeah, it's a lot :) | |
| 14:54:44 | sean-k-mooney | its like 5% | |
| 14:54:54 | dansmith | it's the single biggest user | |
| 14:55:08 | dansmith | and it puts it down closer to many of the other users like rabbit | |
| 14:55:25 | sean-k-mooney | i just checked and we are alredy limiting nova to 2 worksers as well so we likely wont get much form limiting that more | |
| 14:55:34 | dansmith | yeah | |
| 14:56:49 | bauzas | true but neutron-api takes its own big piece of cake https://zuul.opendev.org/t/openstack/build/ec34b5fa7a354e19a6919d167268cb8b/log/controller/logs/screen-memory_tracker.txt#1993 | |
| 14:58:00 | sean-k-mooney | sure but it looks like they are also limiting to two workers | |
| 14:58:19 | sean-k-mooney | and the reduction in mysql is around the same as both of them combined | |
| 14:58:26 | dansmith | bauzas: if you can find any other 50% reductions and 400m of free ram, please do let me know :) | |
| 14:58:35 | sean-k-mooney | also the nutorn server acts both as the api and conductor for neutron | |
| 14:58:38 | sean-k-mooney | and schduler | |
| 14:59:01 | sean-k-mooney | as in it impelmente everything the contoler would do for neutron | |
| 14:59:05 | bauzas | dansmith: fwiw, I +2d your patch, | |
| 14:59:12 | bauzas | so I'm not debating it :) | |
| 14:59:14 | dansmith | I suspect other gains to be had will be much smaller and much harder to enact :) | |
| 14:59:32 | bauzas | true | |
| 14:59:38 | dansmith | bauzas: I know, you're for it, you're just not impressed, I get it | |
| 14:59:42 | dansmith | I'll just go cry in the corner | |
| 14:59:47 | bauzas | :) | |
| 14:59:47 | sean-k-mooney | i sent it to the ci | |
| 14:59:56 | bauzas | I wish I would have a magic wand | |
| 15:00:09 | sean-k-mooney | what i have wanted to try for a while is enabel zswap | |
| 15:00:34 | bauzas | Bibbidi-Bobbidi-Boo ! | |
| 15:00:50 | sean-k-mooney | that shoudl speed up swap usage a bit and help a little with swap size too | |
| 15:00:51 | bauzas | (shit, doesn't work) | |
| 15:01:04 | dansmith | sean-k-mooney: yeah that might be a thing, but we're also legitimately timing out a lot of jobs, so I'm concerned about slowing anything down with a memory boost causing more of those | |
| 15:01:10 | sean-k-mooney | https://www.omgubuntu.co.uk/2022/01/ubuntu-on-raspberry-pi-4-2gb-zswap | |
| 15:01:33 | dansmith | if we're thrashing I think it will slow us down a lot, if we're stashing bloat we never reference, then it will help | |
| 15:01:43 | sean-k-mooney | once we swap to 22.04 it will be better | |
| 15:01:46 | bauzas | and I suspect this may be related | |
| 15:01:59 | dansmith | sean-k-mooney: what will? | |
| 15:02:31 | sean-k-mooney | dansmith: its much simpler to enable zswap in 22.04 | |
| 15:02:35 | dansmith | oh | |
| 15:02:39 | sean-k-mooney | and they did some performace optiomistaions | |
| 15:02:46 | sean-k-mooney | https://waldorf.waveform.org.uk/2021/6-months-with-the-pi-desktop.html | |
| 15:02:49 | dansmith | but still, you're trading cpu for memory | |
| 15:03:05 | sean-k-mooney | cpu vs disk io performace really | |
| 15:03:07 | dansmith | and right now in addition to running against the memory barrier, we're also running against the cpu barrier | |
| 15:03:27 | sean-k-mooney | compression effectivly makes it like swappign to faster storage | |
| 15:03:37 | sean-k-mooney | if you have the cpu to do it | |
| 15:03:56 | dansmith | sure, if you're io constrained it will help there, but it's all trading for cpu | |
| 15:04:07 | sean-k-mooney | yep | |