| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-03-03 | |||
| 14:46:23 | sean-k-mooney | im just looking to see if i can confim that in the logs | |
| 14:46:45 | sean-k-mooney | but that is why i was checkign the memory tracker i think we are runnign very close to out of memory if we have not hit it | |
| 14:47:11 | dansmith | on the ceph job my tweaks dropped mysql to half of what it was using (~800m to ~400m) | |
| 14:47:23 | sean-k-mooney | ar 03 13:53:09 np0033355853 kernel: sshd invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0 | |
| 14:47:34 | bauzas | https://zuul.opendev.org/t/openstack/build/ec34b5fa7a354e19a6919d167268cb8b | |
| 14:47:34 | dansmith | but, less memory usage could impair performance and make other things worse of course | |
| 14:47:48 | sean-k-mooney | so yes we are | |
| 14:47:50 | sean-k-mooney | https://zuul.opendev.org/t/openstack/build/ec34b5fa7a354e19a6919d167268cb8b/log/controller/logs/syslog.txt#5871 | |
| 14:47:56 | bauzas | we had two problems | |
| 14:48:01 | bauzas | a unresponsive API | |
| 14:48:11 | bauzas | and some leaked allocs | |
| 14:48:42 | sean-k-mooney | the api issue are because mysql got killed | |
| 14:48:45 | sean-k-mooney | Mar 03 13:53:09 np0033355853 kernel: Out of memory: Killed process 47910 (mysqld) total-vm:5223564kB, anon-rss:328112kB, file-rss:0kB, shmem-rss:0kB, UID:116 pgtables:2648kB oom_score_adj:0 | |
| 14:49:04 | dansmith | yeah that's usually how it works | |
| 14:49:16 | dansmith | mysql is killed and then we stop being able to talk to keystone (et al) | |
| 14:49:36 | sean-k-mooney | yep | |
| 14:50:10 | sean-k-mooney | so 1 we shoudl enabel that devstack feature for nova-next 2 we shoudl consider doing it by default | |
| 14:50:26 | sean-k-mooney | assuming it does not regress over all job time too much | |
| 14:50:40 | dansmith | we were going to do it by default after everyone is branched, to see if it is reasonable across the board, but not to break anyone before release | |
| 14:50:41 | sean-k-mooney | * default in jobs | |
| 14:50:45 | sean-k-mooney | not sure about defaulting in devstack | |
| 14:50:56 | sean-k-mooney | ack | |
| 14:51:00 | dansmith | we have it enabled in some jobs, so we should do that for -next if it can't make it worse | |
| 14:51:06 | dansmith | and then maybe we'll get the defaulting in a month or so | |
| 14:51:25 | dansmith | I can post a patch for -next | |
| 14:51:37 | sean-k-mooney | sounds good | |
| 14:51:48 | bauzas | cool | |
| 14:52:05 | sean-k-mooney | if we are swappign that hard its going to really slow down the job too | |
| 14:52:19 | sean-k-mooney | is the memory reduction much overall | |
| 14:53:05 | dansmith | like I said above, about 800m to 400m rss for mysql | |
| 14:53:21 | bauzas | that's not big | |
| 14:53:43 | bauzas | mysqld seems to be a canary | |
| 14:54:10 | sean-k-mooney | 400m is a lot when we only have 8192mb of ram in the vms | |
| 14:54:31 | opendevreview | Dan Smith proposed openstack/nova master: Make nova-next reduce mysql memory https://review.opendev.org/c/openstack/nova/+/876391 | |
| 14:54:42 | dansmith | yeah, it's a lot :) | |
| 14:54:44 | sean-k-mooney | its like 5% | |
| 14:54:54 | dansmith | it's the single biggest user | |
| 14:55:08 | dansmith | and it puts it down closer to many of the other users like rabbit | |
| 14:55:25 | sean-k-mooney | i just checked and we are alredy limiting nova to 2 worksers as well so we likely wont get much form limiting that more | |
| 14:55:34 | dansmith | yeah | |
| 14:56:49 | bauzas | true but neutron-api takes its own big piece of cake https://zuul.opendev.org/t/openstack/build/ec34b5fa7a354e19a6919d167268cb8b/log/controller/logs/screen-memory_tracker.txt#1993 | |
| 14:58:00 | sean-k-mooney | sure but it looks like they are also limiting to two workers | |
| 14:58:19 | sean-k-mooney | and the reduction in mysql is around the same as both of them combined | |
| 14:58:26 | dansmith | bauzas: if you can find any other 50% reductions and 400m of free ram, please do let me know :) | |
| 14:58:35 | sean-k-mooney | also the nutorn server acts both as the api and conductor for neutron | |
| 14:58:38 | sean-k-mooney | and schduler | |
| 14:59:01 | sean-k-mooney | as in it impelmente everything the contoler would do for neutron | |
| 14:59:05 | bauzas | dansmith: fwiw, I +2d your patch, | |
| 14:59:12 | bauzas | so I'm not debating it :) | |
| 14:59:14 | dansmith | I suspect other gains to be had will be much smaller and much harder to enact :) | |
| 14:59:32 | bauzas | true | |
| 14:59:38 | dansmith | bauzas: I know, you're for it, you're just not impressed, I get it | |
| 14:59:42 | dansmith | I'll just go cry in the corner | |
| 14:59:47 | bauzas | :) | |
| 14:59:47 | sean-k-mooney | i sent it to the ci | |
| 14:59:56 | bauzas | I wish I would have a magic wand | |
| 15:00:09 | sean-k-mooney | what i have wanted to try for a while is enabel zswap | |
| 15:00:34 | bauzas | Bibbidi-Bobbidi-Boo ! | |
| 15:00:50 | sean-k-mooney | that shoudl speed up swap usage a bit and help a little with swap size too | |
| 15:00:51 | bauzas | (shit, doesn't work) | |
| 15:01:04 | dansmith | sean-k-mooney: yeah that might be a thing, but we're also legitimately timing out a lot of jobs, so I'm concerned about slowing anything down with a memory boost causing more of those | |
| 15:01:10 | sean-k-mooney | https://www.omgubuntu.co.uk/2022/01/ubuntu-on-raspberry-pi-4-2gb-zswap | |
| 15:01:33 | dansmith | if we're thrashing I think it will slow us down a lot, if we're stashing bloat we never reference, then it will help | |
| 15:01:43 | sean-k-mooney | once we swap to 22.04 it will be better | |
| 15:01:46 | bauzas | and I suspect this may be related | |
| 15:01:59 | dansmith | sean-k-mooney: what will? | |
| 15:02:31 | sean-k-mooney | dansmith: its much simpler to enable zswap in 22.04 | |
| 15:02:35 | dansmith | oh | |
| 15:02:39 | sean-k-mooney | and they did some performace optiomistaions | |
| 15:02:46 | sean-k-mooney | https://waldorf.waveform.org.uk/2021/6-months-with-the-pi-desktop.html | |
| 15:02:49 | dansmith | but still, you're trading cpu for memory | |
| 15:03:05 | sean-k-mooney | cpu vs disk io performace really | |
| 15:03:07 | dansmith | and right now in addition to running against the memory barrier, we're also running against the cpu barrier | |
| 15:03:27 | sean-k-mooney | compression effectivly makes it like swappign to faster storage | |
| 15:03:37 | sean-k-mooney | if you have the cpu to do it | |
| 15:03:56 | dansmith | sure, if you're io constrained it will help there, but it's all trading for cpu | |
| 15:04:07 | sean-k-mooney | yep | |
| 15:04:26 | sean-k-mooney | i looked into this a while ago when those blogs first came out | |
| 15:04:36 | sean-k-mooney | it proably worth doing an expermient with at least | |
| 15:05:05 | dansmith | yeah, any shorter but fatter jobs will benefit from it for sure, and it could reduce our own noisy neighbor effects | |
| 15:05:06 | sean-k-mooney | honestly just moving to 16G vms in ci woudl be the better solution | |
| 15:05:13 | sean-k-mooney | but since that is not really an option | |
| 15:05:27 | dansmith | I'm just worried about slowing the tests down at all because we're already hitting widespread legit timeouts | |
| 15:05:30 | dansmith | on slower nodes | |
| 15:05:44 | sean-k-mooney | ya which is a concern | |
| 15:08:00 | sean-k-mooney | wow gerrit found https://review.opendev.org/c/openstack/devstack/+/828639 becasue i mention zswap in my last comment on the top level | |
| 16:46:35 | darkhorse | Hi team - I read about service tokens in cinder documentation which is really cool. Can service tokens be used in nova and other services too? | |
| 17:08:41 | dansmith | wow, the nova-next mysql patch passed check the first time through | |
| 17:08:54 | dansmith | bauzas: sean-k-mooney ^ | |
| 17:39:04 | dansmith | melwitt: is this something you saw in the gate? https://review.opendev.org/c/openstack/nova/+/875991 | |
| 17:39:13 | dansmith | I don't see that error message in opensearch | |
| #openstack-nova - 2023-03-04 | |||
| 02:02:21 | opendevreview | Merged openstack/nova master: Revert "Add logging to find test cases leaking libvirt threads" https://review.opendev.org/c/openstack/nova/+/873584 | |
| 07:09:56 | gibi | yeah the revert finally merged | |
| 07:20:34 | gibi | I've updated the release patch | |
| 11:41:03 | bauzas | huzzah | |
| 11:49:24 | bauzas | gibi: +1d https://review.opendev.org/c/openstack/releases/+/875447 | |
| 15:26:17 | opendevreview | Takashi Natsume proposed openstack/nova master: Update contributor guide for 2023.2 Bobcat https://review.opendev.org/c/openstack/nova/+/876447 | |
| 21:05:36 | jrwr | There is a possible bug I've discovered in Zed version of Nova, With Kernel 6.0, Openstack Kolla, and a AMD EYPC 7702, I've ran into a issue with the hyper-v enlightenment that where enabled as of late, | |
| 21:06:28 | jrwr | qemu is throwing errors causing instance creation failures, "qemu-kvm: Hyper-V enlightened VMCS (hv-evmcs) is not supported by kernel" and "qemu-kvm: warning: This feature depends on other features that were not requested: CPUID.8000000AH:EDX.svme-addr-chk [bit 28]" | |
| 21:07:37 | jrwr | /sys/module/kvm_amd/parameters/nested is reporting "1" showing that it should be supported here | |