| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-01-17 | |||
| 19:47:17 | sean-k-mooney | i.e. the fact they broke ping | |
| 19:47:23 | sean-k-mooney | making it require root to run | |
| 19:47:55 | ade_lee | sean-k-mooney, not sure .. that woud certainly muddy things | |
| 19:48:14 | ade_lee | fungi, clarkb ^^ any idea? | |
| 19:49:01 | fungi | sean-k-mooney: ade_lee: i don't know if red hat has fixed centos stream 8's packages yet | |
| 19:49:42 | sean-k-mooney | the fips failrues might be unrealtaed but that is breaking all centos8 stream jobs currently right | |
| 19:49:45 | fungi | technically, centos stream 8 was (maybe still is) broken/regressed. i recommend not relying on it since it doesn't seem like they actually test it like they did the old centos | |
| 19:50:18 | sean-k-mooney | fungi: for what its worth i agree with not working aroudn this in our jobs | |
| 19:50:21 | fungi | i think there's work underway to get rocky linux added, and we have openeuler though it sounds like it may have a slightly too old kernel for the fips testing | |
| 19:50:50 | sean-k-mooney | fungi: really centos 8 stream should be fixed and maintianed as if it was rhel | |
| 19:51:36 | fungi | rocky or alma seems like it might be a better option if centos stream is going to be the unstable hopper where free users get to find the bugs and tell red hat what to fix before they tag a new release of their commercial product | |
| 19:51:36 | sean-k-mooney | ade_lee: qemu-kvm: -accel tcg: mprotect of jit buffer: Cannot allocate memory | |
| 19:52:41 | sean-k-mooney | fungi: as someone who has to maintain said product if they dont keep centos stream stable i will continue to do all my dev on ubuntu | |
| 19:54:23 | sean-k-mooney | i really dont like how closely tied our openstack product is to rhel or the rhel lifecycle today. if it is to contibute more value then the bruden it brings they need to keep centos 8 stream stable or make licenses avaiabel for opensrouce to sue for ci on rhel itself | |
| 19:54:50 | fungi | but yeah, maybe this is a one-off, or maybe it'll be the kick needed to get more thorough testing in place for centos stream package updates | |
| 19:54:51 | sean-k-mooney | if they dont using rocky or alma and not support centos makes complete sense | |
| 19:55:53 | sean-k-mooney | fungi: we have a log runnign downstream backlog itme to get a centos stream devstack jobs running in our donwstream ci at somepoint but making it work with our patched repos is non tivial | |
| 19:56:11 | ade_lee | sean-k-mooney, so -- the mprotect thing -- what does that mean? | |
| 19:56:33 | sean-k-mooney | ade_lee: it looks like the host vm ran out of memory | |
| 19:57:46 | ade_lee | sean-k-mooney, so we should be testing using larger vms? I think I recall folks having to do that somewhere -- checking .. | |
| 19:58:05 | sean-k-mooney | ade_lee: no you need to reduce the concurancy | |
| 19:58:12 | sean-k-mooney | ade_lee: how have you defiend the regex | |
| 19:58:20 | fungi | unless you can get by with smaller flavors for the cirros guests or something | |
| 19:58:24 | sean-k-mooney | the senario test need to be run serially after the other test | |
| 19:59:03 | sean-k-mooney | fungi: the vms are using 128mb flavors | |
| 19:59:12 | sean-k-mooney | that should be ok | |
| 19:59:41 | sean-k-mooney | i wonder if the host vm is missing the extended swap | |
| 20:00:05 | sean-k-mooney | that we are using to workaround the 1G tcg buffer | |
| 20:00:13 | ade_lee | sean-k-mooney, the cinder tests are running with swap .. https://review.opendev.org/c/openstack/cinder/+/790535/24/.zuul.yaml | |
| 20:00:22 | ade_lee | configure_swap_size: 4096 | |
| 20:00:39 | sean-k-mooney | that proably needs to be increased | |
| 20:01:00 | sean-k-mooney | fungi: do you know what we set it to for the debian jobs | |
| 20:01:31 | ade_lee | sean-k-mooney, ack - well , its not set at all in the nova tests we just ran -- so we need at least that .. | |
| 20:01:50 | sean-k-mooney | devstack allocates swap by default i belive | |
| 20:02:17 | sean-k-mooney | 4096 for centos https://github.com/openstack/devstack/blob/3c98c21fec60da8d2d39df2e1d9b845a51817a0e/.zuul.yaml#L644 | |
| 20:03:16 | sean-k-mooney | which is the same as debain https://github.com/openstack/devstack/commit/f8e00b86aee9a8f9646bf5aed2c618843307b963 | |
| 20:04:21 | sean-k-mooney | im not sure if the vars section are merged | |
| 20:04:25 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/790519/18/.zuul.yaml | |
| 20:04:32 | sean-k-mooney | so you might be overwriting it | |
| 20:05:19 | ade_lee | sean-k-mooney, we made the same change in swift tests - and got better results .. | |
| 20:05:23 | sean-k-mooney | ya you are https://0d2f8596f517931fcac8-e8ce9722d0830f321adb7d7d98f7ea38.ssl.cf1.rackcdn.com/790519/18/check/nova-centos8-stream-fips/372ef4e/zuul-info/inventory.yaml | |
| 20:05:45 | sean-k-mooney | ade_lee: swift does not spwan vms | |
| 20:06:06 | ade_lee | sean-k-mooney, ack | |
| 20:06:14 | sean-k-mooney | the swap is needed because qemu started allocating a 1GB cache for tcg byty code per qemu process | |
| 20:06:41 | ade_lee | sean-k-mooney, so try 8192? | |
| 20:06:44 | sean-k-mooney | meanign your 128mb vms now takes 1156mb | |
| 20:06:53 | sean-k-mooney | well right now you have 0 | |
| 20:07:03 | sean-k-mooney | so add configure_swap_size: 4096 | |
| 20:07:12 | ade_lee | ok so lets try 4096 .. | |
| 20:07:33 | sean-k-mooney | you can bump it to configure_swap_size: 8192 but we also only have 80G of disk space in the vms so dont set it too large | |
| 20:07:48 | sean-k-mooney | fungi: have we considerd using zram by the way | |
| 20:07:58 | sean-k-mooney | in the long run that might help in general | |
| 20:08:53 | opendevreview | Ade Lee proposed openstack/nova master: Add check job for FIPS https://review.opendev.org/c/openstack/nova/+/790519 | |
| 20:09:32 | sean-k-mooney | lets see if that helps and we can take another look tomorow | |
| 20:15:08 | ade_lee | sean-k-mooney, ack | |
| 20:27:04 | fungi | sean-k-mooney: i'm not familiar with zram, is that on-the-fly memory compression? | |
| 20:53:17 | sean-k-mooney | fungi: yes basically it creats a compressed ramdisk that you then confirure for swap with a higher priority then normal swap on disk | |
| 20:53:39 | sean-k-mooney | with zstd compression you can get up to a 4x ratio | |
| 20:53:59 | sean-k-mooney | fungi: fedora started using it by default for laptops and low memeory isntalls | |
| 20:54:32 | fungi | mmm... then why not just compress the ram itself? i guess it's a simple composition to do the same, the extra layers just give me a moment's pause | |
| 20:55:03 | sean-k-mooney | fungi: it is simpler to integrate in the kernel this way | |
| 20:55:34 | sean-k-mooney | you can use zswap which will do it transparently but zram+swap has is more compatiable | |
| 20:55:55 | sean-k-mooney | windows and osx just compress ram transparently | |
| 20:56:24 | sean-k-mooney | but on linux the least intrusive way is to just add the zram device as a higher priority swap device | |
| 20:56:41 | sean-k-mooney | https://fedoraproject.org/wiki/Changes/Scale_ZRAM_to_full_memory_size | |
| 20:57:55 | sean-k-mooney | https://fedoraproject.org/wiki/Changes/SwapOnZRAM | |
| 20:58:06 | sean-k-mooney | thos are the two relevent fedora propsoals | |
| 21:01:01 | sean-k-mooney | oh ubuntu 22.04 might have it on by defaul https://www.cnx-software.com/2022/01/13/ubuntu-22-04-zswap-raspberry-pi-4-2gb-ram/ | |
| 21:03:49 | sean-k-mooney | looks like they are going with zswap directly which also works | |
| 21:19:23 | fungi | neat, well i expect we'll have 22.04 lts beta images available within a couple months | |
| 21:20:45 | sean-k-mooney | the zram feature has been avaiable in the herel since 2013 so technially we coudl enable it for our existing image by just adding zswap.enabled=1 to the kernel boot line in dib | |
| 21:21:25 | sean-k-mooney | it might be fun to play with that and see if it helps in general | |
| 21:21:41 | sean-k-mooney | https://ubuntu.com//blog/how-low-can-you-go-running-ubuntu-desktop-on-a-2gb-raspberry-pi-4 is a nice write up on how they used it to make the 2GB rpi work | |
| 21:21:59 | sean-k-mooney | im pretty sure the same parmaters would work well for us in our ci images | |
| 21:31:36 | fungi | if you want to test with that, the fips mode role in zuul-jobs is an example of rebooting with custom kernel options at the start of a job | |
| 21:32:00 | fungi | (bringing this discussion full-circle) | |
| 21:33:00 | sean-k-mooney | oh good point. ya i might put a DNM patch on top to try enabling it and see if it helps with the job. | |
| 21:33:18 | sean-k-mooney | if it does i could take a look at creating a dib element to configure this i guess | |
| 21:34:01 | sean-k-mooney | the default should speed things up a little but you can tune it a little more by cahnging the algoritim and compresor values | |
| 21:34:43 | sean-k-mooney | as always its a trade off between speed and compression ratio but lz4 and zstd are both good choices | |
| 21:35:02 | sean-k-mooney | i think it default to lzo which is not bad but the alternitives are bettter | |
| 21:38:13 | opendevreview | sean mooney proposed openstack/nova master: [WIP] add initial healthcheck support https://review.opendev.org/c/openstack/nova/+/825015 | |
| 21:39:51 | sean-k-mooney | gibi: bauzas i have a lot more work todo but that is the basic infra in place more or less. | |
| 21:40:39 | sean-k-mooney | ill start wiring it into the nova context object and such in follow up patches as well as start working testing in parallel. | |
| 22:03:08 | ade_lee | sean-k-mooney, results are in - that worked a lot better .. | |
| 22:03:25 | ade_lee | sean-k-mooney, https://review.opendev.org/c/openstack/nova/+/790519 | |
| 23:00:56 | clarkb | re working around the centos 8 issue in jobs I think that might be ok as long as you understand other people may not be able to reproduce. From the CI system perspective we won't (and shouldn't) work around it for you as exposing these problems is exactly why we test | |
| #openstack-nova - 2022-01-18 | |||
| 00:48:54 | sean-k-mooney[m] | ade_lee ack. ill take a look again tomorrow but looks like everything passed. i think it would still be worth trying to enable zswap as a seperate patch but glad the oom issue are gone | |
| 06:27:00 | opendevreview | Merged openstack/nova master: Move ReaderWriterLock to the test tree https://review.opendev.org/c/openstack/nova/+/824931 | |
| 07:33:57 | opendevreview | Lior Friedman proposed openstack/nova master: support use_multipath for nvme driver. Update nvme UT. Add release notes. Change-Id: I20f6aa2d422259601c44d024099cd5760a8e21c6 https://review.opendev.org/c/openstack/nova/+/823941 | |
| 08:23:55 | plibeau2 | lyarwood: hello, sorry to push you my pull request again :) https://review.opendev.org/c/openstack/nova/+/820531/5 | |
| 14:18:35 | elodilles | bauzas: are you updating the nova meeting wiki right now? because if not, then i'll update the stable part there | |
| 14:20:39 | bauzas | elodilles: do the stable things | |
| 14:20:44 | bauzas | elodilles: I'll update later | |
| 14:20:50 | bauzas | gibi: food for thoughts https://bugs.launchpad.net/nova/+bug/1956983 | |
| 14:32:19 | sean-k-mooney1 | bauzas: that is propably realted to the other bug we have in that area | |
| 14:32:29 | sean-k-mooney1 | currently FFU is broken | |
| 14:32:51 | sean-k-mooney1 | we are checkign the min service version not the min service version fo the running computes | |
| 14:32:57 | elodilles | bauzas: thanks, done | |
| 14:33:36 | sean-k-mooney1 | so you cant skip level upgrade the contoler to n+3 and start it as it will fail the check even if you have stopped all compute unless you work aroudn it with force down | |