| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-01-17 | |||
| 20:02:17 | sean-k-mooney | 4096 for centos https://github.com/openstack/devstack/blob/3c98c21fec60da8d2d39df2e1d9b845a51817a0e/.zuul.yaml#L644 | |
| 20:03:16 | sean-k-mooney | which is the same as debain https://github.com/openstack/devstack/commit/f8e00b86aee9a8f9646bf5aed2c618843307b963 | |
| 20:04:21 | sean-k-mooney | im not sure if the vars section are merged | |
| 20:04:25 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/790519/18/.zuul.yaml | |
| 20:04:32 | sean-k-mooney | so you might be overwriting it | |
| 20:05:19 | ade_lee | sean-k-mooney, we made the same change in swift tests - and got better results .. | |
| 20:05:23 | sean-k-mooney | ya you are https://0d2f8596f517931fcac8-e8ce9722d0830f321adb7d7d98f7ea38.ssl.cf1.rackcdn.com/790519/18/check/nova-centos8-stream-fips/372ef4e/zuul-info/inventory.yaml | |
| 20:05:45 | sean-k-mooney | ade_lee: swift does not spwan vms | |
| 20:06:06 | ade_lee | sean-k-mooney, ack | |
| 20:06:14 | sean-k-mooney | the swap is needed because qemu started allocating a 1GB cache for tcg byty code per qemu process | |
| 20:06:41 | ade_lee | sean-k-mooney, so try 8192? | |
| 20:06:44 | sean-k-mooney | meanign your 128mb vms now takes 1156mb | |
| 20:06:53 | sean-k-mooney | well right now you have 0 | |
| 20:07:03 | sean-k-mooney | so add configure_swap_size: 4096 | |
| 20:07:12 | ade_lee | ok so lets try 4096 .. | |
| 20:07:33 | sean-k-mooney | you can bump it to configure_swap_size: 8192 but we also only have 80G of disk space in the vms so dont set it too large | |
| 20:07:48 | sean-k-mooney | fungi: have we considerd using zram by the way | |
| 20:07:58 | sean-k-mooney | in the long run that might help in general | |
| 20:08:53 | opendevreview | Ade Lee proposed openstack/nova master: Add check job for FIPS https://review.opendev.org/c/openstack/nova/+/790519 | |
| 20:09:32 | sean-k-mooney | lets see if that helps and we can take another look tomorow | |
| 20:15:08 | ade_lee | sean-k-mooney, ack | |
| 20:27:04 | fungi | sean-k-mooney: i'm not familiar with zram, is that on-the-fly memory compression? | |
| 20:53:17 | sean-k-mooney | fungi: yes basically it creats a compressed ramdisk that you then confirure for swap with a higher priority then normal swap on disk | |
| 20:53:39 | sean-k-mooney | with zstd compression you can get up to a 4x ratio | |
| 20:53:59 | sean-k-mooney | fungi: fedora started using it by default for laptops and low memeory isntalls | |
| 20:54:32 | fungi | mmm... then why not just compress the ram itself? i guess it's a simple composition to do the same, the extra layers just give me a moment's pause | |
| 20:55:03 | sean-k-mooney | fungi: it is simpler to integrate in the kernel this way | |
| 20:55:34 | sean-k-mooney | you can use zswap which will do it transparently but zram+swap has is more compatiable | |
| 20:55:55 | sean-k-mooney | windows and osx just compress ram transparently | |
| 20:56:24 | sean-k-mooney | but on linux the least intrusive way is to just add the zram device as a higher priority swap device | |
| 20:56:41 | sean-k-mooney | https://fedoraproject.org/wiki/Changes/Scale_ZRAM_to_full_memory_size | |
| 20:57:55 | sean-k-mooney | https://fedoraproject.org/wiki/Changes/SwapOnZRAM | |
| 20:58:06 | sean-k-mooney | thos are the two relevent fedora propsoals | |
| 21:01:01 | sean-k-mooney | oh ubuntu 22.04 might have it on by defaul https://www.cnx-software.com/2022/01/13/ubuntu-22-04-zswap-raspberry-pi-4-2gb-ram/ | |
| 21:03:49 | sean-k-mooney | looks like they are going with zswap directly which also works | |
| 21:19:23 | fungi | neat, well i expect we'll have 22.04 lts beta images available within a couple months | |
| 21:20:45 | sean-k-mooney | the zram feature has been avaiable in the herel since 2013 so technially we coudl enable it for our existing image by just adding zswap.enabled=1 to the kernel boot line in dib | |
| 21:21:25 | sean-k-mooney | it might be fun to play with that and see if it helps in general | |
| 21:21:41 | sean-k-mooney | https://ubuntu.com//blog/how-low-can-you-go-running-ubuntu-desktop-on-a-2gb-raspberry-pi-4 is a nice write up on how they used it to make the 2GB rpi work | |
| 21:21:59 | sean-k-mooney | im pretty sure the same parmaters would work well for us in our ci images | |
| 21:31:36 | fungi | if you want to test with that, the fips mode role in zuul-jobs is an example of rebooting with custom kernel options at the start of a job | |
| 21:32:00 | fungi | (bringing this discussion full-circle) | |
| 21:33:00 | sean-k-mooney | oh good point. ya i might put a DNM patch on top to try enabling it and see if it helps with the job. | |
| 21:33:18 | sean-k-mooney | if it does i could take a look at creating a dib element to configure this i guess | |
| 21:34:01 | sean-k-mooney | the default should speed things up a little but you can tune it a little more by cahnging the algoritim and compresor values | |
| 21:34:43 | sean-k-mooney | as always its a trade off between speed and compression ratio but lz4 and zstd are both good choices | |
| 21:35:02 | sean-k-mooney | i think it default to lzo which is not bad but the alternitives are bettter | |
| 21:38:13 | opendevreview | sean mooney proposed openstack/nova master: [WIP] add initial healthcheck support https://review.opendev.org/c/openstack/nova/+/825015 | |
| 21:39:51 | sean-k-mooney | gibi: bauzas i have a lot more work todo but that is the basic infra in place more or less. | |
| 21:40:39 | sean-k-mooney | ill start wiring it into the nova context object and such in follow up patches as well as start working testing in parallel. | |
| 22:03:08 | ade_lee | sean-k-mooney, results are in - that worked a lot better .. | |
| 22:03:25 | ade_lee | sean-k-mooney, https://review.opendev.org/c/openstack/nova/+/790519 | |
| 23:00:56 | clarkb | re working around the centos 8 issue in jobs I think that might be ok as long as you understand other people may not be able to reproduce. From the CI system perspective we won't (and shouldn't) work around it for you as exposing these problems is exactly why we test | |
| #openstack-nova - 2022-01-18 | |||
| 00:48:54 | sean-k-mooney[m] | ade_lee ack. ill take a look again tomorrow but looks like everything passed. i think it would still be worth trying to enable zswap as a seperate patch but glad the oom issue are gone | |
| 06:27:00 | opendevreview | Merged openstack/nova master: Move ReaderWriterLock to the test tree https://review.opendev.org/c/openstack/nova/+/824931 | |
| 07:33:57 | opendevreview | Lior Friedman proposed openstack/nova master: support use_multipath for nvme driver. Update nvme UT. Add release notes. Change-Id: I20f6aa2d422259601c44d024099cd5760a8e21c6 https://review.opendev.org/c/openstack/nova/+/823941 | |
| 08:23:55 | plibeau2 | lyarwood: hello, sorry to push you my pull request again :) https://review.opendev.org/c/openstack/nova/+/820531/5 | |
| 14:18:35 | elodilles | bauzas: are you updating the nova meeting wiki right now? because if not, then i'll update the stable part there | |
| 14:20:39 | bauzas | elodilles: do the stable things | |
| 14:20:44 | bauzas | elodilles: I'll update later | |
| 14:20:50 | bauzas | gibi: food for thoughts https://bugs.launchpad.net/nova/+bug/1956983 | |
| 14:32:19 | sean-k-mooney1 | bauzas: that is propably realted to the other bug we have in that area | |
| 14:32:29 | sean-k-mooney1 | currently FFU is broken | |
| 14:32:51 | sean-k-mooney1 | we are checkign the min service version not the min service version fo the running computes | |
| 14:32:57 | elodilles | bauzas: thanks, done | |
| 14:33:36 | sean-k-mooney1 | so you cant skip level upgrade the contoler to n+3 and start it as it will fail the check even if you have stopped all compute unless you work aroudn it with force down | |
| 14:38:24 | gibi | bauzas: context L398 https://etherpad.opendev.org/p/nova-wallaby-ptg | |
| 14:38:58 | gibi | we prevent starting services with older than N-1 computes since https://review.opendev.org/c/openstack/nova/+/738482 | |
| 14:39:23 | gibi | which is merged in wallaby | |
| 14:39:35 | sean-k-mooney1 | gibi: yep but that has a bug in it | |
| 14:39:45 | sean-k-mooney1 | i assume you have seen the mail thread | |
| 14:39:55 | gibi | sean-k-mooney1: I saw the ML thread, I did not see the bug report yet | |
| 14:40:05 | sean-k-mooney1 | i dont think they filed one | |
| 14:40:25 | gibi | bauzas: so I think the bug is already fixed since wallaby | |
| 14:40:26 | gibi | :) | |
| 14:40:35 | gibi | sean-k-mooney1: as of the bug in the N-1 check | |
| 14:40:37 | sean-k-mooney1 | but looking at the code you cannot start the nova conductor of you have n-2 comptue in the db not just n-2 compute running | |
| 14:40:43 | gibi | sean-k-mooney1: I still need to dig up more context | |
| 14:40:53 | gibi | sean-k-mooney1: I feel our original intention was to detect even stopped computes | |
| 14:40:57 | gibi | but I'm not sure | |
| 14:41:06 | sean-k-mooney1 | well that makes FFU impossible | |
| 14:41:10 | gibi | if FFU is broken then we have to do someting | |
| 14:42:11 | sean-k-mooney1 | well it depend the only escape hatch in the current code is to force all compute as down | |
| 14:42:24 | gibi | which is probably unintentional too :) | |
| 14:42:35 | sean-k-mooney1 | possible | |
| 14:43:09 | gibi | I'm wondering what would happen in an N could if an N-2 compute that was stopped now suddenly started up | |
| 14:43:17 | sean-k-mooney1 | in the ffucase if you update the contoler to n+3 then the condcutre wont be able to start until after you have updated all compute agent but they wont be abel to start cause the conductor is stopped | |
| 14:43:44 | sean-k-mooney1 | gibi: ideally the comptue should fail to start but the conductor should keep running | |
| 14:44:30 | gibi | sean-k-mooney1: OK, so now we probably fail backward as we not allow to start the new controller, but we should fail forward and only not allow the old compute to start up | |
| 14:45:34 | sean-k-mooney1 | no | |
| 14:45:46 | sean-k-mooney1 | we want the contoler to start we dont want the computes to start | |
| 14:45:54 | sean-k-mooney1 | until they are upgraded | |
| 14:46:17 | gibi | do we allow starting up an N-2 controller in an N cloud? | |
| 14:46:24 | gibi | or even an N-1 controller? | |
| 14:46:25 | sean-k-mooney1 | the contoler process, conductor in particalar need to be running so the compute can update its version in the db | |
| 14:46:50 | sean-k-mooney1 | gibi: today proably | |
| 14:46:55 | sean-k-mooney1 | which woudl be unsupported | |
| 14:47:03 | sean-k-mooney1 | since the contoler shoudl always be upgraded first | |
| 14:47:44 | sean-k-mooney1 | i mean if you clamp the rpc apis in the config you can praobly make it work but the assumtion we have is contoler are always newer or the same version as the computes | |
| 14:52:00 | sean-k-mooney | gibi: basicaly i think we shoudl either modify the check to only consider up compute servics or we shoudl move the check to the compute agent instead | |