| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-01-17 | |||
| 21:21:59 | sean-k-mooney | im pretty sure the same parmaters would work well for us in our ci images | |
| 21:31:36 | fungi | if you want to test with that, the fips mode role in zuul-jobs is an example of rebooting with custom kernel options at the start of a job | |
| 21:32:00 | fungi | (bringing this discussion full-circle) | |
| 21:33:00 | sean-k-mooney | oh good point. ya i might put a DNM patch on top to try enabling it and see if it helps with the job. | |
| 21:33:18 | sean-k-mooney | if it does i could take a look at creating a dib element to configure this i guess | |
| 21:34:01 | sean-k-mooney | the default should speed things up a little but you can tune it a little more by cahnging the algoritim and compresor values | |
| 21:34:43 | sean-k-mooney | as always its a trade off between speed and compression ratio but lz4 and zstd are both good choices | |
| 21:35:02 | sean-k-mooney | i think it default to lzo which is not bad but the alternitives are bettter | |
| 21:38:13 | opendevreview | sean mooney proposed openstack/nova master: [WIP] add initial healthcheck support https://review.opendev.org/c/openstack/nova/+/825015 | |
| 21:39:51 | sean-k-mooney | gibi: bauzas i have a lot more work todo but that is the basic infra in place more or less. | |
| 21:40:39 | sean-k-mooney | ill start wiring it into the nova context object and such in follow up patches as well as start working testing in parallel. | |
| 22:03:08 | ade_lee | sean-k-mooney, results are in - that worked a lot better .. | |
| 22:03:25 | ade_lee | sean-k-mooney, https://review.opendev.org/c/openstack/nova/+/790519 | |
| 23:00:56 | clarkb | re working around the centos 8 issue in jobs I think that might be ok as long as you understand other people may not be able to reproduce. From the CI system perspective we won't (and shouldn't) work around it for you as exposing these problems is exactly why we test | |
| #openstack-nova - 2022-01-18 | |||
| 00:48:54 | sean-k-mooney[m] | ade_lee ack. ill take a look again tomorrow but looks like everything passed. i think it would still be worth trying to enable zswap as a seperate patch but glad the oom issue are gone | |
| 06:27:00 | opendevreview | Merged openstack/nova master: Move ReaderWriterLock to the test tree https://review.opendev.org/c/openstack/nova/+/824931 | |
| 07:33:57 | opendevreview | Lior Friedman proposed openstack/nova master: support use_multipath for nvme driver. Update nvme UT. Add release notes. Change-Id: I20f6aa2d422259601c44d024099cd5760a8e21c6 https://review.opendev.org/c/openstack/nova/+/823941 | |
| 08:23:55 | plibeau2 | lyarwood: hello, sorry to push you my pull request again :) https://review.opendev.org/c/openstack/nova/+/820531/5 | |
| 14:18:35 | elodilles | bauzas: are you updating the nova meeting wiki right now? because if not, then i'll update the stable part there | |
| 14:20:39 | bauzas | elodilles: do the stable things | |
| 14:20:44 | bauzas | elodilles: I'll update later | |
| 14:20:50 | bauzas | gibi: food for thoughts https://bugs.launchpad.net/nova/+bug/1956983 | |
| 14:32:19 | sean-k-mooney1 | bauzas: that is propably realted to the other bug we have in that area | |
| 14:32:29 | sean-k-mooney1 | currently FFU is broken | |
| 14:32:51 | sean-k-mooney1 | we are checkign the min service version not the min service version fo the running computes | |
| 14:32:57 | elodilles | bauzas: thanks, done | |
| 14:33:36 | sean-k-mooney1 | so you cant skip level upgrade the contoler to n+3 and start it as it will fail the check even if you have stopped all compute unless you work aroudn it with force down | |
| 14:38:24 | gibi | bauzas: context L398 https://etherpad.opendev.org/p/nova-wallaby-ptg | |
| 14:38:58 | gibi | we prevent starting services with older than N-1 computes since https://review.opendev.org/c/openstack/nova/+/738482 | |
| 14:39:23 | gibi | which is merged in wallaby | |
| 14:39:35 | sean-k-mooney1 | gibi: yep but that has a bug in it | |
| 14:39:45 | sean-k-mooney1 | i assume you have seen the mail thread | |
| 14:39:55 | gibi | sean-k-mooney1: I saw the ML thread, I did not see the bug report yet | |
| 14:40:05 | sean-k-mooney1 | i dont think they filed one | |
| 14:40:25 | gibi | bauzas: so I think the bug is already fixed since wallaby | |
| 14:40:26 | gibi | :) | |
| 14:40:35 | gibi | sean-k-mooney1: as of the bug in the N-1 check | |
| 14:40:37 | sean-k-mooney1 | but looking at the code you cannot start the nova conductor of you have n-2 comptue in the db not just n-2 compute running | |
| 14:40:43 | gibi | sean-k-mooney1: I still need to dig up more context | |
| 14:40:53 | gibi | sean-k-mooney1: I feel our original intention was to detect even stopped computes | |
| 14:40:57 | gibi | but I'm not sure | |
| 14:41:06 | sean-k-mooney1 | well that makes FFU impossible | |
| 14:41:10 | gibi | if FFU is broken then we have to do someting | |
| 14:42:11 | sean-k-mooney1 | well it depend the only escape hatch in the current code is to force all compute as down | |
| 14:42:24 | gibi | which is probably unintentional too :) | |
| 14:42:35 | sean-k-mooney1 | possible | |
| 14:43:09 | gibi | I'm wondering what would happen in an N could if an N-2 compute that was stopped now suddenly started up | |
| 14:43:17 | sean-k-mooney1 | in the ffucase if you update the contoler to n+3 then the condcutre wont be able to start until after you have updated all compute agent but they wont be abel to start cause the conductor is stopped | |
| 14:43:44 | sean-k-mooney1 | gibi: ideally the comptue should fail to start but the conductor should keep running | |
| 14:44:30 | gibi | sean-k-mooney1: OK, so now we probably fail backward as we not allow to start the new controller, but we should fail forward and only not allow the old compute to start up | |
| 14:45:34 | sean-k-mooney1 | no | |
| 14:45:46 | sean-k-mooney1 | we want the contoler to start we dont want the computes to start | |
| 14:45:54 | sean-k-mooney1 | until they are upgraded | |
| 14:46:17 | gibi | do we allow starting up an N-2 controller in an N cloud? | |
| 14:46:24 | gibi | or even an N-1 controller? | |
| 14:46:25 | sean-k-mooney1 | the contoler process, conductor in particalar need to be running so the compute can update its version in the db | |
| 14:46:50 | sean-k-mooney1 | gibi: today proably | |
| 14:46:55 | sean-k-mooney1 | which woudl be unsupported | |
| 14:47:03 | sean-k-mooney1 | since the contoler shoudl always be upgraded first | |
| 14:47:44 | sean-k-mooney1 | i mean if you clamp the rpc apis in the config you can praobly make it work but the assumtion we have is contoler are always newer or the same version as the computes | |
| 14:52:00 | sean-k-mooney | gibi: basicaly i think we shoudl either modify the check to only consider up compute servics or we shoudl move the check to the compute agent instead | |
| 14:52:41 | gibi | hehe, I had considered disabled old computes https://review.opendev.org/c/openstack/nova/+/738482/4..10/nova/service.py#430 | |
| 14:52:42 | sean-k-mooney | or add a config option to opt out of the check for ffu | |
| 14:53:21 | sean-k-mooney | for what its worth i did some testing with devstack not so long ago | |
| 14:53:49 | sean-k-mooney | if i disable the version check you can mix wallaby contoler with train computes | |
| 14:54:57 | sean-k-mooney | to enable better upgrades we are considering supproting that as an upgrade path in the future downstream. not for train to wallaby but from wallaby to whatever our next release would be based on | |
| 14:55:03 | gibi | hm, a disable but running old compute can cause trouble in the cluster, a disable + force down compute cannot as we define force_down as "admin-promised-it-is-not-running" | |
| 14:55:24 | gibi | also based on the discussion in the original patch | |
| 14:55:44 | sean-k-mooney | yes force down means this is down nova just has not noticed yet i promise | |
| 14:56:05 | gibi | so at leat keeping the force_down make sense | |
| 14:56:26 | gibi | I'm not sure about the disable but running compute | |
| 14:56:36 | sean-k-mooney | well i was not thinking about disable | |
| 14:56:42 | sean-k-mooney | i was thinking more up=false | |
| 14:56:45 | gibi | ahh | |
| 14:56:55 | sean-k-mooney | so heartbeat is missing and we mark it as down | |
| 14:56:59 | gibi | so the compute definition exists in the db but the compute is timed out | |
| 14:57:19 | sean-k-mooney | ya presujmably because you stop it on the compute hosts as part of the upgrade | |
| 14:57:28 | gibi | it could be a network partition but it also can be a manual stop yes | |
| 14:58:10 | sean-k-mooney | well if it a network partiion it cant talk to the conductor :) but honestly i think the check is better done on the comptue side then the conductor side | |
| 14:58:33 | gibi | yeah but the check on the compute side only work if the compute is restarted | |
| 14:58:37 | gibi | if the compute is running | |
| 14:58:40 | gibi | then no check happens | |
| 14:58:45 | sean-k-mooney | that is fair | |
| 14:58:47 | gibi | that prevent the old compute to send nonsense to new conductor | |
| 14:59:05 | gibi | probably there is no perfect solution | |
| 14:59:23 | gibi | as from the controller perspective a timed out compute can mean multiple things | |
| 14:59:29 | sean-k-mooney | i mean for now we have the workaround of before you FFU stop all compute agaents and force themn down in the api | |
| 15:00:00 | gibi | yeah, and while that is an extra step it does not create an extra inconsistency | |
| 15:00:12 | gibi | you just need to double confirm that those computes are not coming back | |
| 15:00:16 | gibi | without a restart | |
| 15:00:59 | sean-k-mooney | honestly unless we have done a major rpc bump the comptue wont impact the conductor if they start | |
| 15:01:30 | sean-k-mooney | but yes if you use force down you shoudl fulfil its requiremetns | |
| 15:02:38 | gibi | hm an old compute cannot decided if it is too old, as the old code does not know what is the service version of the N+2 release | |
| 15:03:01 | gibi | except if we backport fixes to stable branches after the N | |
| 15:03:04 | gibi | N+2 release | |
| 15:03:23 | sean-k-mooney | well i was more thingink it would try to connect to the conductor and it would say its too old | |
| 15:03:32 | sean-k-mooney | but ya they cant really | |
| 15:04:10 | gibi | yeah the conductor could tell the compute to go away you are too old | |
| 15:04:35 | sean-k-mooney | it will do that if a new compute tries to regeister | |
| 15:17:13 | bauzas | gibi: I could be late for the meeting, could you please start it if so ? | |