Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2022-01-18
14:18:35 elodilles bauzas: are you updating the nova meeting wiki right now? because if not, then i'll update the stable part there
14:20:39 bauzas elodilles: do the stable things
14:20:44 bauzas elodilles: I'll update later
14:20:50 bauzas gibi: food for thoughts https://bugs.launchpad.net/nova/+bug/1956983
14:32:19 sean-k-mooney1 bauzas: that is propably realted to the other bug we have in that area
14:32:29 sean-k-mooney1 currently FFU is broken
14:32:51 sean-k-mooney1 we are checkign the min service version not the min service version fo the running computes
14:32:57 elodilles bauzas: thanks, done
14:33:36 sean-k-mooney1 so you cant skip level upgrade the contoler to n+3 and start it as it will fail the check even if you have stopped all compute unless you work aroudn it with force down
14:38:24 gibi bauzas: context L398 https://etherpad.opendev.org/p/nova-wallaby-ptg
14:38:58 gibi we prevent starting services with older than N-1 computes since https://review.opendev.org/c/openstack/nova/+/738482
14:39:23 gibi which is merged in wallaby
14:39:35 sean-k-mooney1 gibi: yep but that has a bug in it
14:39:45 sean-k-mooney1 i assume you have seen the mail thread
14:39:55 gibi sean-k-mooney1: I saw the ML thread, I did not see the bug report yet
14:40:05 sean-k-mooney1 i dont think they filed one
14:40:25 gibi bauzas: so I think the bug is already fixed since wallaby
14:40:26 gibi :)
14:40:35 gibi sean-k-mooney1: as of the bug in the N-1 check
14:40:37 sean-k-mooney1 but looking at the code you cannot start the nova conductor of you have n-2 comptue in the db not just n-2 compute running
14:40:43 gibi sean-k-mooney1: I still need to dig up more context
14:40:53 gibi sean-k-mooney1: I feel our original intention was to detect even stopped computes
14:40:57 gibi but I'm not sure
14:41:06 sean-k-mooney1 well that makes FFU impossible
14:41:10 gibi if FFU is broken then we have to do someting
14:42:11 sean-k-mooney1 well it depend the only escape hatch in the current code is to force all compute as down
14:42:24 gibi which is probably unintentional too :)
14:42:35 sean-k-mooney1 possible
14:43:09 gibi I'm wondering what would happen in an N could if an N-2 compute that was stopped now suddenly started up
14:43:17 sean-k-mooney1 in the ffucase if you update the contoler to n+3 then the condcutre wont be able to start until after you have updated all compute agent but they wont be abel to start cause the conductor is stopped
14:43:44 sean-k-mooney1 gibi: ideally the comptue should fail to start but the conductor should keep running
14:44:30 gibi sean-k-mooney1: OK, so now we probably fail backward as we not allow to start the new controller, but we should fail forward and only not allow the old compute to start up
14:45:34 sean-k-mooney1 no
14:45:46 sean-k-mooney1 we want the contoler to start we dont want the computes to start
14:45:54 sean-k-mooney1 until they are upgraded
14:46:17 gibi do we allow starting up an N-2 controller in an N cloud?
14:46:24 gibi or even an N-1 controller?
14:46:25 sean-k-mooney1 the contoler process, conductor in particalar need to be running so the compute can update its version in the db
14:46:50 sean-k-mooney1 gibi: today proably
14:46:55 sean-k-mooney1 which woudl be unsupported
14:47:03 sean-k-mooney1 since the contoler shoudl always be upgraded first
14:47:44 sean-k-mooney1 i mean if you clamp the rpc apis in the config you can praobly make it work but the assumtion we have is contoler are always newer or the same version as the computes
14:52:00 sean-k-mooney gibi: basicaly i think we shoudl either modify the check to only consider up compute servics or we shoudl move the check to the compute agent instead
14:52:41 gibi hehe, I had considered disabled old computes https://review.opendev.org/c/openstack/nova/+/738482/4..10/nova/service.py#430
14:52:42 sean-k-mooney or add a config option to opt out of the check for ffu
14:53:21 sean-k-mooney for what its worth i did some testing with devstack not so long ago
14:53:49 sean-k-mooney if i disable the version check you can mix wallaby contoler with train computes
14:54:57 sean-k-mooney to enable better upgrades we are considering supproting that as an upgrade path in the future downstream. not for train to wallaby but from wallaby to whatever our next release would be based on
14:55:03 gibi hm, a disable but running old compute can cause trouble in the cluster, a disable + force down compute cannot as we define force_down as "admin-promised-it-is-not-running"
14:55:24 gibi also based on the discussion in the original patch
14:55:44 sean-k-mooney yes force down means this is down nova just has not noticed yet i promise
14:56:05 gibi so at leat keeping the force_down make sense
14:56:26 gibi I'm not sure about the disable but running compute
14:56:36 sean-k-mooney well i was not thinking about disable
14:56:42 sean-k-mooney i was thinking more up=false
14:56:45 gibi ahh
14:56:55 sean-k-mooney so heartbeat is missing and we mark it as down
14:56:59 gibi so the compute definition exists in the db but the compute is timed out
14:57:19 sean-k-mooney ya presujmably because you stop it on the compute hosts as part of the upgrade
14:57:28 gibi it could be a network partition but it also can be a manual stop yes
14:58:10 sean-k-mooney well if it a network partiion it cant talk to the conductor :) but honestly i think the check is better done on the comptue side then the conductor side
14:58:33 gibi yeah but the check on the compute side only work if the compute is restarted
14:58:37 gibi if the compute is running
14:58:40 gibi then no check happens
14:58:45 sean-k-mooney that is fair
14:58:47 gibi that prevent the old compute to send nonsense to new conductor
14:59:05 gibi probably there is no perfect solution
14:59:23 gibi as from the controller perspective a timed out compute can mean multiple things
14:59:29 sean-k-mooney i mean for now we have the workaround of before you FFU stop all compute agaents and force themn down in the api
15:00:00 gibi yeah, and while that is an extra step it does not create an extra inconsistency
15:00:12 gibi you just need to double confirm that those computes are not coming back
15:00:16 gibi without a restart
15:00:59 sean-k-mooney honestly unless we have done a major rpc bump the comptue wont impact the conductor if they start
15:01:30 sean-k-mooney but yes if you use force down you shoudl fulfil its requiremetns
15:02:38 gibi hm an old compute cannot decided if it is too old, as the old code does not know what is the service version of the N+2 release
15:03:01 gibi except if we backport fixes to stable branches after the N
15:03:04 gibi N+2 release
15:03:23 sean-k-mooney well i was more thingink it would try to connect to the conductor and it would say its too old
15:03:32 sean-k-mooney but ya they cant really
15:04:10 gibi yeah the conductor could tell the compute to go away you are too old
15:04:35 sean-k-mooney it will do that if a new compute tries to regeister
15:17:13 bauzas gibi: I could be late for the meeting, could you please start it if so ?
15:17:20 bauzas the agenda is updated
15:18:08 bauzas reminder : nova meeting starts in 42 mins here
15:22:20 gibi bauzas: sure
15:47:02 gibi just a quick reminder nova meeting starts in 13 minutes here in the channel
15:57:03 bauzas I'm back
15:57:09 bauzas right on time
16:00:00 opendevmeet The meeting name has been set to 'nova'
16:00:00 opendevmeet Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:00 opendevmeet Meeting started Tue Jan 18 16:00:00 2022 UTC and is due to finish in 60 minutes. The chair is bauzas. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:00 bauzas #startmeeting nova
16:00:12 gibi o.
16:00:13 gibi o/
16:00:55 chateaulav \o
16:00:56 elodilles o/
16:01:19 bauzas #link https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting
16:01:53 bauzas ok, let's start
16:02:01 bauzas #topic Bugs (stuck/critical)
16:02:06 bauzas #info No Critical bug

Earlier   Later