Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2022-01-18
14:40:37 sean-k-mooney1 but looking at the code you cannot start the nova conductor of you have n-2 comptue in the db not just n-2 compute running
14:40:43 gibi sean-k-mooney1: I still need to dig up more context
14:40:53 gibi sean-k-mooney1: I feel our original intention was to detect even stopped computes
14:40:57 gibi but I'm not sure
14:41:06 sean-k-mooney1 well that makes FFU impossible
14:41:10 gibi if FFU is broken then we have to do someting
14:42:11 sean-k-mooney1 well it depend the only escape hatch in the current code is to force all compute as down
14:42:24 gibi which is probably unintentional too :)
14:42:35 sean-k-mooney1 possible
14:43:09 gibi I'm wondering what would happen in an N could if an N-2 compute that was stopped now suddenly started up
14:43:17 sean-k-mooney1 in the ffucase if you update the contoler to n+3 then the condcutre wont be able to start until after you have updated all compute agent but they wont be abel to start cause the conductor is stopped
14:43:44 sean-k-mooney1 gibi: ideally the comptue should fail to start but the conductor should keep running
14:44:30 gibi sean-k-mooney1: OK, so now we probably fail backward as we not allow to start the new controller, but we should fail forward and only not allow the old compute to start up
14:45:34 sean-k-mooney1 no
14:45:46 sean-k-mooney1 we want the contoler to start we dont want the computes to start
14:45:54 sean-k-mooney1 until they are upgraded
14:46:17 gibi do we allow starting up an N-2 controller in an N cloud?
14:46:24 gibi or even an N-1 controller?
14:46:25 sean-k-mooney1 the contoler process, conductor in particalar need to be running so the compute can update its version in the db
14:46:50 sean-k-mooney1 gibi: today proably
14:46:55 sean-k-mooney1 which woudl be unsupported
14:47:03 sean-k-mooney1 since the contoler shoudl always be upgraded first
14:47:44 sean-k-mooney1 i mean if you clamp the rpc apis in the config you can praobly make it work but the assumtion we have is contoler are always newer or the same version as the computes
14:52:00 sean-k-mooney gibi: basicaly i think we shoudl either modify the check to only consider up compute servics or we shoudl move the check to the compute agent instead
14:52:41 gibi hehe, I had considered disabled old computes https://review.opendev.org/c/openstack/nova/+/738482/4..10/nova/service.py#430
14:52:42 sean-k-mooney or add a config option to opt out of the check for ffu
14:53:21 sean-k-mooney for what its worth i did some testing with devstack not so long ago
14:53:49 sean-k-mooney if i disable the version check you can mix wallaby contoler with train computes
14:54:57 sean-k-mooney to enable better upgrades we are considering supproting that as an upgrade path in the future downstream. not for train to wallaby but from wallaby to whatever our next release would be based on
14:55:03 gibi hm, a disable but running old compute can cause trouble in the cluster, a disable + force down compute cannot as we define force_down as "admin-promised-it-is-not-running"
14:55:24 gibi also based on the discussion in the original patch
14:55:44 sean-k-mooney yes force down means this is down nova just has not noticed yet i promise
14:56:05 gibi so at leat keeping the force_down make sense
14:56:26 gibi I'm not sure about the disable but running compute
14:56:36 sean-k-mooney well i was not thinking about disable
14:56:42 sean-k-mooney i was thinking more up=false
14:56:45 gibi ahh
14:56:55 sean-k-mooney so heartbeat is missing and we mark it as down
14:56:59 gibi so the compute definition exists in the db but the compute is timed out
14:57:19 sean-k-mooney ya presujmably because you stop it on the compute hosts as part of the upgrade
14:57:28 gibi it could be a network partition but it also can be a manual stop yes
14:58:10 sean-k-mooney well if it a network partiion it cant talk to the conductor :) but honestly i think the check is better done on the comptue side then the conductor side
14:58:33 gibi yeah but the check on the compute side only work if the compute is restarted
14:58:37 gibi if the compute is running
14:58:40 gibi then no check happens
14:58:45 sean-k-mooney that is fair
14:58:47 gibi that prevent the old compute to send nonsense to new conductor
14:59:05 gibi probably there is no perfect solution
14:59:23 gibi as from the controller perspective a timed out compute can mean multiple things
14:59:29 sean-k-mooney i mean for now we have the workaround of before you FFU stop all compute agaents and force themn down in the api
15:00:00 gibi yeah, and while that is an extra step it does not create an extra inconsistency
15:00:12 gibi you just need to double confirm that those computes are not coming back
15:00:16 gibi without a restart
15:00:59 sean-k-mooney honestly unless we have done a major rpc bump the comptue wont impact the conductor if they start
15:01:30 sean-k-mooney but yes if you use force down you shoudl fulfil its requiremetns
15:02:38 gibi hm an old compute cannot decided if it is too old, as the old code does not know what is the service version of the N+2 release
15:03:01 gibi except if we backport fixes to stable branches after the N
15:03:04 gibi N+2 release
15:03:23 sean-k-mooney well i was more thingink it would try to connect to the conductor and it would say its too old
15:03:32 sean-k-mooney but ya they cant really
15:04:10 gibi yeah the conductor could tell the compute to go away you are too old
15:04:35 sean-k-mooney it will do that if a new compute tries to regeister
15:17:13 bauzas gibi: I could be late for the meeting, could you please start it if so ?
15:17:20 bauzas the agenda is updated
15:18:08 bauzas reminder : nova meeting starts in 42 mins here
15:22:20 gibi bauzas: sure
15:47:02 gibi just a quick reminder nova meeting starts in 13 minutes here in the channel
15:57:03 bauzas I'm back
15:57:09 bauzas right on time
16:00:00 bauzas #startmeeting nova
16:00:00 opendevmeet Meeting started Tue Jan 18 16:00:00 2022 UTC and is due to finish in 60 minutes. The chair is bauzas. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:00 opendevmeet Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:00 opendevmeet The meeting name has been set to 'nova'
16:00:12 gibi o.
16:00:13 gibi o/
16:00:55 chateaulav \o
16:00:56 elodilles o/
16:01:19 bauzas #link https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting
16:01:53 bauzas ok, let's start
16:02:01 bauzas #topic Bugs (stuck/critical)
16:02:06 bauzas #info No Critical bug
16:02:10 bauzas #info No Critical bug
16:02:15 bauzas #link https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New 32 new untriaged bugs (-4 since the last meeting)
16:02:20 bauzas #help Nova bug triage help is appreciated https://wiki.openstack.org/wiki/Nova/BugTriage
16:02:24 bauzas #link https://storyboard.openstack.org/#!/project/openstack/placement 23 open stories (-2 since the last meeting) in Storyboard for Placement
16:02:34 bauzas thanks to those who helped to triage
16:02:43 bauzas (I dunno who about placement :p)
16:04:06 gibi bauzas: can be that the any-trait spec merges closed the related ticjket
16:04:09 gibi I will check
16:04:33 bauzas maybe
16:04:35 bauzas anyway
16:04:42 bauzas any bug to want to discuss ?
16:05:21 bauzas looks not
16:05:27 bauzas #topic Gate status
16:05:32 bauzas #link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs
16:05:37 bauzas #link https://zuul.openstack.org/builds?project=openstack%2Fplacement&pipeline=periodic-weekly Placement periodic job status
16:05:43 bauzas #info Please look at the gate failures, file a bug, and add an elastic-recheck signature in the opendev/elastic-recheck repo (example: https://review.opendev.org/#/c/759967)
16:06:21 bauzas let's discuss about the centos8 job with elodilles in the stable branches topic
16:06:21 gibi (yepp the two missing placement ticket is the any-trait support, I will reopen them by creating an open task on them)
16:06:55 bauzas apart from this job, nothing to say

Earlier   Later