Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2025-01-07
16:27:29 bauzas excellenbt
16:27:34 bauzas thanks for checking
16:27:46 bauzas are we then ok with the design ?
16:28:05 sean-k-mooney i think so. i capture dmost of my feedback inthe blueprint already
16:28:29 sean-k-mooney so im more or less ok with proceedign to the implemenation reivew based on that design
16:29:35 bauzas ++
16:29:58 bauzas okay, then let's approve it as specless
16:30:14 bauzas #approved https://blueprints.launchpad.net/nova/+spec/image-metadata-props-weigher approved as specless
16:30:20 bauzas doh
16:30:35 bauzas #agreed https://blueprints.launchpad.net/nova/+spec/image-metadata-props-weigher approved as specless
16:30:39 bauzas that was it for me
16:30:44 bauzas anything else to mention ?
16:30:51 sean-k-mooney if there is time/enerty i have one topic
16:31:09 bauzas I don't have energy but we have time
16:31:36 sean-k-mooney https://blueprints.launchpad.net/nova/+spec/host-discovery-distributed-lock
16:31:57 sean-k-mooney we talked about this at the ptg
16:32:06 sean-k-mooney and htere is a related spec https://review.opendev.org/c/openstack/nova-specs/+/936389
16:32:27 sean-k-mooney tldr they woud like to be able to enabel the discover host periodic on multiple schduler at a time
16:32:38 sean-k-mooney so i dont actully like the proposal as written
16:32:49 sean-k-mooney but i did a short poc of one of the alternitives
16:32:58 sean-k-mooney https://review.opendev.org/c/openstack/nova/+/938523
16:33:15 sean-k-mooney i don think this could be a reasonable minor imporvment
16:33:39 sean-k-mooney the change is basically just this https://review.opendev.org/c/openstack/nova/+/938523/2/nova/scheduler/manager.py#113
16:34:16 sean-k-mooney what i wanted to know is if we took this supper simpler approch of doing leader election in the perodic and returnign if not the leader
16:34:26 sean-k-mooney woudl this be a spec/bluepirnt or bugfix?
16:35:01 bauzas hmmmm
16:35:06 sean-k-mooney im askign becasue of the time constratint for the first two options
16:35:12 gibi it has no API impact so I lean towards specless bp
16:35:36 s3rj1k specless from me if that counts :)
16:36:08 sean-k-mooney s3rj1k: input is always welcome :)
16:36:25 gibi sean-k-mooney: but you can formulate it as a bug as nova allows enabling the periodic in multiple schedulers today
16:36:27 s3rj1k (plus it somewhat covered in alternatives section of mentioned above spec)
16:36:55 gibi sean-k-mooney: without a safety net
16:37:15 sean-k-mooney ya so my perspecitve was. if we are takign the heviry weight approch of adding a distibuted lock manager that needed a spec because its complex and a large change
16:37:29 bauzas can't we just add a config option ?
16:37:39 sean-k-mooney if we do a very very small change to just od best effored leader election it might be a bug
16:37:45 bauzas I understand the reasoning, magically nova will elect a new leader
16:38:07 sean-k-mooney bauzas: we could also and a config opiton yes
16:38:08 bauzas but I'm afraid this periodic could silently run somewhere else without the op noticing it
16:38:14 dansmith omg, DLM?
16:38:31 sean-k-mooney bauzas: so you have to opt into the perodic
16:38:33 bauzas and you know, brainsplits and all the likes happen
16:38:40 sean-k-mooney by settign a config option to specify the interval today
16:38:51 bauzas that's my point
16:39:02 sean-k-mooney so this woudl just be a change to the behaiovr when you opt in
16:39:04 bauzas -1 disables the periodic IIRC
16:39:15 sean-k-mooney yep and i dont want to change that
16:39:24 bauzas but I wouldn't trust python for electing my leader
16:39:38 bauzas particularly the sorted command
16:39:42 sean-k-mooney so today the db enfoces that bad thing dont happen
16:39:53 sean-k-mooney you just get excptions in the log
16:40:14 sean-k-mooney if multiple race that is
16:40:22 sean-k-mooney which is annoying for operators
16:40:44 dansmith if you want to automate host discovery active-active, you can do that yourself with nova-manage and your own DLM right?
16:40:54 sean-k-mooney yes
16:41:04 bauzas yes, that's why I don't like that approach
16:41:12 sean-k-mooney that a pain in k8s however which s3rj1k cares about
16:41:18 bauzas you're considering that DLM is just not needed because sorted exists
16:41:25 gibi definitely having a full DLM in nova just for this is way overkill
16:41:39 sean-k-mooney yep which is why i did no liek the dlm/tooz approch
16:41:56 bauzas the SG API isn't also good at doing live healthchecks
16:42:19 bauzas there could be a reasonable amount in time where nova wouldn't see a node done
16:42:26 bauzas gone*
16:42:33 sean-k-mooney which is fine
16:42:37 gibi today if you run the periodic in multiple schedulers you burn power unnecessarily but you don't break nova DB just get exceptions.
16:42:48 sean-k-mooney we can miss runnign the perodic for a protracted period of time with out bad impacts
16:43:56 bauzas we could just let CONF.scheduler.discover_hosts_in_cells_interval be mutable and leave DLMs to manage nova-scheduler A/Ps instead of us
16:44:04 sean-k-mooney nope
16:44:09 dansmith There are also things we could do in nova that don't require a DLM if we really care
16:44:16 sean-k-mooney that does not work for k8s deployments which is the motivating usecase
16:44:31 dansmith like the periodic could say "am I the oldest-started nova-scheduler service that is currently up? If so, then run the discovery, if not, then don't"
16:44:34 sean-k-mooney dansmith: right im proposing not using a DLM
16:44:49 dansmith then only one of them would run it until you shut down the oldest and then the next one would do it.. sort of lazy-consensus election
16:44:52 sean-k-mooney dansmith: that basically what my patch does
16:44:53 gibi dansmith: exactly, that is very close to what sean-k-mooney proposes
16:44:59 bauzas dansmith: I don't like to trust the service group API for electing my leader
16:45:02 dansmith oh, then.. yeah :D
16:45:22 dansmith bauzas: it's not really trust, it's just optimization
16:45:30 bauzas sorted was one option, the oldest is another alternative
16:45:38 gibi bauzas: don't think about as leader election, it is more like limiting the number of discover host runs based on an input
16:45:51 dansmith I see sean-k-mooney's patch now, sorry I'm catching up
16:45:52 sean-k-mooney sorted is just for determinium
16:45:58 sean-k-mooney dansmith: no worries
16:46:15 dansmith determinism .. yeah, let's do something like that if we care, for sure
16:46:20 gibi bauzas: it could be a last run timestamp if we don't like age
16:46:55 gibi bauzas: but as we hav age we have a good set of value to find a single scheduler that min / max in that value to be the only one to do the work
16:46:57 bauzas well,
16:47:24 sean-k-mooney so for today what i really want to knwo is spec, bug or specless bluepritn so i know if we have till thurday or m3 to finalise the design and implamations
16:47:47 bauzas there are possibilities where a nova-scheduler could see itself being the oldest while another one, runnning 2 secs later could also see it as the oldest
16:47:59 sean-k-mooney that why im not using time
16:48:06 sean-k-mooney im sorting on the value of host
16:48:10 bauzas I know
16:48:18 bauzas but that's still the same
16:48:23 dansmith bauzas: sort by id, filter by up, that should be stable
16:48:33 bauzas some host could see itself as up while someother too, 1 sec later
16:48:55 dansmith if it does, then it's flapping from the view of the operator, which I think is not likely to be unnoticed
16:49:23 sean-k-mooney also people run with this in production without this protection or any kind fo arbitation today.
16:49:31 bauzas dansmith: if you are okay with operators unnoticing which nova-scheduler runs where, then ok
16:49:35 sean-k-mooney i.e. the just ignor the longs when there is a collioson
16:49:47 dansmith bauzas: which nova scheduler runs...host discovery?

Earlier   Later