| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-01-07 | |||
| 16:34:26 | sean-k-mooney | woudl this be a spec/bluepirnt or bugfix? | |
| 16:35:01 | bauzas | hmmmm | |
| 16:35:06 | sean-k-mooney | im askign becasue of the time constratint for the first two options | |
| 16:35:12 | gibi | it has no API impact so I lean towards specless bp | |
| 16:35:36 | s3rj1k | specless from me if that counts :) | |
| 16:36:08 | sean-k-mooney | s3rj1k: input is always welcome :) | |
| 16:36:25 | gibi | sean-k-mooney: but you can formulate it as a bug as nova allows enabling the periodic in multiple schedulers today | |
| 16:36:27 | s3rj1k | (plus it somewhat covered in alternatives section of mentioned above spec) | |
| 16:36:55 | gibi | sean-k-mooney: without a safety net | |
| 16:37:15 | sean-k-mooney | ya so my perspecitve was. if we are takign the heviry weight approch of adding a distibuted lock manager that needed a spec because its complex and a large change | |
| 16:37:29 | bauzas | can't we just add a config option ? | |
| 16:37:39 | sean-k-mooney | if we do a very very small change to just od best effored leader election it might be a bug | |
| 16:37:45 | bauzas | I understand the reasoning, magically nova will elect a new leader | |
| 16:38:07 | sean-k-mooney | bauzas: we could also and a config opiton yes | |
| 16:38:08 | bauzas | but I'm afraid this periodic could silently run somewhere else without the op noticing it | |
| 16:38:14 | dansmith | omg, DLM? | |
| 16:38:31 | sean-k-mooney | bauzas: so you have to opt into the perodic | |
| 16:38:33 | bauzas | and you know, brainsplits and all the likes happen | |
| 16:38:40 | sean-k-mooney | by settign a config option to specify the interval today | |
| 16:38:51 | bauzas | that's my point | |
| 16:39:02 | sean-k-mooney | so this woudl just be a change to the behaiovr when you opt in | |
| 16:39:04 | bauzas | -1 disables the periodic IIRC | |
| 16:39:15 | sean-k-mooney | yep and i dont want to change that | |
| 16:39:24 | bauzas | but I wouldn't trust python for electing my leader | |
| 16:39:38 | bauzas | particularly the sorted command | |
| 16:39:42 | sean-k-mooney | so today the db enfoces that bad thing dont happen | |
| 16:39:53 | sean-k-mooney | you just get excptions in the log | |
| 16:40:14 | sean-k-mooney | if multiple race that is | |
| 16:40:22 | sean-k-mooney | which is annoying for operators | |
| 16:40:44 | dansmith | if you want to automate host discovery active-active, you can do that yourself with nova-manage and your own DLM right? | |
| 16:40:54 | sean-k-mooney | yes | |
| 16:41:04 | bauzas | yes, that's why I don't like that approach | |
| 16:41:12 | sean-k-mooney | that a pain in k8s however which s3rj1k cares about | |
| 16:41:18 | bauzas | you're considering that DLM is just not needed because sorted exists | |
| 16:41:25 | gibi | definitely having a full DLM in nova just for this is way overkill | |
| 16:41:39 | sean-k-mooney | yep which is why i did no liek the dlm/tooz approch | |
| 16:41:56 | bauzas | the SG API isn't also good at doing live healthchecks | |
| 16:42:19 | bauzas | there could be a reasonable amount in time where nova wouldn't see a node done | |
| 16:42:26 | bauzas | gone* | |
| 16:42:33 | sean-k-mooney | which is fine | |
| 16:42:37 | gibi | today if you run the periodic in multiple schedulers you burn power unnecessarily but you don't break nova DB just get exceptions. | |
| 16:42:48 | sean-k-mooney | we can miss runnign the perodic for a protracted period of time with out bad impacts | |
| 16:43:56 | bauzas | we could just let CONF.scheduler.discover_hosts_in_cells_interval be mutable and leave DLMs to manage nova-scheduler A/Ps instead of us | |
| 16:44:04 | sean-k-mooney | nope | |
| 16:44:09 | dansmith | There are also things we could do in nova that don't require a DLM if we really care | |
| 16:44:16 | sean-k-mooney | that does not work for k8s deployments which is the motivating usecase | |
| 16:44:31 | dansmith | like the periodic could say "am I the oldest-started nova-scheduler service that is currently up? If so, then run the discovery, if not, then don't" | |
| 16:44:34 | sean-k-mooney | dansmith: right im proposing not using a DLM | |
| 16:44:49 | dansmith | then only one of them would run it until you shut down the oldest and then the next one would do it.. sort of lazy-consensus election | |
| 16:44:52 | sean-k-mooney | dansmith: that basically what my patch does | |
| 16:44:53 | gibi | dansmith: exactly, that is very close to what sean-k-mooney proposes | |
| 16:44:59 | bauzas | dansmith: I don't like to trust the service group API for electing my leader | |
| 16:45:02 | dansmith | oh, then.. yeah :D | |
| 16:45:22 | dansmith | bauzas: it's not really trust, it's just optimization | |
| 16:45:30 | bauzas | sorted was one option, the oldest is another alternative | |
| 16:45:38 | gibi | bauzas: don't think about as leader election, it is more like limiting the number of discover host runs based on an input | |
| 16:45:51 | dansmith | I see sean-k-mooney's patch now, sorry I'm catching up | |
| 16:45:52 | sean-k-mooney | sorted is just for determinium | |
| 16:45:58 | sean-k-mooney | dansmith: no worries | |
| 16:46:15 | dansmith | determinism .. yeah, let's do something like that if we care, for sure | |
| 16:46:20 | gibi | bauzas: it could be a last run timestamp if we don't like age | |
| 16:46:55 | gibi | bauzas: but as we hav age we have a good set of value to find a single scheduler that min / max in that value to be the only one to do the work | |
| 16:46:57 | bauzas | well, | |
| 16:47:24 | sean-k-mooney | so for today what i really want to knwo is spec, bug or specless bluepritn so i know if we have till thurday or m3 to finalise the design and implamations | |
| 16:47:47 | bauzas | there are possibilities where a nova-scheduler could see itself being the oldest while another one, runnning 2 secs later could also see it as the oldest | |
| 16:47:59 | sean-k-mooney | that why im not using time | |
| 16:48:06 | sean-k-mooney | im sorting on the value of host | |
| 16:48:10 | bauzas | I know | |
| 16:48:18 | bauzas | but that's still the same | |
| 16:48:23 | dansmith | bauzas: sort by id, filter by up, that should be stable | |
| 16:48:33 | bauzas | some host could see itself as up while someother too, 1 sec later | |
| 16:48:55 | dansmith | if it does, then it's flapping from the view of the operator, which I think is not likely to be unnoticed | |
| 16:49:23 | sean-k-mooney | also people run with this in production without this protection or any kind fo arbitation today. | |
| 16:49:31 | bauzas | dansmith: if you are okay with operators unnoticing which nova-scheduler runs where, then ok | |
| 16:49:35 | sean-k-mooney | i.e. the just ignor the longs when there is a collioson | |
| 16:49:47 | dansmith | bauzas: which nova scheduler runs...host discovery? | |
| 16:49:53 | bauzas | yes | |
| 16:50:10 | sean-k-mooney | why would the care? | |
| 16:50:15 | dansmith | isn't that the point of this? instead of run-everywhere it's run-one-place, automatically managed? | |
| 16:50:28 | dansmith | right now, they can all run it in parallel all the time, if you care, but that's expensive | |
| 16:50:46 | gibi | ^^ yepp | |
| 16:50:51 | dansmith | I thought the point was to make it less expensive and automatically decide that (hopefully) only one does it each time.. seems fine to me | |
| 16:50:52 | bauzas | OK, then let's go for the proposal | |
| 16:50:57 | dansmith | worst case, two do it, no problem | |
| 16:51:10 | sean-k-mooney | yep ^ | |
| 16:51:12 | bauzas | well, you're right | |
| 16:51:22 | bauzas | two running at same time don't split brains | |
| 16:51:38 | bauzas | because of the periodic itself | |
| 16:51:41 | sean-k-mooney | they dont even cause an error if you have not added any hosts | |
| 16:51:52 | dansmith | yeah | |
| 16:51:58 | bauzas | okay, then to answer sean-k-mooney's question, I'm fine with it being specless | |
| 16:52:08 | bauzas | I actually would prefer it to be specless please | |
| 16:52:32 | bauzas | as I wouldn't want to step into DLM muds and leader elections | |
| 16:52:40 | sean-k-mooney | ok i think ill file a new bluepritn for ti and update the description | |
| 16:52:54 | sean-k-mooney | if we are ok with that are we ok to appove that async | |
| 16:52:58 | bauzas | sure | |
| 16:53:08 | sean-k-mooney | ill do it now and ping outside fo the meeting | |
| 16:53:36 | bauzas | #agreed https://review.opendev.org/c/openstack/nova/+/938523 can be filed as a specless blueprint | |
| 16:53:54 | s3rj1k | sean-k-mooney: thanks for spending time on this | |
| 16:53:55 | bauzas | cool thanks | |