Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2025-01-07
16:39:15 sean-k-mooney yep and i dont want to change that
16:39:24 bauzas but I wouldn't trust python for electing my leader
16:39:38 bauzas particularly the sorted command
16:39:42 sean-k-mooney so today the db enfoces that bad thing dont happen
16:39:53 sean-k-mooney you just get excptions in the log
16:40:14 sean-k-mooney if multiple race that is
16:40:22 sean-k-mooney which is annoying for operators
16:40:44 dansmith if you want to automate host discovery active-active, you can do that yourself with nova-manage and your own DLM right?
16:40:54 sean-k-mooney yes
16:41:04 bauzas yes, that's why I don't like that approach
16:41:12 sean-k-mooney that a pain in k8s however which s3rj1k cares about
16:41:18 bauzas you're considering that DLM is just not needed because sorted exists
16:41:25 gibi definitely having a full DLM in nova just for this is way overkill
16:41:39 sean-k-mooney yep which is why i did no liek the dlm/tooz approch
16:41:56 bauzas the SG API isn't also good at doing live healthchecks
16:42:19 bauzas there could be a reasonable amount in time where nova wouldn't see a node done
16:42:26 bauzas gone*
16:42:33 sean-k-mooney which is fine
16:42:37 gibi today if you run the periodic in multiple schedulers you burn power unnecessarily but you don't break nova DB just get exceptions.
16:42:48 sean-k-mooney we can miss runnign the perodic for a protracted period of time with out bad impacts
16:43:56 bauzas we could just let CONF.scheduler.discover_hosts_in_cells_interval be mutable and leave DLMs to manage nova-scheduler A/Ps instead of us
16:44:04 sean-k-mooney nope
16:44:09 dansmith There are also things we could do in nova that don't require a DLM if we really care
16:44:16 sean-k-mooney that does not work for k8s deployments which is the motivating usecase
16:44:31 dansmith like the periodic could say "am I the oldest-started nova-scheduler service that is currently up? If so, then run the discovery, if not, then don't"
16:44:34 sean-k-mooney dansmith: right im proposing not using a DLM
16:44:49 dansmith then only one of them would run it until you shut down the oldest and then the next one would do it.. sort of lazy-consensus election
16:44:52 sean-k-mooney dansmith: that basically what my patch does
16:44:53 gibi dansmith: exactly, that is very close to what sean-k-mooney proposes
16:44:59 bauzas dansmith: I don't like to trust the service group API for electing my leader
16:45:02 dansmith oh, then.. yeah :D
16:45:22 dansmith bauzas: it's not really trust, it's just optimization
16:45:30 bauzas sorted was one option, the oldest is another alternative
16:45:38 gibi bauzas: don't think about as leader election, it is more like limiting the number of discover host runs based on an input
16:45:51 dansmith I see sean-k-mooney's patch now, sorry I'm catching up
16:45:52 sean-k-mooney sorted is just for determinium
16:45:58 sean-k-mooney dansmith: no worries
16:46:15 dansmith determinism .. yeah, let's do something like that if we care, for sure
16:46:20 gibi bauzas: it could be a last run timestamp if we don't like age
16:46:55 gibi bauzas: but as we hav age we have a good set of value to find a single scheduler that min / max in that value to be the only one to do the work
16:46:57 bauzas well,
16:47:24 sean-k-mooney so for today what i really want to knwo is spec, bug or specless bluepritn so i know if we have till thurday or m3 to finalise the design and implamations
16:47:47 bauzas there are possibilities where a nova-scheduler could see itself being the oldest while another one, runnning 2 secs later could also see it as the oldest
16:47:59 sean-k-mooney that why im not using time
16:48:06 sean-k-mooney im sorting on the value of host
16:48:10 bauzas I know
16:48:18 bauzas but that's still the same
16:48:23 dansmith bauzas: sort by id, filter by up, that should be stable
16:48:33 bauzas some host could see itself as up while someother too, 1 sec later
16:48:55 dansmith if it does, then it's flapping from the view of the operator, which I think is not likely to be unnoticed
16:49:23 sean-k-mooney also people run with this in production without this protection or any kind fo arbitation today.
16:49:31 bauzas dansmith: if you are okay with operators unnoticing which nova-scheduler runs where, then ok
16:49:35 sean-k-mooney i.e. the just ignor the longs when there is a collioson
16:49:47 dansmith bauzas: which nova scheduler runs...host discovery?
16:49:53 bauzas yes
16:50:10 sean-k-mooney why would the care?
16:50:15 dansmith isn't that the point of this? instead of run-everywhere it's run-one-place, automatically managed?
16:50:28 dansmith right now, they can all run it in parallel all the time, if you care, but that's expensive
16:50:46 gibi ^^ yepp
16:50:51 dansmith I thought the point was to make it less expensive and automatically decide that (hopefully) only one does it each time.. seems fine to me
16:50:52 bauzas OK, then let's go for the proposal
16:50:57 dansmith worst case, two do it, no problem
16:51:10 sean-k-mooney yep ^
16:51:12 bauzas well, you're right
16:51:22 bauzas two running at same time don't split brains
16:51:38 bauzas because of the periodic itself
16:51:41 sean-k-mooney they dont even cause an error if you have not added any hosts
16:51:52 dansmith yeah
16:51:58 bauzas okay, then to answer sean-k-mooney's question, I'm fine with it being specless
16:52:08 bauzas I actually would prefer it to be specless please
16:52:32 bauzas as I wouldn't want to step into DLM muds and leader elections
16:52:40 sean-k-mooney ok i think ill file a new bluepritn for ti and update the description
16:52:54 sean-k-mooney if we are ok with that are we ok to appove that async
16:52:58 bauzas sure
16:53:08 sean-k-mooney ill do it now and ping outside fo the meeting
16:53:36 bauzas #agreed https://review.opendev.org/c/openstack/nova/+/938523 can be filed as a specless blueprint
16:53:54 s3rj1k sean-k-mooney: thanks for spending time on this
16:53:55 bauzas cool thanks
16:54:30 sean-k-mooney s3rj1k: no worreis i ment to do it before going on pto i just didnt finish hacking on it until i got back on monday
16:54:54 sean-k-mooney ok thats if from me
16:55:05 sean-k-mooney s/if/it/
16:55:52 bauzas cool
16:55:55 bauzas thanks all then
16:55:59 bauzas #endmeeting
16:55:59 opendevmeet Meeting ended Tue Jan 7 16:55:58 2025 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
16:55:59 opendevmeet Minutes: https://meetings.opendev.org/meetings/nova/2025/nova.2025-01-07-16.01.html
16:55:59 opendevmeet Minutes (text): https://meetings.opendev.org/meetings/nova/2025/nova.2025-01-07-16.01.txt
16:55:59 opendevmeet Log: https://meetings.opendev.org/meetings/nova/2025/nova.2025-01-07-16.01.log.html
16:56:27 elodilles thanks o/
16:56:43 s3rj1k thanks
16:58:55 gibi thanks folks
17:22:45 sean-k-mooney s3rj1k: bauzas: i think https://blueprints.launchpad.net/nova/+spec/distributed-host-discovery covers it
17:26:41 s3rj1k yea, looks good
17:40:24 luvn hey guys! in openstack, is it possible — through some workaround ­— to detach a boot volume to extend and/or retype it?
17:41:40 luvn i found an old spec related to this but as far as i know it never moved forward — https://specs.openstack.org/openstack/nova-specs/specs/stein/approved/detach-boot-volume.html
17:42:05 luvn so i was just wondering if we have a workaround for that
17:45:22 dansmith I want to say you might be able to shelve/offload a BFV instance and take action on the volume, but don't quote me
17:51:14 sean-k-mooney dansmith: i think when shleved its in the reserved state, im not sure if cinder will allow exented then but perhaps.
17:51:33 sean-k-mooney luvn: but no we dont allow detach of the root volume for bfv guests
17:51:36 dansmith yeah idk, but I think it's detached from the host which definitely keeps it pinned

Earlier   Later