Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2024-11-27
19:07:38 opendevreview Stephen Finucane proposed openstack/nova master: api: Add new, simpler api_version decorator https://review.opendev.org/c/openstack/nova/+/936367
19:07:38 opendevreview Stephen Finucane proposed openstack/nova master: api: Only run format checks on strings https://review.opendev.org/c/openstack/nova/+/936368
19:07:39 opendevreview Stephen Finucane proposed openstack/nova master: api: Simplify parameter types https://review.opendev.org/c/openstack/nova/+/936369
19:07:40 opendevreview Stephen Finucane proposed openstack/nova master: tests: Ensure all APIs have a response body schema https://review.opendev.org/c/openstack/nova/+/924598
19:07:43 s3rj1k no reason to force CLI with cronjobs
19:08:46 sean-k-mooney what i was orginally thinking was to use a Rendezvous hash to select the leader form the active schduler and only run the perodic form that one
19:09:31 sean-k-mooney that way we dont actully need an external depency (distibuted lock manager) or any addtional rpcs ectra
19:09:40 sean-k-mooney im just not sure if that will work with our current system
19:10:25 sean-k-mooney a concern with a dlm is makeing sure the lock is released if any of the pods get restarted for example
19:10:42 s3rj1k btw, is there a formal doc on tooz deprecation?
19:10:53 sean-k-mooney its not deprecated as a lib
19:11:08 sean-k-mooney so the only usage of it in nova is the ironic driver and we deprecated the fature that uses it in that
19:11:25 s3rj1k so for nova removal in this case
19:11:33 sean-k-mooney currently the ironic drivcer can use manual sharding or a dynmic hashring(provided by tooz)
19:12:08 sean-k-mooney the dynmaic hash ring is deprecated for removal which would have removed tooz as a side effect
19:15:12 sean-k-mooney s3rj1k: each schduler is a memebr of a service group
19:15:14 sean-k-mooney https://github.com/openstack/nova/blob/master/nova/scheduler/manager.py#L69
19:15:57 sean-k-mooney there are 2 service group drivers, the db and memcache
19:17:18 sean-k-mooney im wondering if we can reuse that to do leader elelction and then have only one of the member run the perodic if its currently the leader
19:19:31 sean-k-mooney the problem is the curernt service group api does nto really ahve a way to do that
19:20:04 sean-k-mooney it has a way to test if a member of a service group is up
19:20:16 sean-k-mooney but not a way to list the memebr and or select a leader
19:22:19 opendevreview ribaudr proposed openstack/nova-specs master: WIP: Allow vfio gpu support with Vendor-Specific VFIO framework https://review.opendev.org/c/openstack/nova-specs/+/936407
19:43:21 opendevreview Merged openstack/nova stable/2024.1: Handle neutron-client conflict https://review.opendev.org/c/openstack/nova/+/927732
19:45:04 s3rj1k > but not a way to list the memebr and or select a leader - so that would mean that we would firstly need to extend that and after that try to use that
19:51:20 sean-k-mooney for the db dirver its pretty simple
19:51:36 sean-k-mooney that basically a tweaked version fo https://github.com/openstack/nova/blob/master/nova/db/main/api.py#L492-L503
19:51:43 sean-k-mooney i.e. get service by binary
19:52:52 sean-k-mooney in this case of memcache its a littel more annoying
19:53:39 sean-k-mooney we coudl just ignore the service goups and just use the service api
19:53:53 sean-k-mooney *service db tables not api
19:54:51 sean-k-mooney basically im trying to see what whoudl result in the minium number of addtional rpc/db/rest calls to enable
19:58:19 s3rj1k > minium number of addtional rpc/db/rest calls - solution by documentation :)
20:05:56 sean-k-mooney maybe, or https://paste.opendev.org/show/bFFLh0pVeBZAOYXoNjKs/
20:06:58 sean-k-mooney we woudl just add that here https://github.com/openstack/nova/blob/master/nova/scheduler/manager.py#L111
20:07:50 s3rj1k hmm, if that works that would be nice, can be combined with docs extension
20:07:57 sean-k-mooney sure
20:08:43 sean-k-mooney basicaly that one addtional db call on each invocation fo the perodic. but its a net win because instead of 3 of the perodic doing the work in paralel only 1 of them will execute the rest of the code
20:08:50 sean-k-mooney so overall that shoudl reduce the db load
20:09:03 s3rj1k in that case warning about concurrent CLIs and extension to how periodics work
20:10:08 s3rj1k one thing, in case of AA how would leader election work? like fallback?
20:10:35 sean-k-mooney so the heartbeat woud expire on the failed schduler
20:10:53 s3rj1k and alive one would be elected?
20:10:54 sean-k-mooney and then then it would be elimistaed for the set and we would select a new leader
20:11:19 sean-k-mooney right so if we sort on the uuid then that would be stable or the hostname
20:11:38 sean-k-mooney so if one host/pod dies then it will get remvoed form the set and we just take the first form the list
20:12:14 sean-k-mooney its not ture leader election in that when the down host comes back it takes over as the leader
20:12:28 s3rj1k this is fine
20:12:44 sean-k-mooney where as normally you woudl only do an election when then current wone extits but ya it should work for this case
20:13:06 s3rj1k we can sort by uuid + uptime
20:13:17 s3rj1k if we can get time info
20:13:35 sean-k-mooney well we dont want it really to change often
20:13:58 sean-k-mooney and that woudl be unstable ie.. tow diffent schduler check 1 second apprart
20:14:10 sean-k-mooney that coudl lead to both thinking the other is the leader
20:14:21 s3rj1k I mean prefer ones that run longer
20:14:25 sean-k-mooney but i dotn think the unbiase dload is a problem in this case
20:15:00 sean-k-mooney maybe i would keep it simple initaly
20:15:09 sean-k-mooney we coudl alway teak it later if needed
20:15:15 s3rj1k but simple uuid sort would work as this is not very critical place
20:16:40 sean-k-mooney by the way up is not directly a property on the service (we hsould make it one) we have a function that takes the last time we saw it and a config value to decied if its still up
20:16:56 sean-k-mooney but that pretty easy to handel
20:17:46 sean-k-mooney we do that computation and systisze the up value in teh api when we do a service list
20:20:39 sean-k-mooney basically we have https://github.com/openstack/nova/blob/master/nova/objects/service.py#L316
20:20:56 sean-k-mooney which tells use when the last heartbet was recived
20:21:15 sean-k-mooney and we jus tneed to compare that to the interval and allow missed heatbeats
20:22:07 sean-k-mooney or actully we can jsut check https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.service_down_time driectly
20:22:33 s3rj1k I see report_count in there
20:23:51 sean-k-mooney up = datetime.now() - CONF.service_down_time < last_seen_up
20:24:06 sean-k-mooney something like ^ would work
20:24:29 s3rj1k report_count is per-service right?
20:24:37 sean-k-mooney yes
20:25:07 s3rj1k so if thing goes down and up it gets nulled and count get started over?
20:25:25 sean-k-mooney i dont recall of the top of my head
20:25:44 sean-k-mooney in this context we can ignore i ti think
20:26:03 sean-k-mooney anyway i better finish for today
20:26:13 s3rj1k so basically there are solution how to make that selection more clever
20:26:27 sean-k-mooney yep if needed
20:26:27 s3rj1k thanks for you input
20:26:47 sean-k-mooney if we were trying to distibute load
20:26:54 sean-k-mooney we might want a more clever solution
20:27:08 sean-k-mooney but since we are trying to make sure only one runs i dont think that isneeded
20:27:44 sean-k-mooney we might want a minium uptime before the leader can change
20:27:58 sean-k-mooney i.e. double the perodic interval or somthing like that
20:28:25 sean-k-mooney just to prevent issues on start up
20:28:47 s3rj1k even of we miss, next run will fix that
20:28:49 sean-k-mooney or perhaps double the service downtime interval
20:29:02 s3rj1k s/of/if/
20:29:24 sean-k-mooney but yes i think we can do this in a lock free way and it will self heal
20:29:39 sean-k-mooney actully droping now o/
20:30:00 s3rj1k have a good night
#openstack-nova - 2024-11-28
04:28:46 opendevreview Nisha Brahmankar proposed openstack/nova master: Retain port binding profile after live migration https://review.opendev.org/c/openstack/nova/+/936338
10:24:23 opendevreview Stephen Finucane proposed openstack/nova master: api: Add new, simpler api_version decorator https://review.opendev.org/c/openstack/nova/+/936367
10:24:23 opendevreview Stephen Finucane proposed openstack/nova master: api: Only run format checks on strings https://review.opendev.org/c/openstack/nova/+/936368
10:24:24 opendevreview Stephen Finucane proposed openstack/nova master: api: Simplify parameter types https://review.opendev.org/c/openstack/nova/+/936369
10:24:24 opendevreview Stephen Finucane proposed openstack/nova master: tests: Ensure all APIs have a response body schema https://review.opendev.org/c/openstack/nova/+/924598
11:17:53 zigo dansmith: Hi there! Got a question for you. I'm testing db upgrades, and populate_instance_compute_id is taking like forever.
11:17:53 zigo I wonder, what is this about? Your commit header says it's for "deleted, shelved, never scheduled or otherwise on a compute host that isn't updatating it", I don't really get it.
11:17:53 zigo Also, I have many warnings about: "Unable to migrate instance because host None with node None not found: nova.exception.ComputeHostNotFound: Compute host None could not be found.". Should I be worried?
11:23:27 zigo (of course, I'm not expecting a reply before you wake up ... :P )
13:28:53 opendevreview Merged openstack/nova master: Add share_info parameter to reboot method for each driver (driver part) https://review.opendev.org/c/openstack/nova/+/854823

Earlier   Later