| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-11-27 | |||
| 19:00:01 | s3rj1k | sean-k-mooney: Can you take a quick look on that Spec CR? https://review.opendev.org/c/openstack/nova-specs/+/936389 | |
| 19:00:44 | s3rj1k | I've put also some of the alternatives that you've mentioned in there | |
| 19:01:10 | sean-k-mooney | sure | |
| 19:01:17 | s3rj1k | thanks | |
| 19:01:22 | sean-k-mooney | im not convicd you primay propsol is what we shoudl do | |
| 19:01:27 | sean-k-mooney | btu we can discuss it on the spec | |
| 19:02:13 | s3rj1k | at least I hope to have this finalized in any way) best to have this solved rather than ignoring the issue | |
| 19:02:36 | s3rj1k | so any way works as long as we go forward :) | |
| 19:02:39 | sean-k-mooney | well right now you want to add a distribute lock manager and annoter usage of tooz | |
| 19:02:52 | sean-k-mooney | where as we ahve dicussed how to remove tooz form teh code base | |
| 19:03:03 | sean-k-mooney | at least twice in the last year or two | |
| 19:03:21 | s3rj1k | just like in original rfe, tooz was there yes | |
| 19:03:25 | sean-k-mooney | so we may be able to proced with this but your askign for use to reverse cources on that | |
| 19:03:58 | s3rj1k | we can replace tooz I guess | |
| 19:04:02 | sean-k-mooney | ok jsut so your aware that thsi is a pretty contiversiol approch | |
| 19:04:17 | sean-k-mooney | we generally did not want to ahve a distribute lock manager in nova at all | |
| 19:04:39 | s3rj1k | that's why I invested time in alternatives) | |
| 19:05:04 | sean-k-mooney | the main concern with tooz by the way is its overall mantance | |
| 19:05:37 | sean-k-mooney | what migh be simpler to do is some form of leader elelction | |
| 19:05:44 | sean-k-mooney | rather then a distributed lock | |
| 19:06:14 | s3rj1k | that won't cover CLI | |
| 19:06:29 | sean-k-mooney | well to be clear it does not have too | |
| 19:06:40 | sean-k-mooney | currently the sideefect does nto break anything | |
| 19:07:08 | sean-k-mooney | the db enforces consitenty so the worst case side effecti is the command exits with an error but it does not break anything | |
| 19:07:17 | s3rj1k | yea, I guess so, if periodics can be safely fixed for concurrency | |
| 19:07:18 | sean-k-mooney | so that ok on the cli | |
| 19:07:36 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Allow min/max_version arguments to response https://review.opendev.org/c/openstack/nova/+/936364 | |
| 19:07:36 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Allow min/max_version arguments to expected_errors https://review.opendev.org/c/openstack/nova/+/936363 | |
| 19:07:37 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Stop using wsgi.Controller.api_version to switch between API versions https://review.opendev.org/c/openstack/nova/+/936366 | |
| 19:07:37 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Adjust validation helpers for a single-method future https://review.opendev.org/c/openstack/nova/+/936365 | |
| 19:07:38 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Only run format checks on strings https://review.opendev.org/c/openstack/nova/+/936368 | |
| 19:07:38 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Add new, simpler api_version decorator https://review.opendev.org/c/openstack/nova/+/936367 | |
| 19:07:39 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Simplify parameter types https://review.opendev.org/c/openstack/nova/+/936369 | |
| 19:07:40 | opendevreview | Stephen Finucane proposed openstack/nova master: tests: Ensure all APIs have a response body schema https://review.opendev.org/c/openstack/nova/+/924598 | |
| 19:07:43 | s3rj1k | no reason to force CLI with cronjobs | |
| 19:08:46 | sean-k-mooney | what i was orginally thinking was to use a Rendezvous hash to select the leader form the active schduler and only run the perodic form that one | |
| 19:09:31 | sean-k-mooney | that way we dont actully need an external depency (distibuted lock manager) or any addtional rpcs ectra | |
| 19:09:40 | sean-k-mooney | im just not sure if that will work with our current system | |
| 19:10:25 | sean-k-mooney | a concern with a dlm is makeing sure the lock is released if any of the pods get restarted for example | |
| 19:10:42 | s3rj1k | btw, is there a formal doc on tooz deprecation? | |
| 19:10:53 | sean-k-mooney | its not deprecated as a lib | |
| 19:11:08 | sean-k-mooney | so the only usage of it in nova is the ironic driver and we deprecated the fature that uses it in that | |
| 19:11:25 | s3rj1k | so for nova removal in this case | |
| 19:11:33 | sean-k-mooney | currently the ironic drivcer can use manual sharding or a dynmic hashring(provided by tooz) | |
| 19:12:08 | sean-k-mooney | the dynmaic hash ring is deprecated for removal which would have removed tooz as a side effect | |
| 19:15:12 | sean-k-mooney | s3rj1k: each schduler is a memebr of a service group | |
| 19:15:14 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/scheduler/manager.py#L69 | |
| 19:15:57 | sean-k-mooney | there are 2 service group drivers, the db and memcache | |
| 19:17:18 | sean-k-mooney | im wondering if we can reuse that to do leader elelction and then have only one of the member run the perodic if its currently the leader | |
| 19:19:31 | sean-k-mooney | the problem is the curernt service group api does nto really ahve a way to do that | |
| 19:20:04 | sean-k-mooney | it has a way to test if a member of a service group is up | |
| 19:20:16 | sean-k-mooney | but not a way to list the memebr and or select a leader | |
| 19:22:19 | opendevreview | ribaudr proposed openstack/nova-specs master: WIP: Allow vfio gpu support with Vendor-Specific VFIO framework https://review.opendev.org/c/openstack/nova-specs/+/936407 | |
| 19:43:21 | opendevreview | Merged openstack/nova stable/2024.1: Handle neutron-client conflict https://review.opendev.org/c/openstack/nova/+/927732 | |
| 19:45:04 | s3rj1k | > but not a way to list the memebr and or select a leader - so that would mean that we would firstly need to extend that and after that try to use that | |
| 19:51:20 | sean-k-mooney | for the db dirver its pretty simple | |
| 19:51:36 | sean-k-mooney | that basically a tweaked version fo https://github.com/openstack/nova/blob/master/nova/db/main/api.py#L492-L503 | |
| 19:51:43 | sean-k-mooney | i.e. get service by binary | |
| 19:52:52 | sean-k-mooney | in this case of memcache its a littel more annoying | |
| 19:53:39 | sean-k-mooney | we coudl just ignore the service goups and just use the service api | |
| 19:53:53 | sean-k-mooney | *service db tables not api | |
| 19:54:51 | sean-k-mooney | basically im trying to see what whoudl result in the minium number of addtional rpc/db/rest calls to enable | |
| 19:58:19 | s3rj1k | > minium number of addtional rpc/db/rest calls - solution by documentation :) | |
| 20:05:56 | sean-k-mooney | maybe, or https://paste.opendev.org/show/bFFLh0pVeBZAOYXoNjKs/ | |
| 20:06:58 | sean-k-mooney | we woudl just add that here https://github.com/openstack/nova/blob/master/nova/scheduler/manager.py#L111 | |
| 20:07:50 | s3rj1k | hmm, if that works that would be nice, can be combined with docs extension | |
| 20:07:57 | sean-k-mooney | sure | |
| 20:08:43 | sean-k-mooney | basicaly that one addtional db call on each invocation fo the perodic. but its a net win because instead of 3 of the perodic doing the work in paralel only 1 of them will execute the rest of the code | |
| 20:08:50 | sean-k-mooney | so overall that shoudl reduce the db load | |
| 20:09:03 | s3rj1k | in that case warning about concurrent CLIs and extension to how periodics work | |
| 20:10:08 | s3rj1k | one thing, in case of AA how would leader election work? like fallback? | |
| 20:10:35 | sean-k-mooney | so the heartbeat woud expire on the failed schduler | |
| 20:10:53 | s3rj1k | and alive one would be elected? | |
| 20:10:54 | sean-k-mooney | and then then it would be elimistaed for the set and we would select a new leader | |
| 20:11:19 | sean-k-mooney | right so if we sort on the uuid then that would be stable or the hostname | |
| 20:11:38 | sean-k-mooney | so if one host/pod dies then it will get remvoed form the set and we just take the first form the list | |
| 20:12:14 | sean-k-mooney | its not ture leader election in that when the down host comes back it takes over as the leader | |
| 20:12:28 | s3rj1k | this is fine | |
| 20:12:44 | sean-k-mooney | where as normally you woudl only do an election when then current wone extits but ya it should work for this case | |
| 20:13:06 | s3rj1k | we can sort by uuid + uptime | |
| 20:13:17 | s3rj1k | if we can get time info | |
| 20:13:35 | sean-k-mooney | well we dont want it really to change often | |
| 20:13:58 | sean-k-mooney | and that woudl be unstable ie.. tow diffent schduler check 1 second apprart | |
| 20:14:10 | sean-k-mooney | that coudl lead to both thinking the other is the leader | |
| 20:14:21 | s3rj1k | I mean prefer ones that run longer | |
| 20:14:25 | sean-k-mooney | but i dotn think the unbiase dload is a problem in this case | |
| 20:15:00 | sean-k-mooney | maybe i would keep it simple initaly | |
| 20:15:09 | sean-k-mooney | we coudl alway teak it later if needed | |
| 20:15:15 | s3rj1k | but simple uuid sort would work as this is not very critical place | |
| 20:16:40 | sean-k-mooney | by the way up is not directly a property on the service (we hsould make it one) we have a function that takes the last time we saw it and a config value to decied if its still up | |
| 20:16:56 | sean-k-mooney | but that pretty easy to handel | |
| 20:17:46 | sean-k-mooney | we do that computation and systisze the up value in teh api when we do a service list | |
| 20:20:39 | sean-k-mooney | basically we have https://github.com/openstack/nova/blob/master/nova/objects/service.py#L316 | |
| 20:20:56 | sean-k-mooney | which tells use when the last heartbet was recived | |
| 20:21:15 | sean-k-mooney | and we jus tneed to compare that to the interval and allow missed heatbeats | |
| 20:22:07 | sean-k-mooney | or actully we can jsut check https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.service_down_time driectly | |
| 20:22:33 | s3rj1k | I see report_count in there | |
| 20:23:51 | sean-k-mooney | up = datetime.now() - CONF.service_down_time < last_seen_up | |
| 20:24:06 | sean-k-mooney | something like ^ would work | |
| 20:24:29 | s3rj1k | report_count is per-service right? | |