| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-11-27 | |||
| 19:20:04 | sean-k-mooney | it has a way to test if a member of a service group is up | |
| 19:20:16 | sean-k-mooney | but not a way to list the memebr and or select a leader | |
| 19:22:19 | opendevreview | ribaudr proposed openstack/nova-specs master: WIP: Allow vfio gpu support with Vendor-Specific VFIO framework https://review.opendev.org/c/openstack/nova-specs/+/936407 | |
| 19:43:21 | opendevreview | Merged openstack/nova stable/2024.1: Handle neutron-client conflict https://review.opendev.org/c/openstack/nova/+/927732 | |
| 19:45:04 | s3rj1k | > but not a way to list the memebr and or select a leader - so that would mean that we would firstly need to extend that and after that try to use that | |
| 19:51:20 | sean-k-mooney | for the db dirver its pretty simple | |
| 19:51:36 | sean-k-mooney | that basically a tweaked version fo https://github.com/openstack/nova/blob/master/nova/db/main/api.py#L492-L503 | |
| 19:51:43 | sean-k-mooney | i.e. get service by binary | |
| 19:52:52 | sean-k-mooney | in this case of memcache its a littel more annoying | |
| 19:53:39 | sean-k-mooney | we coudl just ignore the service goups and just use the service api | |
| 19:53:53 | sean-k-mooney | *service db tables not api | |
| 19:54:51 | sean-k-mooney | basically im trying to see what whoudl result in the minium number of addtional rpc/db/rest calls to enable | |
| 19:58:19 | s3rj1k | > minium number of addtional rpc/db/rest calls - solution by documentation :) | |
| 20:05:56 | sean-k-mooney | maybe, or https://paste.opendev.org/show/bFFLh0pVeBZAOYXoNjKs/ | |
| 20:06:58 | sean-k-mooney | we woudl just add that here https://github.com/openstack/nova/blob/master/nova/scheduler/manager.py#L111 | |
| 20:07:50 | s3rj1k | hmm, if that works that would be nice, can be combined with docs extension | |
| 20:07:57 | sean-k-mooney | sure | |
| 20:08:43 | sean-k-mooney | basicaly that one addtional db call on each invocation fo the perodic. but its a net win because instead of 3 of the perodic doing the work in paralel only 1 of them will execute the rest of the code | |
| 20:08:50 | sean-k-mooney | so overall that shoudl reduce the db load | |
| 20:09:03 | s3rj1k | in that case warning about concurrent CLIs and extension to how periodics work | |
| 20:10:08 | s3rj1k | one thing, in case of AA how would leader election work? like fallback? | |
| 20:10:35 | sean-k-mooney | so the heartbeat woud expire on the failed schduler | |
| 20:10:53 | s3rj1k | and alive one would be elected? | |
| 20:10:54 | sean-k-mooney | and then then it would be elimistaed for the set and we would select a new leader | |
| 20:11:19 | sean-k-mooney | right so if we sort on the uuid then that would be stable or the hostname | |
| 20:11:38 | sean-k-mooney | so if one host/pod dies then it will get remvoed form the set and we just take the first form the list | |
| 20:12:14 | sean-k-mooney | its not ture leader election in that when the down host comes back it takes over as the leader | |
| 20:12:28 | s3rj1k | this is fine | |
| 20:12:44 | sean-k-mooney | where as normally you woudl only do an election when then current wone extits but ya it should work for this case | |
| 20:13:06 | s3rj1k | we can sort by uuid + uptime | |
| 20:13:17 | s3rj1k | if we can get time info | |
| 20:13:35 | sean-k-mooney | well we dont want it really to change often | |
| 20:13:58 | sean-k-mooney | and that woudl be unstable ie.. tow diffent schduler check 1 second apprart | |
| 20:14:10 | sean-k-mooney | that coudl lead to both thinking the other is the leader | |
| 20:14:21 | s3rj1k | I mean prefer ones that run longer | |
| 20:14:25 | sean-k-mooney | but i dotn think the unbiase dload is a problem in this case | |
| 20:15:00 | sean-k-mooney | maybe i would keep it simple initaly | |
| 20:15:09 | sean-k-mooney | we coudl alway teak it later if needed | |
| 20:15:15 | s3rj1k | but simple uuid sort would work as this is not very critical place | |
| 20:16:40 | sean-k-mooney | by the way up is not directly a property on the service (we hsould make it one) we have a function that takes the last time we saw it and a config value to decied if its still up | |
| 20:16:56 | sean-k-mooney | but that pretty easy to handel | |
| 20:17:46 | sean-k-mooney | we do that computation and systisze the up value in teh api when we do a service list | |
| 20:20:39 | sean-k-mooney | basically we have https://github.com/openstack/nova/blob/master/nova/objects/service.py#L316 | |
| 20:20:56 | sean-k-mooney | which tells use when the last heartbet was recived | |
| 20:21:15 | sean-k-mooney | and we jus tneed to compare that to the interval and allow missed heatbeats | |
| 20:22:07 | sean-k-mooney | or actully we can jsut check https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.service_down_time driectly | |
| 20:22:33 | s3rj1k | I see report_count in there | |
| 20:23:51 | sean-k-mooney | up = datetime.now() - CONF.service_down_time < last_seen_up | |
| 20:24:06 | sean-k-mooney | something like ^ would work | |
| 20:24:29 | s3rj1k | report_count is per-service right? | |
| 20:24:37 | sean-k-mooney | yes | |
| 20:25:07 | s3rj1k | so if thing goes down and up it gets nulled and count get started over? | |
| 20:25:25 | sean-k-mooney | i dont recall of the top of my head | |
| 20:25:44 | sean-k-mooney | in this context we can ignore i ti think | |
| 20:26:03 | sean-k-mooney | anyway i better finish for today | |
| 20:26:13 | s3rj1k | so basically there are solution how to make that selection more clever | |
| 20:26:27 | sean-k-mooney | yep if needed | |
| 20:26:27 | s3rj1k | thanks for you input | |
| 20:26:47 | sean-k-mooney | if we were trying to distibute load | |
| 20:26:54 | sean-k-mooney | we might want a more clever solution | |
| 20:27:08 | sean-k-mooney | but since we are trying to make sure only one runs i dont think that isneeded | |
| 20:27:44 | sean-k-mooney | we might want a minium uptime before the leader can change | |
| 20:27:58 | sean-k-mooney | i.e. double the perodic interval or somthing like that | |
| 20:28:25 | sean-k-mooney | just to prevent issues on start up | |
| 20:28:47 | s3rj1k | even of we miss, next run will fix that | |
| 20:28:49 | sean-k-mooney | or perhaps double the service downtime interval | |
| 20:29:02 | s3rj1k | s/of/if/ | |
| 20:29:24 | sean-k-mooney | but yes i think we can do this in a lock free way and it will self heal | |
| 20:29:39 | sean-k-mooney | actully droping now o/ | |
| 20:30:00 | s3rj1k | have a good night | |
| #openstack-nova - 2024-11-28 | |||
| 04:28:46 | opendevreview | Nisha Brahmankar proposed openstack/nova master: Retain port binding profile after live migration https://review.opendev.org/c/openstack/nova/+/936338 | |
| 10:24:23 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Add new, simpler api_version decorator https://review.opendev.org/c/openstack/nova/+/936367 | |
| 10:24:23 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Only run format checks on strings https://review.opendev.org/c/openstack/nova/+/936368 | |
| 10:24:24 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Simplify parameter types https://review.opendev.org/c/openstack/nova/+/936369 | |
| 10:24:24 | opendevreview | Stephen Finucane proposed openstack/nova master: tests: Ensure all APIs have a response body schema https://review.opendev.org/c/openstack/nova/+/924598 | |
| 11:17:53 | zigo | dansmith: Hi there! Got a question for you. I'm testing db upgrades, and populate_instance_compute_id is taking like forever. | |
| 11:17:53 | zigo | I wonder, what is this about? Your commit header says it's for "deleted, shelved, never scheduled or otherwise on a compute host that isn't updatating it", I don't really get it. | |
| 11:17:53 | zigo | Also, I have many warnings about: "Unable to migrate instance because host None with node None not found: nova.exception.ComputeHostNotFound: Compute host None could not be found.". Should I be worried? | |
| 11:23:27 | zigo | (of course, I'm not expecting a reply before you wake up ... :P ) | |
| 13:28:53 | opendevreview | Merged openstack/nova master: Add share_info parameter to reboot method for each driver (driver part) https://review.opendev.org/c/openstack/nova/+/854823 | |
| 13:37:57 | opendevreview | Merged openstack/nova master: Support rebooting an instance with shares (compute manager part) https://review.opendev.org/c/openstack/nova/+/854824 | |
| 13:50:52 | opendevreview | Merged openstack/nova master: Add share_info parameter to resume method for each driver (driver part) https://review.opendev.org/c/openstack/nova/+/860284 | |
| 14:24:29 | ishanwar | Hi team, | |
| 14:25:16 | ishanwar | I wanted to ask what is a good entry point, to start understanding the nova implementation in the codebase. I could not find much resources about the implementations on the openstack website | |
| 14:26:17 | ishanwar | Kindly suggest @Sean-k-mooney and @bauzas ? | |
| 14:33:56 | sean-k-mooney | the nova codebase is quite large btu we have alot of contibutor adn refence docs decibing the archetecture and subssytems | |
| 14:34:29 | sean-k-mooney | ishanwar: the main entry point to our docs is https://docs.openstack.org/nova/latest/ | |
| 14:34:41 | sean-k-mooney | i woudl first start with https://docs.openstack.org/nova/latest/admin/architecture.html | |
| 14:35:26 | sean-k-mooney | oru contibutor docs are here https://docs.openstack.org/nova/latest/contributor/index.html and tecnial refence docs are here https://docs.openstack.org/nova/latest/reference/index.html | |
| 14:36:47 | sean-k-mooney | beyond that there are a number of youtube videos form previosu summits on specific topics | |
| 14:38:25 | sean-k-mooney | zigo: dansmith is on PTO until the end of the year more or less so expect a delay on any pings :) | |
| 14:39:33 | sean-k-mooney | zigo: if the instance has host and node none and is not deleted or in cell0 then that is only valid if its shelved | |
| 14:39:50 | sean-k-mooney | speicifclly shelve_offoloaded | |
| 14:40:26 | sean-k-mooney | if you have an isntnace that is not shelve_offloaded and has host or node None then you have db currption | |
| 14:51:01 | ishanwar | sean-k-mooney: thanks for the docs, I had a look at the architecture, I was wondering if there is any specific command to start the apis for testing locally in the codebase | |
| 14:56:12 | ishanwar | Also, @Sean-k-mooney @bauzas, it would be great if you could point me to any low hanging bugs to solve like good-first-issues in the codebase | |
| 15:09:08 | sean-k-mooney | ishanwar: you can techinally do that but its not hwo developemnt is normally done. nova is a distirbute system with several external depencies | |
| 15:09:41 | sean-k-mooney | so the normall approch is to use devstack https://github.com/openstack/devstack to create a devleopment enviornment in a vam | |
| 15:09:43 | sean-k-mooney | *vm | |
| 15:18:38 | zigo | sean-k-mooney: My db is comming from another cluster, just testing db upgrades and online_data_migrations. Could that be the reason? | |