| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-11-27 | |||
| 20:07:57 | sean-k-mooney | sure | |
| 20:08:43 | sean-k-mooney | basicaly that one addtional db call on each invocation fo the perodic. but its a net win because instead of 3 of the perodic doing the work in paralel only 1 of them will execute the rest of the code | |
| 20:08:50 | sean-k-mooney | so overall that shoudl reduce the db load | |
| 20:09:03 | s3rj1k | in that case warning about concurrent CLIs and extension to how periodics work | |
| 20:10:08 | s3rj1k | one thing, in case of AA how would leader election work? like fallback? | |
| 20:10:35 | sean-k-mooney | so the heartbeat woud expire on the failed schduler | |
| 20:10:53 | s3rj1k | and alive one would be elected? | |
| 20:10:54 | sean-k-mooney | and then then it would be elimistaed for the set and we would select a new leader | |
| 20:11:19 | sean-k-mooney | right so if we sort on the uuid then that would be stable or the hostname | |
| 20:11:38 | sean-k-mooney | so if one host/pod dies then it will get remvoed form the set and we just take the first form the list | |
| 20:12:14 | sean-k-mooney | its not ture leader election in that when the down host comes back it takes over as the leader | |
| 20:12:28 | s3rj1k | this is fine | |
| 20:12:44 | sean-k-mooney | where as normally you woudl only do an election when then current wone extits but ya it should work for this case | |
| 20:13:06 | s3rj1k | we can sort by uuid + uptime | |
| 20:13:17 | s3rj1k | if we can get time info | |
| 20:13:35 | sean-k-mooney | well we dont want it really to change often | |
| 20:13:58 | sean-k-mooney | and that woudl be unstable ie.. tow diffent schduler check 1 second apprart | |
| 20:14:10 | sean-k-mooney | that coudl lead to both thinking the other is the leader | |
| 20:14:21 | s3rj1k | I mean prefer ones that run longer | |
| 20:14:25 | sean-k-mooney | but i dotn think the unbiase dload is a problem in this case | |
| 20:15:00 | sean-k-mooney | maybe i would keep it simple initaly | |
| 20:15:09 | sean-k-mooney | we coudl alway teak it later if needed | |
| 20:15:15 | s3rj1k | but simple uuid sort would work as this is not very critical place | |
| 20:16:40 | sean-k-mooney | by the way up is not directly a property on the service (we hsould make it one) we have a function that takes the last time we saw it and a config value to decied if its still up | |
| 20:16:56 | sean-k-mooney | but that pretty easy to handel | |
| 20:17:46 | sean-k-mooney | we do that computation and systisze the up value in teh api when we do a service list | |
| 20:20:39 | sean-k-mooney | basically we have https://github.com/openstack/nova/blob/master/nova/objects/service.py#L316 | |
| 20:20:56 | sean-k-mooney | which tells use when the last heartbet was recived | |
| 20:21:15 | sean-k-mooney | and we jus tneed to compare that to the interval and allow missed heatbeats | |
| 20:22:07 | sean-k-mooney | or actully we can jsut check https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.service_down_time driectly | |
| 20:22:33 | s3rj1k | I see report_count in there | |
| 20:23:51 | sean-k-mooney | up = datetime.now() - CONF.service_down_time < last_seen_up | |
| 20:24:06 | sean-k-mooney | something like ^ would work | |
| 20:24:29 | s3rj1k | report_count is per-service right? | |
| 20:24:37 | sean-k-mooney | yes | |
| 20:25:07 | s3rj1k | so if thing goes down and up it gets nulled and count get started over? | |
| 20:25:25 | sean-k-mooney | i dont recall of the top of my head | |
| 20:25:44 | sean-k-mooney | in this context we can ignore i ti think | |
| 20:26:03 | sean-k-mooney | anyway i better finish for today | |
| 20:26:13 | s3rj1k | so basically there are solution how to make that selection more clever | |
| 20:26:27 | sean-k-mooney | yep if needed | |
| 20:26:27 | s3rj1k | thanks for you input | |
| 20:26:47 | sean-k-mooney | if we were trying to distibute load | |
| 20:26:54 | sean-k-mooney | we might want a more clever solution | |
| 20:27:08 | sean-k-mooney | but since we are trying to make sure only one runs i dont think that isneeded | |
| 20:27:44 | sean-k-mooney | we might want a minium uptime before the leader can change | |
| 20:27:58 | sean-k-mooney | i.e. double the perodic interval or somthing like that | |
| 20:28:25 | sean-k-mooney | just to prevent issues on start up | |
| 20:28:47 | s3rj1k | even of we miss, next run will fix that | |
| 20:28:49 | sean-k-mooney | or perhaps double the service downtime interval | |
| 20:29:02 | s3rj1k | s/of/if/ | |
| 20:29:24 | sean-k-mooney | but yes i think we can do this in a lock free way and it will self heal | |
| 20:29:39 | sean-k-mooney | actully droping now o/ | |
| 20:30:00 | s3rj1k | have a good night | |
| #openstack-nova - 2024-11-28 | |||
| 04:28:46 | opendevreview | Nisha Brahmankar proposed openstack/nova master: Retain port binding profile after live migration https://review.opendev.org/c/openstack/nova/+/936338 | |
| 10:24:23 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Add new, simpler api_version decorator https://review.opendev.org/c/openstack/nova/+/936367 | |
| 10:24:23 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Only run format checks on strings https://review.opendev.org/c/openstack/nova/+/936368 | |
| 10:24:24 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Simplify parameter types https://review.opendev.org/c/openstack/nova/+/936369 | |
| 10:24:24 | opendevreview | Stephen Finucane proposed openstack/nova master: tests: Ensure all APIs have a response body schema https://review.opendev.org/c/openstack/nova/+/924598 | |
| 11:17:53 | zigo | dansmith: Hi there! Got a question for you. I'm testing db upgrades, and populate_instance_compute_id is taking like forever. | |
| 11:17:53 | zigo | I wonder, what is this about? Your commit header says it's for "deleted, shelved, never scheduled or otherwise on a compute host that isn't updatating it", I don't really get it. | |
| 11:17:53 | zigo | Also, I have many warnings about: "Unable to migrate instance because host None with node None not found: nova.exception.ComputeHostNotFound: Compute host None could not be found.". Should I be worried? | |
| 11:23:27 | zigo | (of course, I'm not expecting a reply before you wake up ... :P ) | |
| 13:28:53 | opendevreview | Merged openstack/nova master: Add share_info parameter to reboot method for each driver (driver part) https://review.opendev.org/c/openstack/nova/+/854823 | |
| 13:37:57 | opendevreview | Merged openstack/nova master: Support rebooting an instance with shares (compute manager part) https://review.opendev.org/c/openstack/nova/+/854824 | |
| 13:50:52 | opendevreview | Merged openstack/nova master: Add share_info parameter to resume method for each driver (driver part) https://review.opendev.org/c/openstack/nova/+/860284 | |
| 14:24:29 | ishanwar | Hi team, | |
| 14:25:16 | ishanwar | I wanted to ask what is a good entry point, to start understanding the nova implementation in the codebase. I could not find much resources about the implementations on the openstack website | |
| 14:26:17 | ishanwar | Kindly suggest @Sean-k-mooney and @bauzas ? | |
| 14:33:56 | sean-k-mooney | the nova codebase is quite large btu we have alot of contibutor adn refence docs decibing the archetecture and subssytems | |
| 14:34:29 | sean-k-mooney | ishanwar: the main entry point to our docs is https://docs.openstack.org/nova/latest/ | |
| 14:34:41 | sean-k-mooney | i woudl first start with https://docs.openstack.org/nova/latest/admin/architecture.html | |
| 14:35:26 | sean-k-mooney | oru contibutor docs are here https://docs.openstack.org/nova/latest/contributor/index.html and tecnial refence docs are here https://docs.openstack.org/nova/latest/reference/index.html | |
| 14:36:47 | sean-k-mooney | beyond that there are a number of youtube videos form previosu summits on specific topics | |
| 14:38:25 | sean-k-mooney | zigo: dansmith is on PTO until the end of the year more or less so expect a delay on any pings :) | |
| 14:39:33 | sean-k-mooney | zigo: if the instance has host and node none and is not deleted or in cell0 then that is only valid if its shelved | |
| 14:39:50 | sean-k-mooney | speicifclly shelve_offoloaded | |
| 14:40:26 | sean-k-mooney | if you have an isntnace that is not shelve_offloaded and has host or node None then you have db currption | |
| 14:51:01 | ishanwar | sean-k-mooney: thanks for the docs, I had a look at the architecture, I was wondering if there is any specific command to start the apis for testing locally in the codebase | |
| 14:56:12 | ishanwar | Also, @Sean-k-mooney @bauzas, it would be great if you could point me to any low hanging bugs to solve like good-first-issues in the codebase | |
| 15:09:08 | sean-k-mooney | ishanwar: you can techinally do that but its not hwo developemnt is normally done. nova is a distirbute system with several external depencies | |
| 15:09:41 | sean-k-mooney | so the normall approch is to use devstack https://github.com/openstack/devstack to create a devleopment enviornment in a vam | |
| 15:09:43 | sean-k-mooney | *vm | |
| 15:18:38 | zigo | sean-k-mooney: My db is comming from another cluster, just testing db upgrades and online_data_migrations. Could that be the reason? | |
| 15:19:37 | zigo | FYI, that's form Antelope to Caracal. | |
| 15:19:42 | zigo | from | |
| 15:28:50 | sean-k-mooney | populate_instance_compute_id should be using data form within the cell db | |
| 15:29:31 | sean-k-mooney | so if your doign this on a local backup ectra then it should not cause issues that im aware of | |
| 15:30:47 | sean-k-mooney | this is what its actully doing https://github.com/openstack/nova/blob/master/nova/objects/instance.py#L1357-L1377 | |
| 15:31:47 | sean-k-mooney | for each instance we get the node and host values and lookup the compute node id then set that in teh instance compute_id column | |
| 15:32:32 | sean-k-mooney | so that shoudl be relitivly quick although if you have a lot of instnaces then it will take a while | |
| 15:33:16 | zigo | That cluster has ~8k VM, though many deleted, I guess. | |
| 15:33:42 | zigo | It's rather fast, what I'm worried about, is the None thingy, plus compute nodes that we decomitionned. | |
| 15:33:44 | sean-k-mooney | we are reading deleted isntanfe too | |
| 15:34:01 | zigo | Yeah, and I don't understand why. | |
| 15:34:25 | sean-k-mooney | its backpopulating deleted but not archived isntances | |
| 15:34:33 | sean-k-mooney | to keep it consitent | |
| 15:34:44 | zigo | Oh, maybe I should just archive them then ... | |
| 15:35:30 | sean-k-mooney | so if the compute node is delete then this will skip them | |
| 15:35:46 | sean-k-mooney | but if your rungin this in a loop it will never fully complete | |