| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-11-22 | |||
| 00:13:20 | ildikov | thanks again for checking my bug report! | |
| 00:13:25 | ildikov | good night! | |
| 02:19:14 | opendevreview | melanie witt proposed openstack/nova master: Add [quota]unified_limits_resource_(strategy|list) https://review.opendev.org/c/openstack/nova/+/924025 | |
| 09:07:30 | tkajinam | bauzas, hi could you check https://review.opendev.org/c/openstack/releases/+/935892 when you have time ? | |
| 09:09:44 | bauzas | tkajinam: just did | |
| 09:21:40 | tkajinam | bauzas, thanks 1 | |
| 09:21:42 | tkajinam | ! | |
| 09:48:33 | opendevreview | renliang proposed openstack/nova master: Update documentation link for openSUSE index. https://review.opendev.org/c/openstack/nova/+/927705 | |
| 12:37:42 | stephenfin | gmann: Thanks for the very thorough review on https://review.opendev.org/c/openstack/nova/+/915743 That must have taken you a while. I appreciate it and will respin quickly 🙏 | |
| 12:38:51 | ratailor | bauzas, sean-k-mooney could you please check https://review.opendev.org/c/openstack/nova/+/934294 https://review.opendev.org/c/openstack/nova/+/873901 | |
| 12:42:24 | ratailor | gibi, could you please check this https://review.opendev.org/c/openstack/nova/+/927732 its clean cherry-pick and required 1 more +2. | |
| 12:47:14 | gibi | ratailor: done | |
| 12:47:58 | ratailor | gibi, Thanks! | |
| 12:50:10 | ratailor | gibi, also is this approach looks good. WDYT https://review.opendev.org/c/openstack/nova/+/933107 | |
| 14:06:57 | s3rj1k | sean-k-mooney: can you please take a look at https://bugs.launchpad.net/nova/+bug/2089386 when you have time, thanks | |
| 14:08:20 | sean-k-mooney | s3rj1k: so we dont really recomemnt usign the periodic in production | |
| 14:08:30 | sean-k-mooney | you can but it does not scale | |
| 14:09:21 | sean-k-mooney | the database constraits should be sufficent to have correct behaivor even without a distributed lock | |
| 14:09:42 | s3rj1k | sean-k-mooney: even without periodics there is an issue with concurrency for running multiple CLI jobs | |
| 14:10:16 | sean-k-mooney | how so. the unique constraits will fail to allow multipel cli jobs to add the same host | |
| 14:10:29 | sean-k-mooney | so one will just fail and the other will succeed | |
| 14:11:07 | sean-k-mooney | s3rj1k: nova would like to remove our use of tooz in general as an fyi | |
| 14:11:24 | sean-k-mooney | its currently only used for the ironic driver and the functionaltiy that uses it is drepcated for removal | |
| 14:12:19 | s3rj1k | hmm, so maybe periodics should be marked as deprecated? | |
| 14:12:32 | sean-k-mooney | s3rj1k: we could impove this behaivor but our current docs say that you may only enable the perodic in one schduler instnace | |
| 14:12:52 | sean-k-mooney | so this is a known limitation in the curent design | |
| 14:13:26 | sean-k-mooney | s3rj1k: the enhancement is not invlaid but its somethin we have decied not to adress in the past | |
| 14:13:39 | s3rj1k | this is for HA setups, where there are multiple schedulers, so there is a need to have at least 3 periodics enabled | |
| 14:14:06 | sean-k-mooney | right so thats not supported today | |
| 14:14:15 | s3rj1k | think Nova in k8s basically | |
| 14:14:36 | sean-k-mooney | yep in our new k8s operator we expiclty dont use this because of this limitation | |
| 14:14:59 | sean-k-mooney | currntly we requre the human ot manualy run it if they scale out comptue nodes | |
| 14:15:14 | sean-k-mooney | given that is an infrequet operation | |
| 14:15:42 | sean-k-mooney | gibi: ^ you might be interested in thsi conversation | |
| 14:15:58 | s3rj1k | it would be nicer to remove a need of manual tasking here | |
| 14:16:11 | sean-k-mooney | s3rj1k: a better appoch might be to make the mappign code more robost | |
| 14:16:36 | sean-k-mooney | i.e. to catch the db conflict/duplicate key error and proceed with the rest of the mappings | |
| 14:18:00 | sean-k-mooney | s3rj1k: for what its worth we did consier automating it in a k8s chron job | |
| 14:18:09 | s3rj1k | what about periodics? I mean we should decide what we should do with it, having this around and saying that it should not be used looks a bit off | |
| 14:18:26 | sean-k-mooney | that not what we say | |
| 14:18:40 | sean-k-mooney | we say it can be used but must only be enabled in one schduler instance | |
| 14:18:55 | sean-k-mooney | and htat for large deployment its avised to not run it frequently | |
| 14:19:13 | s3rj1k | same a basically setting up plain Linux cron with CLI | |
| 14:19:19 | sean-k-mooney | but yes we coudl deprecate it for removal, but i dont think we want to break existing users | |
| 14:19:42 | sean-k-mooney | s3rj1k: yes and no. the delta is that its ha | |
| 14:20:03 | sean-k-mooney | i.e. as long asa you k8s cluster is functional it can schduler to any approate worker | |
| 14:20:25 | s3rj1k | I mean in case of single node setups, cron vs periodic is same thin, no? | |
| 14:20:29 | sean-k-mooney | the reason we didnt do that of our installer is we did not think this was a good default for production | |
| 14:20:52 | sean-k-mooney | sure but then you cant have the conflcit | |
| 14:21:29 | sean-k-mooney | unless your running multipel schduler on a single node cluster? doable but proably not super useful | |
| 14:22:26 | s3rj1k | yea, point is case env setups what are ok to have periodics enabled are not HA at all and there is no point in supporting periodics in case of not also supporting HA with it, since plain cron can replace it on non HA | |
| 14:23:10 | s3rj1k | so effectively dead code in terms of production, fine for small setups only or dev clusters | |
| 14:23:41 | sean-k-mooney | so where this is used in production today its not enabeld on multiple hosts | |
| 14:23:53 | sean-k-mooney | but i agree its not as useful as it could be | |
| 14:24:31 | sean-k-mooney | s3rj1k: i dont think the request for enhancement is invlaid but pulling in tooz for jsut this is a hard cell | |
| 14:25:21 | sean-k-mooney | *sell | |
| 14:25:50 | sean-k-mooney | we generally want the scheuler ot be stateless and nova does not use any distibuted lockign today partly for horizontal scaleableity | |
| 14:26:38 | sean-k-mooney | we may be able to do https://en.wikipedia.org/wiki/Rendezvous_hashing | |
| 14:27:36 | sean-k-mooney | so that only one schdluler will try to map a given host on any given invocation but that woudl make the time to mapping less deterministic so im conflicted on that | |
| 14:39:05 | sean-k-mooney | s3rj1k: for right now im going to set the bug to Opinion but i think it would be good to either bring this up in the next team metting, ask others for input on irc or start a mailing list thread on this topic | |
| 14:39:49 | sean-k-mooney | s3rj1k: at a minium we could imporve the docuemation aroudn this | |
| 14:40:19 | sean-k-mooney | we may be able to do more but adding a distribute lock manager to nova is definally a feature not a bug | |
| 14:42:42 | s3rj1k | yea, feature, tagged as rfe | |
| 14:42:48 | s3rj1k | for sure nit a bug | |
| 14:42:54 | sean-k-mooney | another approch might be to catch the "host is not mapped to a cell" excption w eget today and use that to trigger a mapping reactively | |
| 14:44:34 | s3rj1k | also would like to note that this is no k8s specific and oldschool phys servers HA setup is probably even more impacted by this | |
| 14:45:07 | sean-k-mooney | sure as i said this is an old topic | |
| 14:45:36 | sean-k-mooney | the tension is that this really only make sense in an env wehre the number of compute nodes changes frequently | |
| 14:45:51 | sean-k-mooney | in such a env setting along interval is not palatable | |
| 14:45:59 | sean-k-mooney | so you have to set the period to be run often | |
| 14:46:21 | sean-k-mooney | which mean it adds a lot of load espcitlly if you run multipel isntances of the scduelr each doing it on the db | |
| 14:47:53 | sean-k-mooney | so the more frequently you add comptues, the more often you have to run it, and the more load it creates, which also gets more expensive the more host you have as there are more hosts to process | |
| 14:48:49 | sean-k-mooney | s3rj1k: in our operator we orgingally hoped to have the golang reconsile loop run the host mapping when compute were added | |
| 14:49:11 | sean-k-mooney | but architecturla design reasons prevent that form happing today | |
| 14:49:22 | sean-k-mooney | s3rj1k: im not sure if you have the same desgin limistiaon | |
| 14:50:08 | s3rj1k | I guess that the baremininum thing is to ensure that CLI Jobs are running sequentially | |
| 14:50:53 | sean-k-mooney | tl;dr our nova-operator is not allowed to monitor the addtion/removal of compute nodes, and our compute node deployment code is not allowed to acess the db (run nova-manage) or talk to k8s | |
| 14:51:07 | s3rj1k | but for that it would be nice to have some more visible descision around internal periodics, as they always come up | |
| 14:52:14 | sean-k-mooney | s3rj1k: melwitt had a propsoal for a seperate nova-audit deamon a long time ago to have a seperate process dedicated to running them | |
| 14:52:58 | sean-k-mooney | we did not move forward with that but the idea wwas to move this perodic and the db archive/peruge operatiosn to that seperate optional demaon | |
| 14:54:12 | sean-k-mooney | the idea was to centralise all of the perodic mantance and self healign task into a seperatly deployable but optional compoent | |
| 14:54:42 | s3rj1k | and how to handle HA in this case? | |
| 14:55:08 | sean-k-mooney | intially active passive so in k8s you would only run 1 as deplyment/statefulset | |
| 14:55:34 | sean-k-mooney | for non k8s deploymetsn you would use pacemaker ectra to manage which one was running | |
| 14:55:48 | sean-k-mooney | you could use a DLM as well | |
| 14:56:10 | sean-k-mooney | but given this is not a performace critical part that is less concerning | |
| 14:59:38 | s3rj1k | so a lot of extra things to hack this up in case of plain servers | |
| 15:00:09 | s3rj1k | for k8s case, active/passive could probably work | |
| 15:00:43 | sean-k-mooney | yep as i siad this is old ground but there is a renewd interst in makign nova be more "ks8 native" | |
| 15:01:31 | sean-k-mooney | s3rj1k: This was the last time nova-audit was serisoly dicussed https://specs.openstack.org/openstack/nova-specs/specs/xena/approved/nova-audit.html | |
| 15:02:06 | sean-k-mooney | s3rj1k: we discussed soem other k8s painpoint like graceful shutdown too at the last ptg | |
| 15:02:13 | sean-k-mooney | and healthchecks | |
| 15:02:49 | s3rj1k | all of those would be very nice to have, yes | |
| 15:02:52 | sean-k-mooney | so i intentilaly didnt close https://bugs.launchpad.net/nova/+bug/2089386 as invialid, but hostircally we have confired issues like this as out of scope | |
| 15:04:02 | sean-k-mooney | s/confired/considered/ | |
| 15:04:52 | sean-k-mooney | s3rj1k: i need to go finsih something but do bring this up in the next irc meeting or start a mailing list thrad about how to move this forward to get input form others | |
| 15:06:22 | s3rj1k | sure, let's let it brew over a weekend, I'll ping back on this in next meetup | |
| 15:06:34 | s3rj1k | have a nice weekend! | |
| 15:06:56 | sean-k-mooney | o/ | |
| 15:11:58 | opendevreview | ribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (manila abstraction) https://review.opendev.org/c/openstack/nova/+/831194 | |