| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-11-21 | |||
| 23:55:20 | sean-k-mooney | ah ok that is coming form the openstack docs theme then | |
| 23:55:33 | sean-k-mooney | rather then form what we have configured on launchpad | |
| 23:55:39 | sean-k-mooney | its fine either way | |
| 23:55:50 | sean-k-mooney | most peopel ignow the bug text anyway :) | |
| 23:56:20 | ildikov | I would assume so, I intentionally did not change the template I got, as I assumed that wherever it's coming from, it's there for a reason | |
| 23:56:32 | ildikov | lol, I bet | |
| 23:57:04 | ildikov | I used to be a core reviewer a long time ago, so I try my best to create decent bug reports :) | |
| 23:57:40 | sean-k-mooney | its more actionable then a lot of the ones we get so you have not lost your touch | |
| 23:58:12 | ildikov | thank you :) | |
| 23:59:46 | sean-k-mooney | https://bugs.launchpad.net/nova/+bug/2085585 is a particallay fine example | |
| #openstack-nova - 2024-11-22 | |||
| 00:01:43 | ildikov | lol, wow | |
| 00:02:37 | ildikov | there's technically no usable information in that one | |
| 00:03:28 | sean-k-mooney | i was hopign it was a duplicate of https://bugs.launchpad.net/nova/+bug/2088341 since it has the same title but that is fileed by someone else | |
| 00:05:50 | ildikov | I mean, with the amount of information in the first one, saying it's a duplicate is just as accurate as saying it's not | |
| 00:06:14 | ildikov | maybe the two people know each other and worked smh together :) | |
| 00:07:34 | sean-k-mooney | i think they just took oru generic api error header too litrally | |
| 00:07:37 | sean-k-mooney | "Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible." | |
| 00:08:20 | sean-k-mooney | some other software project put an identifyer that allows them to track wehre the issue came form in messages like that | |
| 00:08:24 | s3rj1k | reporter fact-proved that "Unexpected API Error" exists | |
| 00:09:13 | sean-k-mooney | hopefully there issue either went away or they cna come back and tell us a littel more | |
| 00:09:29 | sean-k-mooney | anywya im going to drop again for the night o/ | |
| 00:12:53 | ildikov | ah, I see | |
| 00:13:09 | ildikov | I have my fingers crossed they'll come back if the issue persists! | |
| 00:13:20 | ildikov | thanks again for checking my bug report! | |
| 00:13:25 | ildikov | good night! | |
| 02:19:14 | opendevreview | melanie witt proposed openstack/nova master: Add [quota]unified_limits_resource_(strategy|list) https://review.opendev.org/c/openstack/nova/+/924025 | |
| 09:07:30 | tkajinam | bauzas, hi could you check https://review.opendev.org/c/openstack/releases/+/935892 when you have time ? | |
| 09:09:44 | bauzas | tkajinam: just did | |
| 09:21:40 | tkajinam | bauzas, thanks 1 | |
| 09:21:42 | tkajinam | ! | |
| 09:48:33 | opendevreview | renliang proposed openstack/nova master: Update documentation link for openSUSE index. https://review.opendev.org/c/openstack/nova/+/927705 | |
| 12:37:42 | stephenfin | gmann: Thanks for the very thorough review on https://review.opendev.org/c/openstack/nova/+/915743 That must have taken you a while. I appreciate it and will respin quickly 🙏 | |
| 12:38:51 | ratailor | bauzas, sean-k-mooney could you please check https://review.opendev.org/c/openstack/nova/+/934294 https://review.opendev.org/c/openstack/nova/+/873901 | |
| 12:42:24 | ratailor | gibi, could you please check this https://review.opendev.org/c/openstack/nova/+/927732 its clean cherry-pick and required 1 more +2. | |
| 12:47:14 | gibi | ratailor: done | |
| 12:47:58 | ratailor | gibi, Thanks! | |
| 12:50:10 | ratailor | gibi, also is this approach looks good. WDYT https://review.opendev.org/c/openstack/nova/+/933107 | |
| 14:06:57 | s3rj1k | sean-k-mooney: can you please take a look at https://bugs.launchpad.net/nova/+bug/2089386 when you have time, thanks | |
| 14:08:20 | sean-k-mooney | s3rj1k: so we dont really recomemnt usign the periodic in production | |
| 14:08:30 | sean-k-mooney | you can but it does not scale | |
| 14:09:21 | sean-k-mooney | the database constraits should be sufficent to have correct behaivor even without a distributed lock | |
| 14:09:42 | s3rj1k | sean-k-mooney: even without periodics there is an issue with concurrency for running multiple CLI jobs | |
| 14:10:16 | sean-k-mooney | how so. the unique constraits will fail to allow multipel cli jobs to add the same host | |
| 14:10:29 | sean-k-mooney | so one will just fail and the other will succeed | |
| 14:11:07 | sean-k-mooney | s3rj1k: nova would like to remove our use of tooz in general as an fyi | |
| 14:11:24 | sean-k-mooney | its currently only used for the ironic driver and the functionaltiy that uses it is drepcated for removal | |
| 14:12:19 | s3rj1k | hmm, so maybe periodics should be marked as deprecated? | |
| 14:12:32 | sean-k-mooney | s3rj1k: we could impove this behaivor but our current docs say that you may only enable the perodic in one schduler instnace | |
| 14:12:52 | sean-k-mooney | so this is a known limitation in the curent design | |
| 14:13:26 | sean-k-mooney | s3rj1k: the enhancement is not invlaid but its somethin we have decied not to adress in the past | |
| 14:13:39 | s3rj1k | this is for HA setups, where there are multiple schedulers, so there is a need to have at least 3 periodics enabled | |
| 14:14:06 | sean-k-mooney | right so thats not supported today | |
| 14:14:15 | s3rj1k | think Nova in k8s basically | |
| 14:14:36 | sean-k-mooney | yep in our new k8s operator we expiclty dont use this because of this limitation | |
| 14:14:59 | sean-k-mooney | currntly we requre the human ot manualy run it if they scale out comptue nodes | |
| 14:15:14 | sean-k-mooney | given that is an infrequet operation | |
| 14:15:42 | sean-k-mooney | gibi: ^ you might be interested in thsi conversation | |
| 14:15:58 | s3rj1k | it would be nicer to remove a need of manual tasking here | |
| 14:16:11 | sean-k-mooney | s3rj1k: a better appoch might be to make the mappign code more robost | |
| 14:16:36 | sean-k-mooney | i.e. to catch the db conflict/duplicate key error and proceed with the rest of the mappings | |
| 14:18:00 | sean-k-mooney | s3rj1k: for what its worth we did consier automating it in a k8s chron job | |
| 14:18:09 | s3rj1k | what about periodics? I mean we should decide what we should do with it, having this around and saying that it should not be used looks a bit off | |
| 14:18:26 | sean-k-mooney | that not what we say | |
| 14:18:40 | sean-k-mooney | we say it can be used but must only be enabled in one schduler instance | |
| 14:18:55 | sean-k-mooney | and htat for large deployment its avised to not run it frequently | |
| 14:19:13 | s3rj1k | same a basically setting up plain Linux cron with CLI | |
| 14:19:19 | sean-k-mooney | but yes we coudl deprecate it for removal, but i dont think we want to break existing users | |
| 14:19:42 | sean-k-mooney | s3rj1k: yes and no. the delta is that its ha | |
| 14:20:03 | sean-k-mooney | i.e. as long asa you k8s cluster is functional it can schduler to any approate worker | |
| 14:20:25 | s3rj1k | I mean in case of single node setups, cron vs periodic is same thin, no? | |
| 14:20:29 | sean-k-mooney | the reason we didnt do that of our installer is we did not think this was a good default for production | |
| 14:20:52 | sean-k-mooney | sure but then you cant have the conflcit | |
| 14:21:29 | sean-k-mooney | unless your running multipel schduler on a single node cluster? doable but proably not super useful | |
| 14:22:26 | s3rj1k | yea, point is case env setups what are ok to have periodics enabled are not HA at all and there is no point in supporting periodics in case of not also supporting HA with it, since plain cron can replace it on non HA | |
| 14:23:10 | s3rj1k | so effectively dead code in terms of production, fine for small setups only or dev clusters | |
| 14:23:41 | sean-k-mooney | so where this is used in production today its not enabeld on multiple hosts | |
| 14:23:53 | sean-k-mooney | but i agree its not as useful as it could be | |
| 14:24:31 | sean-k-mooney | s3rj1k: i dont think the request for enhancement is invlaid but pulling in tooz for jsut this is a hard cell | |
| 14:25:21 | sean-k-mooney | *sell | |
| 14:25:50 | sean-k-mooney | we generally want the scheuler ot be stateless and nova does not use any distibuted lockign today partly for horizontal scaleableity | |
| 14:26:38 | sean-k-mooney | we may be able to do https://en.wikipedia.org/wiki/Rendezvous_hashing | |
| 14:27:36 | sean-k-mooney | so that only one schdluler will try to map a given host on any given invocation but that woudl make the time to mapping less deterministic so im conflicted on that | |
| 14:39:05 | sean-k-mooney | s3rj1k: for right now im going to set the bug to Opinion but i think it would be good to either bring this up in the next team metting, ask others for input on irc or start a mailing list thread on this topic | |
| 14:39:49 | sean-k-mooney | s3rj1k: at a minium we could imporve the docuemation aroudn this | |
| 14:40:19 | sean-k-mooney | we may be able to do more but adding a distribute lock manager to nova is definally a feature not a bug | |
| 14:42:42 | s3rj1k | yea, feature, tagged as rfe | |
| 14:42:48 | s3rj1k | for sure nit a bug | |
| 14:42:54 | sean-k-mooney | another approch might be to catch the "host is not mapped to a cell" excption w eget today and use that to trigger a mapping reactively | |
| 14:44:34 | s3rj1k | also would like to note that this is no k8s specific and oldschool phys servers HA setup is probably even more impacted by this | |
| 14:45:07 | sean-k-mooney | sure as i said this is an old topic | |
| 14:45:36 | sean-k-mooney | the tension is that this really only make sense in an env wehre the number of compute nodes changes frequently | |
| 14:45:51 | sean-k-mooney | in such a env setting along interval is not palatable | |
| 14:45:59 | sean-k-mooney | so you have to set the period to be run often | |
| 14:46:21 | sean-k-mooney | which mean it adds a lot of load espcitlly if you run multipel isntances of the scduelr each doing it on the db | |
| 14:47:53 | sean-k-mooney | so the more frequently you add comptues, the more often you have to run it, and the more load it creates, which also gets more expensive the more host you have as there are more hosts to process | |
| 14:48:49 | sean-k-mooney | s3rj1k: in our operator we orgingally hoped to have the golang reconsile loop run the host mapping when compute were added | |
| 14:49:11 | sean-k-mooney | but architecturla design reasons prevent that form happing today | |
| 14:49:22 | sean-k-mooney | s3rj1k: im not sure if you have the same desgin limistiaon | |
| 14:50:08 | s3rj1k | I guess that the baremininum thing is to ensure that CLI Jobs are running sequentially | |
| 14:50:53 | sean-k-mooney | tl;dr our nova-operator is not allowed to monitor the addtion/removal of compute nodes, and our compute node deployment code is not allowed to acess the db (run nova-manage) or talk to k8s | |