| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-11-22 | |||
| 14:18:00 | sean-k-mooney | s3rj1k: for what its worth we did consier automating it in a k8s chron job | |
| 14:18:09 | s3rj1k | what about periodics? I mean we should decide what we should do with it, having this around and saying that it should not be used looks a bit off | |
| 14:18:26 | sean-k-mooney | that not what we say | |
| 14:18:40 | sean-k-mooney | we say it can be used but must only be enabled in one schduler instance | |
| 14:18:55 | sean-k-mooney | and htat for large deployment its avised to not run it frequently | |
| 14:19:13 | s3rj1k | same a basically setting up plain Linux cron with CLI | |
| 14:19:19 | sean-k-mooney | but yes we coudl deprecate it for removal, but i dont think we want to break existing users | |
| 14:19:42 | sean-k-mooney | s3rj1k: yes and no. the delta is that its ha | |
| 14:20:03 | sean-k-mooney | i.e. as long asa you k8s cluster is functional it can schduler to any approate worker | |
| 14:20:25 | s3rj1k | I mean in case of single node setups, cron vs periodic is same thin, no? | |
| 14:20:29 | sean-k-mooney | the reason we didnt do that of our installer is we did not think this was a good default for production | |
| 14:20:52 | sean-k-mooney | sure but then you cant have the conflcit | |
| 14:21:29 | sean-k-mooney | unless your running multipel schduler on a single node cluster? doable but proably not super useful | |
| 14:22:26 | s3rj1k | yea, point is case env setups what are ok to have periodics enabled are not HA at all and there is no point in supporting periodics in case of not also supporting HA with it, since plain cron can replace it on non HA | |
| 14:23:10 | s3rj1k | so effectively dead code in terms of production, fine for small setups only or dev clusters | |
| 14:23:41 | sean-k-mooney | so where this is used in production today its not enabeld on multiple hosts | |
| 14:23:53 | sean-k-mooney | but i agree its not as useful as it could be | |
| 14:24:31 | sean-k-mooney | s3rj1k: i dont think the request for enhancement is invlaid but pulling in tooz for jsut this is a hard cell | |
| 14:25:21 | sean-k-mooney | *sell | |
| 14:25:50 | sean-k-mooney | we generally want the scheuler ot be stateless and nova does not use any distibuted lockign today partly for horizontal scaleableity | |
| 14:26:38 | sean-k-mooney | we may be able to do https://en.wikipedia.org/wiki/Rendezvous_hashing | |
| 14:27:36 | sean-k-mooney | so that only one schdluler will try to map a given host on any given invocation but that woudl make the time to mapping less deterministic so im conflicted on that | |
| 14:39:05 | sean-k-mooney | s3rj1k: for right now im going to set the bug to Opinion but i think it would be good to either bring this up in the next team metting, ask others for input on irc or start a mailing list thread on this topic | |
| 14:39:49 | sean-k-mooney | s3rj1k: at a minium we could imporve the docuemation aroudn this | |
| 14:40:19 | sean-k-mooney | we may be able to do more but adding a distribute lock manager to nova is definally a feature not a bug | |
| 14:42:42 | s3rj1k | yea, feature, tagged as rfe | |
| 14:42:48 | s3rj1k | for sure nit a bug | |
| 14:42:54 | sean-k-mooney | another approch might be to catch the "host is not mapped to a cell" excption w eget today and use that to trigger a mapping reactively | |
| 14:44:34 | s3rj1k | also would like to note that this is no k8s specific and oldschool phys servers HA setup is probably even more impacted by this | |
| 14:45:07 | sean-k-mooney | sure as i said this is an old topic | |
| 14:45:36 | sean-k-mooney | the tension is that this really only make sense in an env wehre the number of compute nodes changes frequently | |
| 14:45:51 | sean-k-mooney | in such a env setting along interval is not palatable | |
| 14:45:59 | sean-k-mooney | so you have to set the period to be run often | |
| 14:46:21 | sean-k-mooney | which mean it adds a lot of load espcitlly if you run multipel isntances of the scduelr each doing it on the db | |
| 14:47:53 | sean-k-mooney | so the more frequently you add comptues, the more often you have to run it, and the more load it creates, which also gets more expensive the more host you have as there are more hosts to process | |
| 14:48:49 | sean-k-mooney | s3rj1k: in our operator we orgingally hoped to have the golang reconsile loop run the host mapping when compute were added | |
| 14:49:11 | sean-k-mooney | but architecturla design reasons prevent that form happing today | |
| 14:49:22 | sean-k-mooney | s3rj1k: im not sure if you have the same desgin limistiaon | |
| 14:50:08 | s3rj1k | I guess that the baremininum thing is to ensure that CLI Jobs are running sequentially | |
| 14:50:53 | sean-k-mooney | tl;dr our nova-operator is not allowed to monitor the addtion/removal of compute nodes, and our compute node deployment code is not allowed to acess the db (run nova-manage) or talk to k8s | |
| 14:51:07 | s3rj1k | but for that it would be nice to have some more visible descision around internal periodics, as they always come up | |
| 14:52:14 | sean-k-mooney | s3rj1k: melwitt had a propsoal for a seperate nova-audit deamon a long time ago to have a seperate process dedicated to running them | |
| 14:52:58 | sean-k-mooney | we did not move forward with that but the idea wwas to move this perodic and the db archive/peruge operatiosn to that seperate optional demaon | |
| 14:54:12 | sean-k-mooney | the idea was to centralise all of the perodic mantance and self healign task into a seperatly deployable but optional compoent | |
| 14:54:42 | s3rj1k | and how to handle HA in this case? | |
| 14:55:08 | sean-k-mooney | intially active passive so in k8s you would only run 1 as deplyment/statefulset | |
| 14:55:34 | sean-k-mooney | for non k8s deploymetsn you would use pacemaker ectra to manage which one was running | |
| 14:55:48 | sean-k-mooney | you could use a DLM as well | |
| 14:56:10 | sean-k-mooney | but given this is not a performace critical part that is less concerning | |
| 14:59:38 | s3rj1k | so a lot of extra things to hack this up in case of plain servers | |
| 15:00:09 | s3rj1k | for k8s case, active/passive could probably work | |
| 15:00:43 | sean-k-mooney | yep as i siad this is old ground but there is a renewd interst in makign nova be more "ks8 native" | |
| 15:01:31 | sean-k-mooney | s3rj1k: This was the last time nova-audit was serisoly dicussed https://specs.openstack.org/openstack/nova-specs/specs/xena/approved/nova-audit.html | |
| 15:02:06 | sean-k-mooney | s3rj1k: we discussed soem other k8s painpoint like graceful shutdown too at the last ptg | |
| 15:02:13 | sean-k-mooney | and healthchecks | |
| 15:02:49 | s3rj1k | all of those would be very nice to have, yes | |
| 15:02:52 | sean-k-mooney | so i intentilaly didnt close https://bugs.launchpad.net/nova/+bug/2089386 as invialid, but hostircally we have confired issues like this as out of scope | |
| 15:04:02 | sean-k-mooney | s/confired/considered/ | |
| 15:04:52 | sean-k-mooney | s3rj1k: i need to go finsih something but do bring this up in the next irc meeting or start a mailing list thrad about how to move this forward to get input form others | |
| 15:06:22 | s3rj1k | sure, let's let it brew over a weekend, I'll ping back on this in next meetup | |
| 15:06:34 | s3rj1k | have a nice weekend! | |
| 15:06:56 | sean-k-mooney | o/ | |
| 15:11:58 | opendevreview | ribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (manila abstraction) https://review.opendev.org/c/openstack/nova/+/831194 | |
| 15:11:58 | opendevreview | ribaudr proposed openstack/nova master: Use client token when talking to manila https://review.opendev.org/c/openstack/nova/+/925277 | |
| 15:11:59 | opendevreview | ribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (drivers and compute manager part) https://review.opendev.org/c/openstack/nova/+/833090 | |
| 15:11:59 | opendevreview | ribaudr proposed openstack/nova master: Mounting the shares as part of the initialization process https://review.opendev.org/c/openstack/nova/+/880075 | |
| 15:12:00 | opendevreview | ribaudr proposed openstack/nova master: Deletion of associated share mappings on instance deletion https://review.opendev.org/c/openstack/nova/+/881472 | |
| 15:12:00 | opendevreview | ribaudr proposed openstack/nova master: Add metadata for shares https://review.opendev.org/c/openstack/nova/+/850500 | |
| 15:12:01 | opendevreview | ribaudr proposed openstack/nova master: Add share_info parameter to reboot method for each driver (driver part) https://review.opendev.org/c/openstack/nova/+/854823 | |
| 15:12:01 | opendevreview | ribaudr proposed openstack/nova master: Support rebooting an instance with shares (compute manager part) https://review.opendev.org/c/openstack/nova/+/854824 | |
| 15:12:02 | opendevreview | ribaudr proposed openstack/nova master: Add share_info parameter to resume method for each driver (driver part) https://review.opendev.org/c/openstack/nova/+/860284 | |
| 15:12:02 | opendevreview | ribaudr proposed openstack/nova master: Support resuming an instance with shares (compute manager part) https://review.opendev.org/c/openstack/nova/+/860285 | |
| 15:12:04 | opendevreview | ribaudr proposed openstack/nova master: Add helper methods to rescue/unrescue shares https://review.opendev.org/c/openstack/nova/+/860286 | |
| 15:12:04 | opendevreview | ribaudr proposed openstack/nova master: Support rescuing an instance with shares https://review.opendev.org/c/openstack/nova/+/860287 | |
| 15:12:06 | opendevreview | ribaudr proposed openstack/nova master: Allow to mount manila share using Cephfs protocol https://review.opendev.org/c/openstack/nova/+/883862 | |
| 15:12:06 | opendevreview | ribaudr proposed openstack/nova master: Check shares support (compute manager) https://review.opendev.org/c/openstack/nova/+/885751 | |
| 15:12:08 | opendevreview | ribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (API) https://review.opendev.org/c/openstack/nova/+/836830 | |
| 15:12:08 | opendevreview | ribaudr proposed openstack/nova master: Add helper methods to attach/detach shares https://review.opendev.org/c/openstack/nova/+/885753 | |
| 15:12:10 | opendevreview | ribaudr proposed openstack/nova master: Add instance.share_attach notification https://review.opendev.org/c/openstack/nova/+/850501 | |
| 15:12:10 | opendevreview | ribaudr proposed openstack/nova master: Add instance.share_detach notification https://review.opendev.org/c/openstack/nova/+/851028 | |
| 15:12:12 | opendevreview | ribaudr proposed openstack/nova master: Add shares to InstancePayload https://review.opendev.org/c/openstack/nova/+/851029 | |
| 15:12:12 | opendevreview | ribaudr proposed openstack/nova master: Add instance.share_attach_error notification https://review.opendev.org/c/openstack/nova/+/860282 | |
| 15:12:14 | opendevreview | ribaudr proposed openstack/nova master: Add instance.share_detach_error notification https://review.opendev.org/c/openstack/nova/+/860283 | |
| 15:12:14 | opendevreview | ribaudr proposed openstack/nova master: Reports instance events to the DB regarding attaching and detaching a share https://review.opendev.org/c/openstack/nova/+/927088 | |
| 15:12:16 | opendevreview | ribaudr proposed openstack/nova master: Add libvirt test to ensure metadata are working. https://review.opendev.org/c/openstack/nova/+/852086 | |
| 15:12:16 | opendevreview | ribaudr proposed openstack/nova master: Add virt/libvirt error test cases https://review.opendev.org/c/openstack/nova/+/852087 | |
| 15:12:18 | opendevreview | ribaudr proposed openstack/nova master: Manila shares admin guide documentation https://review.opendev.org/c/openstack/nova/+/871642 | |
| 15:12:18 | opendevreview | ribaudr proposed openstack/nova master: Refactor test_server_shares: Mock in Base Class and trait verification https://review.opendev.org/c/openstack/nova/+/935861 | |
| 15:55:58 | opendevreview | Elod Illes proposed openstack/nova stable/2024.1: [stable-only] Set grenade-skip-level to non-voting https://review.opendev.org/c/openstack/nova/+/936033 | |
| 16:53:01 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for server action APIs https://review.opendev.org/c/openstack/nova/+/915742 | |
| 16:53:02 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for remaining server action APIs https://review.opendev.org/c/openstack/nova/+/915743 | |
| 16:53:02 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for host aggregate actions API https://review.opendev.org/c/openstack/nova/+/924584 | |
| 16:53:03 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for host aggregate APIs https://review.opendev.org/c/openstack/nova/+/924585 | |
| 16:53:03 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for extensions API https://review.opendev.org/c/openstack/nova/+/924586 | |
| 16:53:04 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for versions APIs https://review.opendev.org/c/openstack/nova/+/924587 | |
| 16:53:04 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for assisted volume snapshots APIs https://review.opendev.org/c/openstack/nova/+/924588 | |
| 16:53:05 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for port interface APIs https://review.opendev.org/c/openstack/nova/+/924589 | |
| 16:53:05 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for availability zone APIs https://review.opendev.org/c/openstack/nova/+/924590 | |
| 16:53:06 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for bare metal node APIs https://review.opendev.org/c/openstack/nova/+/924591 | |
| 16:53:06 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for console auth token APIs https://review.opendev.org/c/openstack/nova/+/924592 | |