| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-06-06 | |||
| 16:33:41 | gibi | there is some nova related question in the etherpad | |
| 16:34:24 | gibi | about getting notified if anything changed with servers | |
| 16:34:30 | gibi | I offered the nova notification inteface | |
| 16:34:35 | bauzas | yeah and it's a public API :) | |
| 16:34:38 | gibi | but apperantly they want something public | |
| 16:34:58 | sean-k-mooney | the notifcation are the only interface we have currently | |
| 16:34:58 | gibi | the API is documented but the message bus access then to be non public | |
| 16:35:03 | bauzas | since someone worked on notifications objects like 6 years ago (guess who :p ) | |
| 16:35:14 | gibi | bauzas: hah :D | |
| 16:35:24 | bauzas | gibi: are you sure that the message bus can't be public ? | |
| 16:35:33 | dansmith | it shouldn't be | |
| 16:35:39 | gibi | yeah | |
| 16:35:43 | dansmith | we have instance events.. that's what they want I think | |
| 16:35:48 | sean-k-mooney | the notification bus is semi privladged | |
| 16:35:50 | gibi | and our notifications tend to contain infra information | |
| 16:35:52 | dansmith | I think they just need an async way to get those | |
| 16:35:58 | sean-k-mooney | the notifications can have private infor in them | |
| 16:36:02 | sean-k-mooney | depening on what you configure | |
| 16:36:04 | sean-k-mooney | like the bdms | |
| 16:36:30 | gibi | yeah, bottom line the notification API is designed for consumed by admins or other openstack service not endusers | |
| 16:36:43 | bauzas | ah they want it to be consumable by endusers ? | |
| 16:36:43 | dansmith | right, it would leak bad things between tenants for sure | |
| 16:36:47 | dansmith | not just infra things | |
| 16:36:52 | gibi | yeah | |
| 16:36:58 | bauzas | urth | |
| 16:37:01 | bauzas | urgh | |
| 16:37:01 | sean-k-mooney | you could have a multi tentant service that converts the notificaton in to a webhook callback or similar | |
| 16:37:09 | dansmith | instance events/actions is the right thing I think, it's just only polling currently | |
| 16:37:13 | sean-k-mooney | but that still not greate | |
| 16:37:28 | sean-k-mooney | dansmith: ya the event stream would work but im not sure that | |
| 16:37:37 | gibi | yeah, a websocket around instance actions wouldbe nice | |
| 16:37:41 | sean-k-mooney | even if it was event based they woud lwant ot liste per instance | |
| 16:37:53 | sean-k-mooney | more liek open a websocket and get all instance events for a project? | |
| 16:38:09 | dansmith | they probably want to be able to register a handler with a scope (one instance, all my instances) that lives for a period of time that we call when there's a new event | |
| 16:38:14 | sean-k-mooney | or that you are allowed to see based on teh scope of the keystone token | |
| 16:38:17 | bauzas | I think everytime someone asks us to monitor some instance action, we tell them 'lookup the notifications' | |
| 16:38:25 | bauzas | but this is for admin usage | |
| 16:38:29 | dansmith | websocket will require a lot of standby resources that I think would be hard for us to manage | |
| 16:38:45 | gibi | true | |
| 16:38:45 | dansmith | anyway, | |
| 16:38:58 | bauzas | so, they want some enduser public subscription mechanism for asynchronously being notified on my instance state changes ? | |
| 16:39:00 | dansmith | not sure how many people will be there to make any sort of headway on that topic, since I think those people are likely here :) | |
| 16:39:11 | dansmith | bauzas: yeah | |
| 16:39:24 | bauzas | sounds a client thing to mre | |
| 16:39:26 | bauzas | me | |
| 16:39:31 | gibi | dansmith: I will be there hence collecting ideas here now :) | |
| 16:39:32 | sean-k-mooney | lets see if they acan at least expand on the usecases | |
| 16:39:58 | dansmith | bauzas: a client can poll (or long poll) but that's much less efficient | |
| 16:39:59 | gibi | yeah I will pull out some specific use case and try to limit the scope to something very simple on our side | |
| 16:40:04 | dansmith | especially when instance actions could be days apart | |
| 16:40:21 | bauzas | but yeah, someone could provide some tool that would listen to the notification bus and scramples all the admin-only data | |
| 16:40:29 | gibi | bauzas: exactly | |
| 16:40:39 | bauzas | sorry, by client I meant something unrelated to nova | |
| 16:40:45 | sean-k-mooney | so the way to do this in the past was ceilometer put the relevetn events in AODH | |
| 16:40:51 | dansmith | they could, but that's basically re-constructing the tenant isolation that nova already has, so it's a big new surface to secure and new services to run | |
| 16:40:54 | sean-k-mooney | and then you woudl set up alarms on the events you cared about | |
| 16:41:15 | dansmith | sean-k-mooney: that's all intended to be operator-focused, not for users to get status/events on their instances right? | |
| 16:41:27 | sean-k-mooney | no | |
| 16:41:34 | gibi | if they only need a trigger to re-read the instance action API then most of of the data can be hidden from our notifications | |
| 16:41:39 | sean-k-mooney | aodh and celoimeter provided user facing events/metrics | |
| 16:41:51 | dansmith | okay I didn't realize | |
| 16:41:52 | bauzas | yeah | |
| 16:41:53 | sean-k-mooney | they didnt actully expose the full notificaiton | |
| 16:42:01 | bauzas | ceilometer was the fit | |
| 16:42:04 | sean-k-mooney | jsut instance boot started and instance boot finsined events | |
| 16:42:19 | bauzas | and that's why we never had this in nova | |
| 16:42:27 | dansmith | honestly, I feel like this is probably something nova can/should be doing | |
| 16:42:40 | dansmith | nowadays this is how stuff plugs together | |
| 16:42:51 | sean-k-mooney | ya i think its something we coudl do | |
| 16:42:56 | sean-k-mooney | but we need to think about how | |
| 16:42:58 | dansmith | making an external tool reconstruct what we already know is kinda :/ | |
| 16:43:18 | gibi | yeah | |
| 16:43:24 | dansmith | it could be a service like console that you run if you want, and scale separately to handle the amoun tof load you want to tolerate | |
| 16:43:31 | bauzas | dansmith: I'm still struggling to find how we would ensure the tenancy isolation by the message bus, but I'm open to ideas | |
| 16:43:45 | bauzas | unless we create a bus per tenant | |
| 16:43:46 | sean-k-mooney | if its in nova | |
| 16:43:47 | dansmith | bauzas: we wouldn't? | |
| 16:43:48 | sean-k-mooney | we can just filter | |
| 16:44:36 | bauzas | I'm maybe misunderstanding the proposal, but I thought we were about saying that we may emit project-related notifications | |
| 16:44:46 | dansmith | not at the rabbit level | |
| 16:44:51 | dansmith | let's let gibi collect some data, | |
| 16:44:59 | dansmith | and then we probably need a high-bandwidth conversation about options | |
| 16:45:01 | gibi | bauzas: we def need to understand their use case better | |
| 16:45:02 | bauzas | dansmith: ah ok | |
| 16:45:18 | bauzas | dansmith: then we need to construct some HTTP/2 layer with keystone auth | |
| 16:45:33 | bauzas | or something like that | |
| 16:45:38 | dansmith | bauzas: not necessarily | |
| 16:45:56 | dansmith | it just depends.. but it should be HTTP-something, either an event stream or callbacks | |
| 16:46:07 | sean-k-mooney | you would do somethign like "openstack project event subsribe (instnace.action.)*" which woudl return a websock url that would only stream the relevent events for the current project based on the keystone token | |
| 16:46:27 | dansmith | yep, could be something like that | |
| 16:46:38 | bauzas | sounds very console-ish | |
| 16:46:40 | sean-k-mooney | so like the console you woudl first create it and then get a handel for where to collect the data | |
| 16:46:42 | bauzas | but okay | |
| 16:46:44 | gibi | I can simplify it down to give me a server uuid and I stream you data about notifications affecting the server, but only with very limited data provided | |
| 16:46:58 | dansmith | bauzas: exactly.. it's the same sort of arrangement, and the same target audience | |
| 16:47:38 | bauzas | ok, then it sounds we have an agreement on the direction, let's not overpaper the technical details | |
| 16:47:42 | sean-k-mooney | and if its a seperate binary nova-event-proxy | |
| 16:47:53 | dansmith | gibi: yeah I just don't think you should need 100 websockets you have to read from if you have 100 instances in your wordpress deployment | |
| 16:47:58 | dansmith | sean-k-mooney: right | |
| 16:48:02 | sean-k-mooney | then its scalablity and wether its deployed is up ot the operator | |