| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-04-13 | |||
| 16:54:54 | Uggla | I will notify on the ML too | |
| 16:55:36 | Uggla | any last 5 minutes topic to discuss ? | |
| 16:57:08 | phildree[m] | Hello guys, I am not sure if this is the right place and time to bring this up because i am new to OpenStack, but I have created a spec and am looking for some feedback: https://review.opendev.org/c/openstack/nova-specs/+/984384 | |
| 16:58:07 | Uggla | phildree[m], can you add a topic in the ptg etherpad ? then I'll define a slot for you next week. | |
| 16:58:32 | phildree[m] | Ok, thank you :) | |
| 16:58:54 | Uggla | any "prefered" timing ? | |
| 16:59:59 | phildree[m] | Nope | |
| 17:00:22 | Uggla | ok | |
| 17:00:27 | Uggla | so I think we are good. | |
| 17:01:00 | Uggla | time to close, thanks for joining this meeting. Have a nice day/evening and see you next week for the vPTG. | |
| 17:01:07 | opendevmeet | Log: https://meetings.opendev.org/meetings/nova/2026/nova.2026-04-13-16.02.log.html | |
| 17:01:07 | opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/nova/2026/nova.2026-04-13-16.02.txt | |
| 17:01:07 | opendevmeet | Minutes: https://meetings.opendev.org/meetings/nova/2026/nova.2026-04-13-16.02.html | |
| 17:01:07 | opendevmeet | Meeting ended Mon Apr 13 17:01:07 2026 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | |
| 17:01:07 | Uggla | #endmeeting | |
| 17:01:11 | fwiesel | o/ thanks everyone | |
| 17:01:17 | elodilles | see you next week o/ | |
| 17:01:24 | sean-k-mooney | phildree[m]: have you talked to oslo about his | |
| 17:01:52 | phildree[m] | No, this IRC is my first point of contact for now | |
| 17:01:56 | sean-k-mooney | phildree[m]: in generally iw oudl prefer if we removed the if we didnt have to have any aplciation level handchakes | |
| 17:02:24 | gibi | thanks | |
| 17:03:21 | sean-k-mooney | we have some existign rpc pings but anything that makes the conductor comptue messageing nosyier by addign background messages direct hurts scalableity fo nova | |
| 17:03:46 | phildree[m] | sean-k-mooney: Ok, I understand. Maybe handshake is a bit much for what i have proposed, its more like an echo | |
| 17:04:11 | sean-k-mooney | right im saying even adding a echo/ping is protically a scaling issues | |
| 17:04:18 | phildree[m] | sean-k-mooney: I have seen the existing ping but it is not applicable to our usecase (I explained why in the spec) | |
| 17:05:00 | phildree[m] | sean-k-mooney: yes you are totally right, I get that. Any other ideas on how to solve the problem we have been facing? | |
| 17:05:30 | opendevreview | Merged openstack/nova stable/2025.1: Fix fill_metadata usage for the ImagePropertiesWeigher https://review.opendev.org/c/openstack/nova/+/964291 | |
| 17:05:36 | sean-k-mooney | i have not read your spec yet but we gerneally perfer to start with the usecase/probelm | |
| 17:05:47 | sean-k-mooney | before going to a feature/design chagne | |
| 17:07:01 | sean-k-mooney | phildree[m]: the prviosu solution which we never implmetned was per binary healthcheck endponts on each nova service | |
| 17:08:21 | sean-k-mooney | phildree[m]: i think teh issue your having is something the outbount rabbit conenction is fucntional | |
| 17:08:28 | sean-k-mooney | but the inbound one is not correct | |
| 17:08:49 | sean-k-mooney | i.e. the compute can call the conductor but the conductor cannot sucessfully call the computes | |
| 17:09:14 | phildree[m] | Yes, we faced this exact issue. | |
| 17:09:33 | sean-k-mooney | do you have the guareented message delivery flag enabled in oslo | |
| 17:09:36 | phildree[m] | I will think about the other approach with individual per binary healthcheck endpoints | |
| 17:10:13 | phildree[m] | That is a good question, which I am not able to answer right now | |
| 17:10:13 | sean-k-mooney | well we didnt impment that or at least we didnt merge it | |
| 17:11:11 | melwitt | bauzas: hi, there are two vtpm related patches that are simple and need a second review if you could get a chance plz https://review.opendev.org/c/openstack/nova/+/978836 and https://review.opendev.org/c/openstack/nova/+/983504 | |
| 17:11:19 | sean-k-mooney | phildree[m]: https://docs.openstack.org/nova/latest/configuration/config.html#oslo_messaging_rabbit.direct_mandatory_flag | |
| 17:11:25 | sean-k-mooney | i woudl check you have that set to true | |
| 17:11:32 | phildree[m] | My issues with the per binary healthcheck is that I still need a way to ensure that full communication from conductor to compute (and back) is up and running, which i cannot check without any actual communication, I think | |
| 17:11:51 | sean-k-mooney | phildree[m]: what that will detect is fi the per compute queue is present | |
| 17:12:02 | phildree[m] | Ok very nice, I will look into it, thank you for the infos :) | |
| 17:12:18 | sean-k-mooney | phildree[m]: well the whole point of the per binary healthcheck si to not use rabbit or the conductor | |
| 17:12:38 | sean-k-mooney | it to move the health moditoring out fo that code path to an external http endpoint | |
| 17:12:51 | sean-k-mooney | that you can have prometheous or whatever scrape instead | |
| 17:14:45 | phildree[m] | I get it, and it makes sense, but then there is no way to check the health of the communication (and the underlying infrastructure) between conductor and compute (only the individual binaries) | |
| 17:14:45 | sean-k-mooney | phildree[m]: the direct_mandatory_flag tuns a slient failrue (the messag eis encued to a new queue that nothign is listenting too) into a driect rpc call failure and failure in teh api | |
| 17:15:41 | sean-k-mooney | right today we do that via the resouce update whcih is a push model | |
| 17:16:18 | sean-k-mooney | its worth a dicussion i guess but my main concern is we dont really run perodic form the conductor to monitor the satues of other agents | |
| 17:16:49 | sean-k-mooney | and we try not to do monitoring in general in nova. so this woudl likely need to be confiugrable and off by defualt | |
| 17:16:57 | sean-k-mooney | at least until we know the overhad of it | |
| 17:17:58 | phildree[m] | I think I have to read up on what you have sent first before I may rethink the idea. Thanks a lot for the infos so far, that was very helpfull and a lot less complicated then i thought :) | |
| 17:17:58 | phildree[m] | I have touched many points of what you have brought up, but as I said, I am not yet very familiar with OpenStack so maybe there are some issues I did not anticipate. | |
| 17:19:14 | sean-k-mooney | phildree[m]: one thing we are currently changing | |
| 17:19:32 | sean-k-mooney | is we will nolonger have a slngle rpc listter per compute agent | |
| 17:19:47 | sean-k-mooney | we will now have two becase fo the graceful shutdown work | |
| 17:20:06 | sean-k-mooney | so gmaan woudl likely need to review your prosal in the context fo that work | |
| 17:20:24 | phildree[m] | Ok, that is something very important as this may have implications on my idea | |
| 17:20:42 | sean-k-mooney | with graceful shutdown we will be shutting down 1 of the 2 lstenter until the inprogress operation complete | |
| 17:21:02 | sean-k-mooney | so any kind fo monitoring would likely have to be on the one we dont shutdown | |
| 17:22:08 | gmaan | sean-k-mooney: reading chat | |
| 17:22:20 | phildree[m] | At least for the shutdown period, yes | |
| 17:22:20 | sean-k-mooney | gmaan: context is https://review.opendev.org/c/openstack/nova-specs/+/984384/1/specs/2026.2/approved/bidirectional-rpc-liveness-handshake-for-compute-nodes.rst | |
| 17:22:50 | sean-k-mooney | phildree[m]: are you goign to present this at the ptg for dicussion? | |
| 17:23:31 | phildree[m] | I have to be honest, I first have to read up on what the PTG exactly is and how it works | |
| 17:24:15 | sean-k-mooney | ack its basically a workign session wehre user/operators/devleoper talk about our plans for the next release and pain points | |
| 17:24:51 | sean-k-mooney | it operats now as a virual event with an eterphad for notes and adgena and jitsi meet as the video confernce tool | |
| 17:25:01 | sean-k-mooney | its free to attend. they used ot be inperson events | |
| 17:25:15 | phildree[m] | Then this would probably make sense from my point of view (even if its just to gain a better understanding of the whole project) | |
| 17:25:23 | sean-k-mooney | if you cant attend thats fine too. | |
| 17:25:42 | gmaan | phildree[m]: I agree with sean-k-mooney that periodic handshake over RPC is expensive especially in scale env | |
| 17:26:26 | sean-k-mooney | gmaan: im kind of wondering if we can maybke marke the node as down sooner by monitoring failed/timedout rpc calls of soemthing liek that | |
| 17:26:35 | gmaan | I also have not read the spec but does compute service up check not enough? that is kind of periodic check if service is down or not | |
| 17:26:44 | sean-k-mooney | i.e. only do a condotor to compute ping in that case to confirm the state | |
| 17:27:26 | sean-k-mooney | gmaan: there was at least one case in the psat where the comptue service is up but the rabbit queue was deleted | |
| 17:27:45 | sean-k-mooney | so it coudl send rpc calles like the status update but not recive them | |
| 17:27:48 | gmaan | " failed/timedout rpc calls" might not give actual result and can end up in false result. | |
| 17:27:59 | sean-k-mooney | https://docs.openstack.org/nova/latest/configuration/config.html#oslo_messaging_rabbit.direct_mandatory_flag was partly added to prevent that | |
| 17:28:06 | phildree[m] | gmaan: I think if you do the service up check it relies on the DB entries based on heartbeats which is exactly what our issues was because the heartbeat may keep working while the requests from conductor to compute failed | |
| 17:28:15 | gmaan | yeah not RPC | |
| 17:28:47 | phildree[m] | Just like sean-k-mooney just described | |
| 17:29:26 | gmaan | yeah, that is true and existing challenge | |
| 17:30:38 | sean-k-mooney | phildree[m]: the reall issue is take an enve with 500 nodes and 3 conductors. we technically ahve a leader election fetrue in the conductor but if we didnt use that ever 300 second (our default perodic interval) we would send 1500 messages 3 form each conduxtor x 500 compute nodes | |
| 17:30:54 | gmaan | so what is actual issue? conductor to compute RPC call fail the it give status to check if compute service is up or not if yes then RPC is down right? | |
| 17:31:34 | sean-k-mooney | in rare cases the compute can be up enough for the status to be up but a call to the compute form the api will fail to boot a vm | |
| 17:31:52 | phildree[m] | In my proposal the handshake is compute initiated in order to avoid leader stuff and states in the conductor | |
| 17:32:37 | sean-k-mooney | phildree[m]: well we typiclly have 3-5 conductors per cell and can have 300-1500 compute per cell in large clouds | |
| 17:33:36 | gmaan | I am not saying 'service up' is reliable source but my question is do we want to automate it in conductor to check compute before call otherwise asl sch to select other dest? or just for clear troubleshooting and error message that what exactly is down? | |
| 17:33:49 | gmaan | s/asl/ask | |
| 17:34:55 | sean-k-mooney | gmaan: ya im not sayign we shoudl do this im just playing devils advocate | |
| 17:35:45 | gmaan | sean-k-mooney: or the healthcheck proposal is good fit here but is the latest proposal check RPC health also? | |
| 17:35:55 | phildree[m] | But in order to reduce RPCs i think also just a periodic ping without compute base initiation might be feasible but will have the issue of coordination etc.. | |
| 17:35:55 | phildree[m] | This way when any part of the communction is broken the node does not appear as up | |
| 17:35:55 | phildree[m] | it like a tcp Syn (compute send a request to any conductor), Syn-Ack (conductor replies to that tiny request), Ack (compute initiates the regular heartbeat as is) | |
| 17:36:21 | sean-k-mooney | gmaan: it was goign to catch connection closed events and similar on the comptue and report if the rpc bus was working | |
| 17:36:26 | sean-k-mooney | but not by active probing | |
| 17:36:34 | gmaan | k | |
| 17:36:34 | sean-k-mooney | only by recoreding if we got an excption | |