| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-04-13 | |||
| 16:49:15 | sean-k-mooney | https://review.opendev.org/c/openstack/oslo.vmware/+/982614 looks indepent of the executor issue we saw in the vmware job so this is just a general urllib3 issue not an eventet oen right | |
| 16:50:16 | sean-k-mooney | im asking because it does not have a bug filed for it and im wodnering if eventlet was masking it | |
| 16:51:20 | gibi | the executor issue was the deadlock on sending dependent tasks to the parent's executor we fixed that in general | |
| 16:51:28 | gibi | this bug apperad after | |
| 16:51:44 | gibi | and was visible as hanging in image tasks in the vmware driver | |
| 16:52:13 | sean-k-mooney | ack so it may have been there but not visable before | |
| 16:52:17 | gibi | yeah | |
| 16:52:25 | sean-k-mooney | i know evently also monkey patches urllib3 | |
| 16:52:35 | sean-k-mooney | so that also complciates things | |
| 16:52:43 | sean-k-mooney | ok we can likely move on | |
| 16:52:53 | Uggla | #topic Bug scrubbing | |
| 16:52:58 | Uggla | #info up to 197 (-2) | |
| 16:53:10 | Uggla | #link: https://etherpad.opendev.org/p/nova-bug-triage-roster | |
| 16:53:19 | Uggla | #link: https://truc.uggla.fr/ to follow the trend. | |
| 16:53:25 | Uggla | Uggla: This week [public] Upstream bug triage. Wednesday, April 15 · 15:30 – 16:00 UTC. Video call link: https://meet.google.com/kmq-eedg-auj | |
| 16:53:47 | Uggla | oops forget to say : Skipping Layos topic as he looks not around and we are a bit late on schedule | |
| 16:54:17 | Uggla | #topic Open discussion | |
| 16:54:25 | Uggla | Uggla: Suggest no meeting next week. 20th April vPTG week. | |
| 16:54:38 | gibi | +1 | |
| 16:54:54 | Uggla | I will notify on the ML too | |
| 16:54:54 | elodilles | +1 | |
| 16:55:36 | Uggla | any last 5 minutes topic to discuss ? | |
| 16:57:08 | phildree[m] | Hello guys, I am not sure if this is the right place and time to bring this up because i am new to OpenStack, but I have created a spec and am looking for some feedback: https://review.opendev.org/c/openstack/nova-specs/+/984384 | |
| 16:58:07 | Uggla | phildree[m], can you add a topic in the ptg etherpad ? then I'll define a slot for you next week. | |
| 16:58:32 | phildree[m] | Ok, thank you :) | |
| 16:58:54 | Uggla | any "prefered" timing ? | |
| 16:59:59 | phildree[m] | Nope | |
| 17:00:22 | Uggla | ok | |
| 17:00:27 | Uggla | so I think we are good. | |
| 17:01:00 | Uggla | time to close, thanks for joining this meeting. Have a nice day/evening and see you next week for the vPTG. | |
| 17:01:07 | Uggla | #endmeeting | |
| 17:01:07 | opendevmeet | Meeting ended Mon Apr 13 17:01:07 2026 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | |
| 17:01:07 | opendevmeet | Minutes: https://meetings.opendev.org/meetings/nova/2026/nova.2026-04-13-16.02.html | |
| 17:01:07 | opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/nova/2026/nova.2026-04-13-16.02.txt | |
| 17:01:07 | opendevmeet | Log: https://meetings.opendev.org/meetings/nova/2026/nova.2026-04-13-16.02.log.html | |
| 17:01:11 | fwiesel | o/ thanks everyone | |
| 17:01:17 | elodilles | see you next week o/ | |
| 17:01:24 | sean-k-mooney | phildree[m]: have you talked to oslo about his | |
| 17:01:52 | phildree[m] | No, this IRC is my first point of contact for now | |
| 17:01:56 | sean-k-mooney | phildree[m]: in generally iw oudl prefer if we removed the if we didnt have to have any aplciation level handchakes | |
| 17:02:24 | gibi | thanks | |
| 17:03:21 | sean-k-mooney | we have some existign rpc pings but anything that makes the conductor comptue messageing nosyier by addign background messages direct hurts scalableity fo nova | |
| 17:03:46 | phildree[m] | sean-k-mooney: Ok, I understand. Maybe handshake is a bit much for what i have proposed, its more like an echo | |
| 17:04:11 | sean-k-mooney | right im saying even adding a echo/ping is protically a scaling issues | |
| 17:04:18 | phildree[m] | sean-k-mooney: I have seen the existing ping but it is not applicable to our usecase (I explained why in the spec) | |
| 17:05:00 | phildree[m] | sean-k-mooney: yes you are totally right, I get that. Any other ideas on how to solve the problem we have been facing? | |
| 17:05:30 | opendevreview | Merged openstack/nova stable/2025.1: Fix fill_metadata usage for the ImagePropertiesWeigher https://review.opendev.org/c/openstack/nova/+/964291 | |
| 17:05:36 | sean-k-mooney | i have not read your spec yet but we gerneally perfer to start with the usecase/probelm | |
| 17:05:47 | sean-k-mooney | before going to a feature/design chagne | |
| 17:07:01 | sean-k-mooney | phildree[m]: the prviosu solution which we never implmetned was per binary healthcheck endponts on each nova service | |
| 17:08:21 | sean-k-mooney | phildree[m]: i think teh issue your having is something the outbount rabbit conenction is fucntional | |
| 17:08:28 | sean-k-mooney | but the inbound one is not correct | |
| 17:08:49 | sean-k-mooney | i.e. the compute can call the conductor but the conductor cannot sucessfully call the computes | |
| 17:09:14 | phildree[m] | Yes, we faced this exact issue. | |
| 17:09:33 | sean-k-mooney | do you have the guareented message delivery flag enabled in oslo | |
| 17:09:36 | phildree[m] | I will think about the other approach with individual per binary healthcheck endpoints | |
| 17:10:13 | sean-k-mooney | well we didnt impment that or at least we didnt merge it | |
| 17:10:13 | phildree[m] | That is a good question, which I am not able to answer right now | |
| 17:11:11 | melwitt | bauzas: hi, there are two vtpm related patches that are simple and need a second review if you could get a chance plz https://review.opendev.org/c/openstack/nova/+/978836 and https://review.opendev.org/c/openstack/nova/+/983504 | |
| 17:11:19 | sean-k-mooney | phildree[m]: https://docs.openstack.org/nova/latest/configuration/config.html#oslo_messaging_rabbit.direct_mandatory_flag | |
| 17:11:25 | sean-k-mooney | i woudl check you have that set to true | |
| 17:11:32 | phildree[m] | My issues with the per binary healthcheck is that I still need a way to ensure that full communication from conductor to compute (and back) is up and running, which i cannot check without any actual communication, I think | |
| 17:11:51 | sean-k-mooney | phildree[m]: what that will detect is fi the per compute queue is present | |
| 17:12:02 | phildree[m] | Ok very nice, I will look into it, thank you for the infos :) | |
| 17:12:18 | sean-k-mooney | phildree[m]: well the whole point of the per binary healthcheck si to not use rabbit or the conductor | |
| 17:12:38 | sean-k-mooney | it to move the health moditoring out fo that code path to an external http endpoint | |
| 17:12:51 | sean-k-mooney | that you can have prometheous or whatever scrape instead | |
| 17:14:45 | sean-k-mooney | phildree[m]: the direct_mandatory_flag tuns a slient failrue (the messag eis encued to a new queue that nothign is listenting too) into a driect rpc call failure and failure in teh api | |
| 17:14:45 | phildree[m] | I get it, and it makes sense, but then there is no way to check the health of the communication (and the underlying infrastructure) between conductor and compute (only the individual binaries) | |
| 17:15:41 | sean-k-mooney | right today we do that via the resouce update whcih is a push model | |
| 17:16:18 | sean-k-mooney | its worth a dicussion i guess but my main concern is we dont really run perodic form the conductor to monitor the satues of other agents | |
| 17:16:49 | sean-k-mooney | and we try not to do monitoring in general in nova. so this woudl likely need to be confiugrable and off by defualt | |
| 17:16:57 | sean-k-mooney | at least until we know the overhad of it | |
| 17:17:58 | phildree[m] | I have touched many points of what you have brought up, but as I said, I am not yet very familiar with OpenStack so maybe there are some issues I did not anticipate. | |
| 17:17:58 | phildree[m] | I think I have to read up on what you have sent first before I may rethink the idea. Thanks a lot for the infos so far, that was very helpfull and a lot less complicated then i thought :) | |
| 17:19:14 | sean-k-mooney | phildree[m]: one thing we are currently changing | |
| 17:19:32 | sean-k-mooney | is we will nolonger have a slngle rpc listter per compute agent | |
| 17:19:47 | sean-k-mooney | we will now have two becase fo the graceful shutdown work | |
| 17:20:06 | sean-k-mooney | so gmaan woudl likely need to review your prosal in the context fo that work | |
| 17:20:24 | phildree[m] | Ok, that is something very important as this may have implications on my idea | |
| 17:20:42 | sean-k-mooney | with graceful shutdown we will be shutting down 1 of the 2 lstenter until the inprogress operation complete | |
| 17:21:02 | sean-k-mooney | so any kind fo monitoring would likely have to be on the one we dont shutdown | |
| 17:22:08 | gmaan | sean-k-mooney: reading chat | |
| 17:22:20 | sean-k-mooney | gmaan: context is https://review.opendev.org/c/openstack/nova-specs/+/984384/1/specs/2026.2/approved/bidirectional-rpc-liveness-handshake-for-compute-nodes.rst | |
| 17:22:20 | phildree[m] | At least for the shutdown period, yes | |
| 17:22:50 | sean-k-mooney | phildree[m]: are you goign to present this at the ptg for dicussion? | |
| 17:23:31 | phildree[m] | I have to be honest, I first have to read up on what the PTG exactly is and how it works | |
| 17:24:15 | sean-k-mooney | ack its basically a workign session wehre user/operators/devleoper talk about our plans for the next release and pain points | |
| 17:24:51 | sean-k-mooney | it operats now as a virual event with an eterphad for notes and adgena and jitsi meet as the video confernce tool | |
| 17:25:01 | sean-k-mooney | its free to attend. they used ot be inperson events | |
| 17:25:15 | phildree[m] | Then this would probably make sense from my point of view (even if its just to gain a better understanding of the whole project) | |
| 17:25:23 | sean-k-mooney | if you cant attend thats fine too. | |
| 17:25:42 | gmaan | phildree[m]: I agree with sean-k-mooney that periodic handshake over RPC is expensive especially in scale env | |
| 17:26:26 | sean-k-mooney | gmaan: im kind of wondering if we can maybke marke the node as down sooner by monitoring failed/timedout rpc calls of soemthing liek that | |
| 17:26:35 | gmaan | I also have not read the spec but does compute service up check not enough? that is kind of periodic check if service is down or not | |
| 17:26:44 | sean-k-mooney | i.e. only do a condotor to compute ping in that case to confirm the state | |
| 17:27:26 | sean-k-mooney | gmaan: there was at least one case in the psat where the comptue service is up but the rabbit queue was deleted | |
| 17:27:45 | sean-k-mooney | so it coudl send rpc calles like the status update but not recive them | |
| 17:27:48 | gmaan | " failed/timedout rpc calls" might not give actual result and can end up in false result. | |
| 17:27:59 | sean-k-mooney | https://docs.openstack.org/nova/latest/configuration/config.html#oslo_messaging_rabbit.direct_mandatory_flag was partly added to prevent that | |