| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-04-13 | |||
| 17:07:01 | sean-k-mooney | phildree[m]: the prviosu solution which we never implmetned was per binary healthcheck endponts on each nova service | |
| 17:08:21 | sean-k-mooney | phildree[m]: i think teh issue your having is something the outbount rabbit conenction is fucntional | |
| 17:08:28 | sean-k-mooney | but the inbound one is not correct | |
| 17:08:49 | sean-k-mooney | i.e. the compute can call the conductor but the conductor cannot sucessfully call the computes | |
| 17:09:14 | phildree[m] | Yes, we faced this exact issue. | |
| 17:09:33 | sean-k-mooney | do you have the guareented message delivery flag enabled in oslo | |
| 17:09:36 | phildree[m] | I will think about the other approach with individual per binary healthcheck endpoints | |
| 17:10:13 | sean-k-mooney | well we didnt impment that or at least we didnt merge it | |
| 17:10:13 | phildree[m] | That is a good question, which I am not able to answer right now | |
| 17:11:11 | melwitt | bauzas: hi, there are two vtpm related patches that are simple and need a second review if you could get a chance plz https://review.opendev.org/c/openstack/nova/+/978836 and https://review.opendev.org/c/openstack/nova/+/983504 | |
| 17:11:19 | sean-k-mooney | phildree[m]: https://docs.openstack.org/nova/latest/configuration/config.html#oslo_messaging_rabbit.direct_mandatory_flag | |
| 17:11:25 | sean-k-mooney | i woudl check you have that set to true | |
| 17:11:32 | phildree[m] | My issues with the per binary healthcheck is that I still need a way to ensure that full communication from conductor to compute (and back) is up and running, which i cannot check without any actual communication, I think | |
| 17:11:51 | sean-k-mooney | phildree[m]: what that will detect is fi the per compute queue is present | |
| 17:12:02 | phildree[m] | Ok very nice, I will look into it, thank you for the infos :) | |
| 17:12:18 | sean-k-mooney | phildree[m]: well the whole point of the per binary healthcheck si to not use rabbit or the conductor | |
| 17:12:38 | sean-k-mooney | it to move the health moditoring out fo that code path to an external http endpoint | |
| 17:12:51 | sean-k-mooney | that you can have prometheous or whatever scrape instead | |
| 17:14:45 | sean-k-mooney | phildree[m]: the direct_mandatory_flag tuns a slient failrue (the messag eis encued to a new queue that nothign is listenting too) into a driect rpc call failure and failure in teh api | |
| 17:14:45 | phildree[m] | I get it, and it makes sense, but then there is no way to check the health of the communication (and the underlying infrastructure) between conductor and compute (only the individual binaries) | |
| 17:15:41 | sean-k-mooney | right today we do that via the resouce update whcih is a push model | |
| 17:16:18 | sean-k-mooney | its worth a dicussion i guess but my main concern is we dont really run perodic form the conductor to monitor the satues of other agents | |
| 17:16:49 | sean-k-mooney | and we try not to do monitoring in general in nova. so this woudl likely need to be confiugrable and off by defualt | |
| 17:16:57 | sean-k-mooney | at least until we know the overhad of it | |
| 17:17:58 | phildree[m] | I have touched many points of what you have brought up, but as I said, I am not yet very familiar with OpenStack so maybe there are some issues I did not anticipate. | |
| 17:17:58 | phildree[m] | I think I have to read up on what you have sent first before I may rethink the idea. Thanks a lot for the infos so far, that was very helpfull and a lot less complicated then i thought :) | |
| 17:19:14 | sean-k-mooney | phildree[m]: one thing we are currently changing | |
| 17:19:32 | sean-k-mooney | is we will nolonger have a slngle rpc listter per compute agent | |
| 17:19:47 | sean-k-mooney | we will now have two becase fo the graceful shutdown work | |
| 17:20:06 | sean-k-mooney | so gmaan woudl likely need to review your prosal in the context fo that work | |
| 17:20:24 | phildree[m] | Ok, that is something very important as this may have implications on my idea | |
| 17:20:42 | sean-k-mooney | with graceful shutdown we will be shutting down 1 of the 2 lstenter until the inprogress operation complete | |
| 17:21:02 | sean-k-mooney | so any kind fo monitoring would likely have to be on the one we dont shutdown | |
| 17:22:08 | gmaan | sean-k-mooney: reading chat | |
| 17:22:20 | sean-k-mooney | gmaan: context is https://review.opendev.org/c/openstack/nova-specs/+/984384/1/specs/2026.2/approved/bidirectional-rpc-liveness-handshake-for-compute-nodes.rst | |
| 17:22:20 | phildree[m] | At least for the shutdown period, yes | |
| 17:22:50 | sean-k-mooney | phildree[m]: are you goign to present this at the ptg for dicussion? | |
| 17:23:31 | phildree[m] | I have to be honest, I first have to read up on what the PTG exactly is and how it works | |
| 17:24:15 | sean-k-mooney | ack its basically a workign session wehre user/operators/devleoper talk about our plans for the next release and pain points | |
| 17:24:51 | sean-k-mooney | it operats now as a virual event with an eterphad for notes and adgena and jitsi meet as the video confernce tool | |
| 17:25:01 | sean-k-mooney | its free to attend. they used ot be inperson events | |
| 17:25:15 | phildree[m] | Then this would probably make sense from my point of view (even if its just to gain a better understanding of the whole project) | |
| 17:25:23 | sean-k-mooney | if you cant attend thats fine too. | |
| 17:25:42 | gmaan | phildree[m]: I agree with sean-k-mooney that periodic handshake over RPC is expensive especially in scale env | |
| 17:26:26 | sean-k-mooney | gmaan: im kind of wondering if we can maybke marke the node as down sooner by monitoring failed/timedout rpc calls of soemthing liek that | |
| 17:26:35 | gmaan | I also have not read the spec but does compute service up check not enough? that is kind of periodic check if service is down or not | |
| 17:26:44 | sean-k-mooney | i.e. only do a condotor to compute ping in that case to confirm the state | |
| 17:27:26 | sean-k-mooney | gmaan: there was at least one case in the psat where the comptue service is up but the rabbit queue was deleted | |
| 17:27:45 | sean-k-mooney | so it coudl send rpc calles like the status update but not recive them | |
| 17:27:48 | gmaan | " failed/timedout rpc calls" might not give actual result and can end up in false result. | |
| 17:27:59 | sean-k-mooney | https://docs.openstack.org/nova/latest/configuration/config.html#oslo_messaging_rabbit.direct_mandatory_flag was partly added to prevent that | |
| 17:28:06 | phildree[m] | gmaan: I think if you do the service up check it relies on the DB entries based on heartbeats which is exactly what our issues was because the heartbeat may keep working while the requests from conductor to compute failed | |
| 17:28:15 | gmaan | yeah not RPC | |
| 17:28:47 | phildree[m] | Just like sean-k-mooney just described | |
| 17:29:26 | gmaan | yeah, that is true and existing challenge | |
| 17:30:38 | sean-k-mooney | phildree[m]: the reall issue is take an enve with 500 nodes and 3 conductors. we technically ahve a leader election fetrue in the conductor but if we didnt use that ever 300 second (our default perodic interval) we would send 1500 messages 3 form each conduxtor x 500 compute nodes | |
| 17:30:54 | gmaan | so what is actual issue? conductor to compute RPC call fail the it give status to check if compute service is up or not if yes then RPC is down right? | |
| 17:31:34 | sean-k-mooney | in rare cases the compute can be up enough for the status to be up but a call to the compute form the api will fail to boot a vm | |
| 17:31:52 | phildree[m] | In my proposal the handshake is compute initiated in order to avoid leader stuff and states in the conductor | |
| 17:32:37 | sean-k-mooney | phildree[m]: well we typiclly have 3-5 conductors per cell and can have 300-1500 compute per cell in large clouds | |
| 17:33:36 | gmaan | I am not saying 'service up' is reliable source but my question is do we want to automate it in conductor to check compute before call otherwise asl sch to select other dest? or just for clear troubleshooting and error message that what exactly is down? | |
| 17:33:49 | gmaan | s/asl/ask | |
| 17:34:55 | sean-k-mooney | gmaan: ya im not sayign we shoudl do this im just playing devils advocate | |
| 17:35:45 | gmaan | sean-k-mooney: or the healthcheck proposal is good fit here but is the latest proposal check RPC health also? | |
| 17:35:55 | phildree[m] | it like a tcp Syn (compute send a request to any conductor), Syn-Ack (conductor replies to that tiny request), Ack (compute initiates the regular heartbeat as is) | |
| 17:35:55 | phildree[m] | This way when any part of the communction is broken the node does not appear as up | |
| 17:35:55 | phildree[m] | But in order to reduce RPCs i think also just a periodic ping without compute base initiation might be feasible but will have the issue of coordination etc.. | |
| 17:36:21 | sean-k-mooney | gmaan: it was goign to catch connection closed events and similar on the comptue and report if the rpc bus was working | |
| 17:36:26 | sean-k-mooney | but not by active probing | |
| 17:36:34 | sean-k-mooney | only by recoreding if we got an excption | |
| 17:36:34 | gmaan | k | |
| 17:36:41 | sean-k-mooney | so it woudl not really help in this case | |
| 17:36:41 | gmaan | ++ | |
| 17:36:50 | gmaan | yeah | |
| 17:38:53 | gmaan | do not have any perfect solution as of now but let's discuss it in PTG | |
| 17:39:47 | phildree[m] | Ok, thanks anyway for the engagement, I really appreciate you guys taking your time :) | |
| 19:20:17 | opendevreview | Merged openstack/nova stable/2026.1: fix: device_by_alias should respect config type https://review.opendev.org/c/openstack/nova/+/983989 | |
| 19:58:52 | opendevreview | John Garbutt proposed openstack/nova master: Make PCPUs not land on VCPUs by default https://review.opendev.org/c/openstack/nova/+/975779 | |
| 19:59:20 | opendevreview | John Garbutt proposed openstack/nova master: Fix ironic instance rebuild with API >=2.93 https://review.opendev.org/c/openstack/nova/+/963298 | |
| 20:24:21 | melwitt | dansmith: there is one more small patch that was split out from the 'user' live migration patch that is ready for review at your convenience https://review.opendev.org/c/openstack/nova/+/983505 | |
| 21:00:29 | dansmith | ah yep, the other one, sorry I guess I missed that when hitting the first one | |
| 21:00:30 | dansmith | done now | |
| 21:16:32 | melwitt | dansmith: thanks!! | |
| 22:46:32 | opendevreview | melanie witt proposed openstack/nova master: [WIP] how borken is ipv6 only ceph. https://review.opendev.org/c/openstack/nova/+/982302 | |
| 22:51:07 | opendevreview | melanie witt proposed openstack/nova master: [WIP] how borken is ipv6 only ceph. https://review.opendev.org/c/openstack/nova/+/982302 | |
| #openstack-nova - 2026-04-14 | |||
| 00:30:42 | opendevreview | melanie witt proposed openstack/nova master: CellDatabases: replace shared lock with thread-local cell routing https://review.opendev.org/c/openstack/nova/+/984480 | |
| 00:30:43 | opendevreview | melanie witt proposed openstack/nova master: Database: use file-backed SQLite for native threading https://review.opendev.org/c/openstack/nova/+/984481 | |
| 00:38:55 | melwitt | sean-k-mooney, gibi: I had read through the review comments on some database fixture patches yall have been working on and tried a couple of changes to try and help out ^ they may not be useful given my limited knowledge on the eventlet removal work | |
| 00:40:38 | melwitt | I have them rebased on Ashish's change https://review.opendev.org/c/openstack/nova/+/979850. the second patch complains of merge conflict for some reason though | |
| 01:02:48 | opendevreview | melanie witt proposed openstack/nova master: Database: use file-backed SQLite for native threading https://review.opendev.org/c/openstack/nova/+/984481 | |
| 01:15:45 | opendevreview | melanie witt proposed openstack/nova master: Database: use file-backed SQLite for native threading https://review.opendev.org/c/openstack/nova/+/984481 | |
| 06:20:42 | opendevreview | Merged openstack/nova master: mem-enc: stop using _get_mem_encryption_config() for SEV checks https://review.opendev.org/c/openstack/nova/+/967970 | |
| 07:14:59 | gibi | melwitt: thanks for that. I wil check it out | |
| 07:38:07 | opendevreview | Kamil Sambor proposed openstack/nova master: Enable threading mode for proxy services https://review.opendev.org/c/openstack/nova/+/976089 | |
| 07:42:05 | opendevreview | Kamil Sambor proposed openstack/nova master: Enable threading mode for proxy services https://review.opendev.org/c/openstack/nova/+/976089 | |
| 08:21:43 | gibi | bauzas: stephenfin: can one of you push this CI timeout bump through https://review.opendev.org/c/openstack/nova/+/984417 | |
| 08:45:03 | opendevreview | Konrad Gube proposed openstack/nova-specs master: Propose assisted-volume-extend API https://review.opendev.org/c/openstack/nova-specs/+/949504 | |
| 09:25:37 | opendevreview | Kamil Sambor proposed openstack/nova master: Change nova-alt-configurations job https://review.opendev.org/c/openstack/nova/+/983179 | |
| 09:31:09 | opendevreview | ribaudr proposed openstack/nova master: Add reproducer for bug #2141722 https://review.opendev.org/c/openstack/nova/+/984521 | |
| 09:31:10 | opendevreview | ribaudr proposed openstack/nova master: Fix os-simple-tenant-usage pagination with duplicate instance_extra https://review.opendev.org/c/openstack/nova/+/984522 | |