| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-03-26 | |||
| 17:44:25 | sean-k-mooney | so maybe ill try with 10 and then add them once it up | |
| 17:44:42 | JayF | sean-k-mooney: Ironic has a long, long, long history of knowing where all devstack timeouts are so we can set them high enough to make an infra team member weep | |
| 17:44:52 | JayF | lol | |
| 17:45:15 | JayF | sean-k-mooney: maybe add 1000 fake nodes, only put 3 of them in a shard managed by nova-compute by default or something :D either way I'll figure it out | |
| 17:47:05 | JayF | sean-k-mooney: clif: I put some notes in here https://etherpad.opendev.org/p/ironic-nova-compute-startup-2026.2 | |
| 17:47:47 | sean-k-mooney | clif: if you want to rebase of rearrage those wip patches feel free | |
| 17:48:29 | sean-k-mooney | i wont get back to them for a while but if they are helpful to you go for it | |
| 17:51:06 | clif | thank you, will do | |
| 18:51:17 | melwitt | gmaan: hey, I have a hopefully easy patch to fix openstacksdk usage for using endpoint discovery for unified limits if you might be able to look at your convenience https://review.opendev.org/c/openstack/nova/+/975106 | |
| 19:04:00 | erlon | sean-k-mooney++ melwitt++ thanks for the througly review on the RBD XML update functionality for migration process patch | |
| 19:05:27 | sean-k-mooney | erlon: im not sur eif keeping the sepreate small funciton for nova provisioned rbd storage is bettere or not for what its worth | |
| 19:05:34 | sean-k-mooney | i could be preswaded either way | |
| 19:05:56 | sean-k-mooney | but i woudl really prefer to see a functional repoducer for that | |
| 19:07:50 | erlon | melwitt: did you tried the XML "<ip family="ipv6" address="[2001:db8:ca2:2::1]" prefix="64"/>" with a ipv6 addr without the []? did it pass the checks? | |
| 19:10:44 | erlon | sean-k-mooney: you mean to have that code inside other function like _update_volume_xml? | |
| 19:11:34 | erlon | I believe that having the code in a separate function is better for readability and testing | |
| 19:14:19 | erlon | Another thing I was considering, is that when I brought it up in the meeting, we discussed wanting another managed command to perform the bulk update. I didn't start to think about how I'll do that implementation, but having things modularized usually help to reuse code. | |
| 19:15:55 | sean-k-mooney | erlon: my conder is driffet between the nova and cidner voulem management | |
| 19:16:27 | sean-k-mooney | erlon: like currently you are just updating the xml but not th bdms right | |
| 19:16:52 | melwitt | erlon: yes I did I pasted the output in the code block of my comment, I ran the virt-xml-validate CLI without brackets (passed) and with brackets (failed) | |
| 19:16:54 | sean-k-mooney | i dont recall if we store rbd info in teh bdms for nova provisioned stroage | |
| 19:17:22 | sean-k-mooney | melwitt: its possible that libvirt version depenent | |
| 19:17:23 | melwitt | no we don't, BDMs with rbd info is for cinder volume only | |
| 19:17:40 | sean-k-mooney | ok then that not an issue with the current patch so | |
| 19:18:05 | melwitt | sean-k-mooney: yes true. it would be ideal if we could test it for real somehow but regardless I think it's safest to do it without brackets bc that's how XML is currently generated for rbd | |
| 19:18:26 | sean-k-mooney | we have a tempest-ipv6 job in our gate | |
| 19:18:32 | sean-k-mooney | i just doen not ahve ceph in it | |
| 19:18:45 | sean-k-mooney | but i think we can look at that in a followup | |
| 19:18:54 | melwitt | I posted a comment on the review but I am wondering if the patch is actually needed right? bc AFAICT all cases should be covered already for nova local/ephemeral rbd | |
| 19:19:33 | melwitt | I wonder if people just want the patch to force an update to XML right away instead of waiting for a lifecycle event. just not sure I get the use case | |
| 19:20:10 | sean-k-mooney | so its not if you reboot the vm | |
| 19:20:35 | sean-k-mooney | but if we try to live migrate with the old mons i think the dest vm will fail to connect to the storage | |
| 19:20:45 | melwitt | ceph handles all of mon IP update transparently while the guest is running and it's not clear from the bug report if the lack of XML update there results in a failure or error of some kind | |
| 19:20:49 | melwitt | right and if you reboot the VM the existing code already grabs fresh mon IPs from ceph | |
| 19:20:49 | sean-k-mooney | and the migration will fail as a result if those mons are note aviabel | |
| 19:21:07 | melwitt | ok, that is what I wanted to know. I wish they had pasted the error in the bug | |
| 19:21:44 | sean-k-mooney | ya so unless libvirt is proxiyng that info without telling us i suspect that is the failure mode | |
| 19:21:48 | melwitt | ok, if it's actually a failure and not just a "I will feel better if the XML is updated now vs later" then I think the patch is useful | |
| 19:21:59 | erlon | melwitt, what would be the lifecicle events that would update that? | |
| 19:23:15 | melwitt | erlon: basically anything that calls _hard_reboot() internally so a stop/start, a hard reboot, an unshelve, I'm not sure what else off the top of my head. the main point is as long as the guest is running, the xml does not need an update, that's for later when it's starting up from being shutdown | |
| 19:23:23 | erlon | afic, ephemeral rbds are not updated unless the instance is sheved/unshelved | |
| 19:23:51 | sean-k-mooney | it should be updated any tiem teh domain is redefiend | |
| 19:24:16 | melwitt | erlon: however, if live migration would in fact _fail_ without the xml updated, then that is the problem that makes the patch needed. it was just not clear to me from the bug report whether this is a real failure or if someone did dumpxml and saw old mon IPs and that made them believe there is a problem | |
| 19:24:21 | sean-k-mooney | so cold migrate, shleve hard,reboot,evacuate | |
| 19:24:21 | erlon | yes, but the use case is to keep the instance running. | |
| 19:24:41 | melwitt | yup every time it's redefined. for local/ephemeral RBD to be clear. Cinder volume is NOT that way | |
| 19:25:22 | melwitt | for Cinder volume, currently, you have to live migrate to make it consume new mon IPs for its next xml redefinition. and I wonder if that might be the confusion | |
| 19:25:43 | erlon | melwitt: it does fail since, when RBD is trying to connect to one of the monitors, if the monitor times out, rbd will not retry | |
| 19:26:27 | melwitt | ack ok | |
| 19:27:19 | erlon | melwitt: no, the patch clearly handles that difference between Cinder back the volumes and RBD ephemerals and config drivers | |
| 19:27:56 | erlon | *cinder backed | |
| 19:28:16 | sean-k-mooney | erlon: sure but its addign supprot for somethign we do not supprot | |
| 19:28:28 | sean-k-mooney | changign the mon ips for ceph cluster attached to a nova instnace | |
| 19:28:43 | sean-k-mooney | we do not actully supprot that even if it works | |
| 19:28:53 | sean-k-mooney | we can make it better but it still operaotr error to ever do that | |
| 19:29:01 | melwitt | to be clear, if live migration is failing with local RBD disk when new mon IP happens, then I am personally ok with the patch. and I have a draft comment written up about that if that is the case | |
| 19:29:45 | sean-k-mooney | im ok with the patch to make thing bettter as long as we are clear this is not a supproted operation that operaors shoudl do freely | |
| 19:29:50 | melwitt | I had thought that as long as the guest remains running, that ceph handles the change transparently and we should not need to update any XML but I was not 100% and realized I could be wrong | |
| 19:30:07 | melwitt | *not 100% sure | |
| 19:30:09 | sean-k-mooney | in the cidner case it causes db cupprtion on our side at least as the connection info will be out of sync | |
| 19:30:51 | melwitt | well it's weird actually bc for live migration specifically, we _are_ refreshing connection_info. so ironically cinder volume works fine in this case already | |
| 19:30:52 | sean-k-mooney | melwitt: i bielive that is only true if 1 of the osd remains active at least that was a limitat of krbd in the past | |
| 19:30:55 | erlon | melwitt, yes, it does fail in that case | |
| 19:31:08 | melwitt | sean-k-mooney: I see | |
| 19:31:28 | sean-k-mooney | melwitt: actully ya your right we get an updated atachment | |
| 19:31:35 | sean-k-mooney | but only if we live migrate | |
| 19:31:37 | melwitt | ok, since failure is confirmed, I will post my draft comments that explain why I think the patch is ok | |
| 19:32:01 | sean-k-mooney | if we jsut hard reboto the vm wont come back up if none of the old mon ips are reachable | |
| 19:33:07 | erlon | If you hard reboot, it will get the correct addresses from the monitor | |
| 19:33:13 | melwitt | sean-k-mooney: right. what we are missing for cinder volume is the domain redefine case. bc we didn't want to be pulling connection_info needlessly for 99.999% cases. but I actually had an idea about that, how we could do a cheaper check by calling ceph and comparing to xml, and only if it's different call cinder for fresh connection_info | |
| 19:33:50 | sean-k-mooney | wwe coudl i jsut dont know if we ever want to really suprot this | |
| 19:34:01 | melwitt | sorry not compare to xml but compare to bdm | |
| 19:34:18 | sean-k-mooney | well either woruld work i guess btu i dont like dependignon the xml | |
| 19:34:22 | sean-k-mooney | if we can avoid it | |
| 19:34:33 | melwitt | yeah, I am not looking at it as much as supporting a thing we said it unsupported but rather making everything consistent | |
| 19:34:56 | sean-k-mooney | my conder with this in genrel is it opens us up to a buch of bugs relate dto snapshot ro other operatoin with stale info | |
| 19:35:18 | erlon | sean-k-mooney: this is something that is floating around for a long time, and many people has showed interest in having this fixed. The bug is the 2nd in heat/affteced: https://bugs.launchpad.net/nova/+bug/1452641 | |
| 19:35:30 | sean-k-mooney | we coudl totally burn all of those down but im not sure what we are opening our selve up to if we try | |
| 19:35:35 | melwitt | because today, we have a mon IP update for ephemeral RBD hard reboot, we have a mon update for cinder volume RBD live migration, but the vice versa for each are missing. and that to me seems ... not great | |
| 19:35:44 | gmaan | melwitt: checking | |
| 19:36:13 | sean-k-mooney | erlon: yep its not a new compleaint | |
| 19:36:30 | sean-k-mooney | i would question the validiy of the bug since we intentally iddint suprpot this in the past | |
| 19:37:11 | sean-k-mooney | erlon: what i really wish we coudl do is jsut use fqdns for the mons instead of ips | |
| 19:37:34 | sean-k-mooney | or multicast/vip ips or simialr | |
| 19:38:00 | erlon | That would not come without problems as well, since you would have to rely on DNS | |
| 19:38:14 | erlon | We have discussed in this possibility | |
| 19:38:15 | sean-k-mooney | anyways erlon if we build out the funcitonal tests to demostrate and validate the fix im ok with improving it as long as we dont docuemnt that htis is somethign operators shoudl expect to work in general | |
| 19:38:22 | melwitt | yeah to be clear, for me it is the inconsistent behavior. but, I do also wonder if it could open up any problems we don't expect. I can't think of anything but of course I guess anything is possible. I am not that much of a RBD expert | |
| 19:39:00 | erlon | sean-k-mooney, so with that I would assume that a nova managed command would be out of the table | |
| 19:39:03 | sean-k-mooney | we do not suprpot krbd but when we had it at a workaroudh chanig the mon ip coudl cuase the host kernel to soft lock | |
| 19:39:38 | sean-k-mooney | similar to how iowrites to iscsi block device that are disconnected would hang for ever | |
| 19:39:55 | sean-k-mooney | althoguht that was a long time ago so may that less terribel now | |
| 19:40:08 | sean-k-mooney | erlon: nova mange wont help here | |
| 19:40:14 | melwitt | gmaan: thanks :) also your ack would be appreciated on this backport proposed that fixes a http 500 => http 400 and if you think that is ok for stable https://review.opendev.org/c/openstack/nova/+/980540 | |
| 19:40:17 | sean-k-mooney | erlon: it woudl for cinder | |
| 19:40:23 | sean-k-mooney | but not for nova provisioned volumes | |
| 19:40:32 | sean-k-mooney | we dont use nova-mange to update xmls | |
| 19:41:05 | erlon | Did you would be that once a ceph mon list had changed, it will be possible to fix all vms without having to migrate or shelve them | |
| 19:41:09 | melwitt | we don't but we have for cinder volume nova-manage for refreshing bdm connection_info which would in turn update XML | |
| 19:41:34 | sean-k-mooney | right but we do it via the contoelr nodes by updating the db | |