Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2026-03-26
19:25:22 melwitt for Cinder volume, currently, you have to live migrate to make it consume new mon IPs for its next xml redefinition. and I wonder if that might be the confusion
19:25:43 erlon melwitt: it does fail since, when RBD is trying to connect to one of the monitors, if the monitor times out, rbd will not retry
19:26:27 melwitt ack ok
19:27:19 erlon melwitt: no, the patch clearly handles that difference between Cinder back the volumes and RBD ephemerals and config drivers
19:27:56 erlon *cinder backed
19:28:16 sean-k-mooney erlon: sure but its addign supprot for somethign we do not supprot
19:28:28 sean-k-mooney changign the mon ips for ceph cluster attached to a nova instnace
19:28:43 sean-k-mooney we do not actully supprot that even if it works
19:28:53 sean-k-mooney we can make it better but it still operaotr error to ever do that
19:29:01 melwitt to be clear, if live migration is failing with local RBD disk when new mon IP happens, then I am personally ok with the patch. and I have a draft comment written up about that if that is the case
19:29:45 sean-k-mooney im ok with the patch to make thing bettter as long as we are clear this is not a supproted operation that operaors shoudl do freely
19:29:50 melwitt I had thought that as long as the guest remains running, that ceph handles the change transparently and we should not need to update any XML but I was not 100% and realized I could be wrong
19:30:07 melwitt *not 100% sure
19:30:09 sean-k-mooney in the cidner case it causes db cupprtion on our side at least as the connection info will be out of sync
19:30:51 melwitt well it's weird actually bc for live migration specifically, we _are_ refreshing connection_info. so ironically cinder volume works fine in this case already
19:30:52 sean-k-mooney melwitt: i bielive that is only true if 1 of the osd remains active at least that was a limitat of krbd in the past
19:30:55 erlon melwitt, yes, it does fail in that case
19:31:08 melwitt sean-k-mooney: I see
19:31:28 sean-k-mooney melwitt: actully ya your right we get an updated atachment
19:31:35 sean-k-mooney but only if we live migrate
19:31:37 melwitt ok, since failure is confirmed, I will post my draft comments that explain why I think the patch is ok
19:32:01 sean-k-mooney if we jsut hard reboto the vm wont come back up if none of the old mon ips are reachable
19:33:07 erlon If you hard reboot, it will get the correct addresses from the monitor
19:33:13 melwitt sean-k-mooney: right. what we are missing for cinder volume is the domain redefine case. bc we didn't want to be pulling connection_info needlessly for 99.999% cases. but I actually had an idea about that, how we could do a cheaper check by calling ceph and comparing to xml, and only if it's different call cinder for fresh connection_info
19:33:50 sean-k-mooney wwe coudl i jsut dont know if we ever want to really suprot this
19:34:01 melwitt sorry not compare to xml but compare to bdm
19:34:18 sean-k-mooney well either woruld work i guess btu i dont like dependignon the xml
19:34:22 sean-k-mooney if we can avoid it
19:34:33 melwitt yeah, I am not looking at it as much as supporting a thing we said it unsupported but rather making everything consistent
19:34:56 sean-k-mooney my conder with this in genrel is it opens us up to a buch of bugs relate dto snapshot ro other operatoin with stale info
19:35:18 erlon sean-k-mooney: this is something that is floating around for a long time, and many people has showed interest in having this fixed. The bug is the 2nd in heat/affteced: https://bugs.launchpad.net/nova/+bug/1452641
19:35:30 sean-k-mooney we coudl totally burn all of those down but im not sure what we are opening our selve up to if we try
19:35:35 melwitt because today, we have a mon IP update for ephemeral RBD hard reboot, we have a mon update for cinder volume RBD live migration, but the vice versa for each are missing. and that to me seems ... not great
19:35:44 gmaan melwitt: checking
19:36:13 sean-k-mooney erlon: yep its not a new compleaint
19:36:30 sean-k-mooney i would question the validiy of the bug since we intentally iddint suprpot this in the past
19:37:11 sean-k-mooney erlon: what i really wish we coudl do is jsut use fqdns for the mons instead of ips
19:37:34 sean-k-mooney or multicast/vip ips or simialr
19:38:00 erlon That would not come without problems as well, since you would have to rely on DNS
19:38:14 erlon We have discussed in this possibility
19:38:15 sean-k-mooney anyways erlon if we build out the funcitonal tests to demostrate and validate the fix im ok with improving it as long as we dont docuemnt that htis is somethign operators shoudl expect to work in general
19:38:22 melwitt yeah to be clear, for me it is the inconsistent behavior. but, I do also wonder if it could open up any problems we don't expect. I can't think of anything but of course I guess anything is possible. I am not that much of a RBD expert
19:39:00 erlon sean-k-mooney, so with that I would assume that a nova managed command would be out of the table
19:39:03 sean-k-mooney we do not suprpot krbd but when we had it at a workaroudh chanig the mon ip coudl cuase the host kernel to soft lock
19:39:38 sean-k-mooney similar to how iowrites to iscsi block device that are disconnected would hang for ever
19:39:55 sean-k-mooney althoguht that was a long time ago so may that less terribel now
19:40:08 sean-k-mooney erlon: nova mange wont help here
19:40:14 melwitt gmaan: thanks :) also your ack would be appreciated on this backport proposed that fixes a http 500 => http 400 and if you think that is ok for stable https://review.opendev.org/c/openstack/nova/+/980540
19:40:17 sean-k-mooney erlon: it woudl for cinder
19:40:23 sean-k-mooney but not for nova provisioned volumes
19:40:32 sean-k-mooney we dont use nova-mange to update xmls
19:41:05 erlon Did you would be that once a ceph mon list had changed, it will be possible to fix all vms without having to migrate or shelve them
19:41:09 melwitt we don't but we have for cinder volume nova-manage for refreshing bdm connection_info which would in turn update XML
19:41:34 sean-k-mooney right but we do it via the contoelr nodes by updating the db
19:41:36 erlon but I believe that would be something we could live without
19:41:37 melwitt ok I think you just said that
19:41:41 melwitt yeah
19:42:02 sean-k-mooney i dont think we should ever add a nova mange command that talks to libvirt and modifes the runing domain
19:42:25 melwitt yeah so for rbd local/ephemeral if you hard rebooted all the VMs then you would get all the new mon IPs today. but I get that a lot of operators don't want to be rebooting all the guests
19:42:49 sean-k-mooney ya so "healing it" on live migratoin is ok
19:43:08 sean-k-mooney healign it in place to me voids yoru warrenty even with somthgn liek nova-mnage
19:43:56 sean-k-mooney we have alwasy said out of band updates to the domain (any write to libvirt not form nova-compute) voids the warrenty on the domain
19:44:32 sean-k-mooney it would be very easy to create security issues by fat fingering the cli
19:44:54 sean-k-mooney and it woudl be inharently racy with api actoins too
19:53:32 opendevreview sean mooney proposed openstack/nova master: [WIP] how borken is ipv6 only ceph. https://review.opendev.org/c/openstack/nova/+/982302
19:54:56 opendevreview sean mooney proposed openstack/nova master: [WIP] how borken is ipv6 only ceph. https://review.opendev.org/c/openstack/nova/+/982302
19:55:26 erlon sean-k-mooney: regarding the direct change/replace all monitors in the function, I understand that comparing that with the old values give us the opportunity to issue a warning, but it should not cause any impact on running instances, since we are just manipulating an in memory XML object, that will be sent to the destination host
19:56:14 sean-k-mooney right but if that warning ever fires it means that an operator did somethign wrong
19:56:44 sean-k-mooney so it warning them that thre may be problems that need to be fixed
20:04:01 gmaan melwitt: done, backport of 500->400 lgtm too
20:04:27 melwitt thank you gmaan
20:06:25 sean-k-mooney looking at zuul there is currently no job wiht ipv6 and (ceph or cinder) in the name so im not going to hold my breath on ^ working
20:07:28 sean-k-mooney it shoudl be doable we woudl jsut need to tweak the devstack plugin to resovle what ever it finds
21:32:35 melwitt dunno if this is a known issue but I noticed grenade-skip-level-always (non-voting) is failing on stable/2025.1 with "AttributeError: module 'setuptools.build_meta' has no attribute 'prepare_metadata_for_build_editable'. Did you mean: 'prepare_metadata_for_build_wheel'?"
21:32:46 melwitt https://zuul.opendev.org/t/openstack/builds?job_name=grenade-skip-level-always&project=openstack%2Fnova&branch=stable%2F2025.1&skip=0
21:45:08 sean-k-mooney melwitt: that might be related to jammy and setuptools/pbr versions when upgrading form 2024.1
21:45:21 sean-k-mooney 2024.2 has not got the pkg_resouces fixes
21:45:36 sean-k-mooney at least not in the requriemetn repo and has not got the newerever pbr release as a result
21:47:35 sean-k-mooney https://review.opendev.org/c/openstack/requirements/+/976602 needs https://review.opendev.org/c/openstack/requirements/+/976903 to be cherry picked first
21:49:36 sean-k-mooney well it might not be that expiclty but its related to setup_tools and pbr and i think those mibg help
21:51:06 sean-k-mooney melwitt https://github.com/openstack/nova/commit/42de440471abf5d1953eacd21f191c1f0ff28ef7
21:51:44 sean-k-mooney so ya its to do with the 2024.1 install
21:51:57 sean-k-mooney 2024.1 is not actully supproted for upgrades anymore
21:52:16 sean-k-mooney so we proably shoudl just turn that job off permently
22:15:49 opendevreview Merged openstack/nova master: unified limits: Fix openstacksdk usage for endpoint discovery https://review.opendev.org/c/openstack/nova/+/975106
22:27:41 opendevreview Merged openstack/nova stable/2025.2: api: Handle empty imageRef alongside null for local BDM check https://review.opendev.org/c/openstack/nova/+/980540
22:27:54 opendevreview Merged openstack/nova stable/2025.1: Add functional reproducer for bug 2125030 https://review.opendev.org/c/openstack/nova/+/979864
22:35:50 opendevreview Merged openstack/nova stable/2025.1: Move cleanup of vTPM secret from driver to compute https://review.opendev.org/c/openstack/nova/+/979865
22:36:03 opendevreview Merged openstack/nova stable/2025.1: Preserve vTPM state between power off and power on https://review.opendev.org/c/openstack/nova/+/979866
22:55:56 opendevreview Merged openstack/nova stable/2025.2: Preserve UEFI NVRAM variable store https://review.opendev.org/c/openstack/nova/+/978766
22:56:08 opendevreview Merged openstack/nova stable/2025.1: Fix string format specifier https://review.opendev.org/c/openstack/nova/+/961337
#openstack-nova - 2026-03-27
05:29:18 opendevreview Nicolai Ruckel proposed openstack/nova stable/2025.1: Preserve UEFI NVRAM variable store https://review.opendev.org/c/openstack/nova/+/982353
05:47:05 nicolairuckel elodilles, melwitt: I have one more backport. :)
10:00:15 zigo stephenfin: Hi there! Shouldn't we get a new release of cliff for Gazpacho, so it includes your patch at https://review.opendev.org/c/openstack/cliff/+/978994 ?
10:00:54 zigo I'm including it in the cliff Debian package, as I bumped cmd2 to 3.2.0, though other downstream may get into the same issues.
10:31:48 stephenfin zigo: Hmm, yes, most likely
11:25:26 opendevreview Balazs Gibizer proposed openstack/nova master: Default native threading for conductor and compute https://review.opendev.org/c/openstack/nova/+/982237
11:26:59 gibi bauzas: gmaan: ^^ fixed your comments
12:41:43 opendevreview Ashish Gupta proposed openstack/nova master: Add native threading support to functional test (DB, RPC, test case ID) https://review.opendev.org/c/openstack/nova/+/980179
13:39:54 opendevreview clemens proposed openstack/nova master: libvirt: added customizable libvirt disk aio https://review.opendev.org/c/openstack/nova/+/982419

Earlier   Later