Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2024-02-14
13:42:10 sean-k-mooney we do sometime back port new config options the impoarnt thing with a backport of this kind is it must not change behaivor by default
13:42:57 sean-k-mooney which is true in this case if you want to move all traffic to the migratoin network with that feature you have to set the config option to use the dedicated networks ip/fqdn/hostname
13:43:10 ganso sean-k-mooney: if the default value of the new config options retains the existing behavior, that is less concerning impact for existing deployments while backporting
13:43:36 sean-k-mooney yep it does which is and we also did that for upgrades
13:43:50 sean-k-mooney we did not want to change the behavior on upgade for the same config file
13:44:33 sean-k-mooney elodilles: ^ i know this is borderline but i think gibis feature is a more reasoable approch then https://review.opendev.org/c/openstack/nova/+/906053
13:45:15 ganso sean-k-mooney: yea if it solves the issue I am happy with backporting that instead. I need to test to be 100% sure, but it probably does as you say
13:45:52 ganso that other patch may even not be that much necessary anymore
13:46:09 sean-k-mooney ganso: i bleivle https://github.com/openstack/nova/commit/6bca37e904e9e56b250b123abde1901e951c8c9a#diff-486ddc37ce8e7c105a51566dd2cffbda034f28fae7eaad2e74abf470ab296352L4514 is the important part
13:47:27 sean-k-mooney i belive get_host_ip_addr is the common fucntion that is causing hte current behavior so yes https://review.opendev.org/c/openstack/nova/+/906053 shoudl not be requried anymore
13:49:04 ganso yep
14:55:38 opendevreview Rodrigo Barbieri proposed openstack/nova stable/2023.2: [libvirt]Add migration_inbound_addr https://review.opendev.org/c/openstack/nova/+/908995
14:56:27 ganso sean-k-mooney: I proposed the backport just to get the discussion started ^. Waiting to hear gibi's thoughts on this
14:56:42 sean-k-mooney ack
15:02:19 dougszu Thanks for looking at https://review.opendev.org/c/openstack/nova/+/906053. It comes from a time before the other patch. I will take a look at it shortly.
15:02:50 elodilles sean-k-mooney: well, if it's more like a feature than a bugfix, then best to avoid the backport, though, there can be exceptions, i mean, if the stable cores agrees that it worth the risk and most probably won't cause any regression, then it can be backported. (the only thing is NOT to change the current behaviour for existing deployments, i'd say)
15:03:41 sean-k-mooney elodilles: ya for upgrade reasons when we added this it preserved the existing behavior
15:04:07 sean-k-mooney elodilles: so if we were to backport it as with master you would have to modify the config option to use it
15:04:14 elodilles +1
15:04:35 sean-k-mooney elodilles: tl;dr we supprot movign the libvirt traffic to a dedicate network but only for live migraton
15:04:43 sean-k-mooney the feature enables that for cold migration
15:05:15 sean-k-mooney and as a sideffect also move the block copy done by nova wich is common for live migration to the new network if you opt in with the config option
15:05:45 sean-k-mooney not all users were aware that the live_migration_inbound_adr is just for the libvirt traffic
15:05:54 sean-k-mooney not also for novas data transfers
15:06:52 sean-k-mooney https://bugs.launchpad.net/nova/+bug/1939869 is partly a misundertaing of how thing work but i dont think its an unresobale expectation and other operator may have fallen faul of that too
15:08:34 elodilles i see, thanks for the details
15:19:51 elodilles sean-k-mooney: added my (long) comment to the backport o:)
15:21:52 sean-k-mooney ack we proably should chat about this either in the channel or in the team meetign to get import form stable folks in general
15:22:02 elodilles +1
15:30:07 opendevreview Merged openstack/nova master: HyperV: Add todo to remove HyperVLiveMigrateData object https://review.opendev.org/c/openstack/nova/+/906636
16:01:57 dougszu sean-k-mooney: Perhaps I missed something, but for https://bugs.launchpad.net/nova/+bug/1939869, i'm not sure how https://review.opendev.org/c/openstack/nova/+/908995 is going to 'fix' it. Since the source HV is calling the target HV by RPC, which then needs to get the IP of the source HV (not the hostname)
16:04:58 sean-k-mooney dougszu: it wont because it should not
16:05:20 sean-k-mooney howeve if you set the other config option then those calls shoudl use that for the data transfer
16:06:28 sean-k-mooney dougszu: defiening live_migration_inbound_addr should not cause the config driver or other data to be copied via that api
16:06:43 sean-k-mooney its a miss understanding fo what the config option currently is for
16:07:35 sean-k-mooney the new config option https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.migration_inbound_addr
16:07:51 sean-k-mooney states """ metadata is then used by the source of the migration traffic to construct the commands used to copy data (e.g. disk image) to the destination."""
16:08:28 sean-k-mooney so migration_inbound_addr should modify the adress used for the ssh/rsync
16:08:39 dougszu Yeah, no problem with rejecting the bug report.
16:08:50 sean-k-mooney but live_migration_inbound_addr should only modify the libvirt traffic
16:09:06 dougszu This bit, from i've seen happens the other way though: "metadata is then used by the source of the migration traffic to construct the commands used to copy data (e.g. disk image) to the destination"
16:09:17 dougszu I.e. the copy is initiated from the destination to the source
16:12:18 dougszu Eg. this runs on the dest: https://review.opendev.org/c/openstack/nova/+/906053/4/nova/virt/libvirt/driver.py#10917
16:12:49 dougszu So there has to be some way of transferring the source IP/FQDN etc to the dest
16:13:07 dougszu Apologies if I've missed something :D
16:19:11 ganso dougszu: I see your point, the custom IP config set in the source would need to be propagated to the destination somehow, as it wouldn't matter that the destination has a custom migration IP too because that's not the relevant one, it is the source one
16:23:53 dougszu ganso: yeah, that's it. Wondering if it's worth converting https://review.opendev.org/c/openstack/nova/+/906053 to a feature request
16:25:57 sean-k-mooney thats not how this work
16:26:19 sean-k-mooney the info is passed form the dest back to the souce
16:26:35 sean-k-mooney as the migration is started form the soucrce
16:26:43 sean-k-mooney and we copy the data form the source to test
16:26:45 sean-k-mooney *dest
16:28:13 ganso sean-k-mooney: the migration of VM data is, the disk copy no. In the logs that I have I see the dest ssh'ing to the src to scp the disk
16:29:43 dougszu +1, I definitely see the scp initiated from the dest. Eg. for fetching config drive during live migrate.
16:30:05 ganso dougszu: to be honest, if the patch already merged doesn't address the issue, then it becomes even more important that the new one does, and being a non-backportable feature prevents customers from accessing the fix unless they upgrade to caracal, which for some is a long time in the future
16:30:07 sean-k-mooney hum really
16:30:16 dougszu As in the bug report at the top: https://bugs.launchpad.net/nova/+bug/1939869
16:30:23 dougszu Happy to share more logs
16:30:23 sean-k-mooney i would have to look at that again but for config dirve specificaly
16:30:31 sean-k-mooney we dont actully need to copy it manually
16:30:36 sean-k-mooney libvirt can and will do that now
16:30:45 sean-k-mooney there was a bug in libvirt several year ago
16:30:54 sean-k-mooney that required us to do that but i tought we removed that code
16:30:57 dougszu It's also the glance image (if it's no longer available in glance)
16:31:32 sean-k-mooney so that should be done form the souce to dest i tought but perhaps its on the dest
16:31:53 sean-k-mooney ok lets do this
16:32:02 sean-k-mooney our normall request for any complex bug
16:32:21 sean-k-mooney is to first create a reproducer fucntional test showing the buggy behavior
16:32:54 sean-k-mooney so it woudl be good to try and simulate this with a fuctional test to show that
16:38:00 ganso live_migration_inbound_addr instead
16:38:00 ganso sean-k-mooney: hmmmmm if it was an older libvirt bug that may not longer be valid or that version no longer used since a given version that matches recent nova version we would backport to (for example focal-zed), so maybe instead of backporting the new config option, or even going with the new patch that bumps the object version, we could perhaps shift things around and move that from the source as you say and use the
16:38:57 ganso actually I don't think zed is used on focal, it is from jammy onwards, and in jammy it may be that every libvirt version available does not have that bug
16:39:12 sean-k-mooney so there are a few issue at play here.
16:39:22 sean-k-mooney first you should not be doing live migation by doing "openstack server migrate --live-migration --block-migration`"
16:39:38 sean-k-mooney --block-migation was depercated about 6+ years ago
16:40:28 ganso sean-k-mooney: oh I didn't know that, the customer is using --block-migration because that's the only way they can get the config drive moved. Is there a better way to do this?
16:42:04 sean-k-mooney in mitaka we added auto to do the right thing
16:42:08 sean-k-mooney https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#maximum-in-mitaka
16:43:26 ganso sean-k-mooney: by "auto" you mean, not specifying anything? in that case the code complains that it is not in shared storage
16:44:00 sean-k-mooney so if you do openstack --os-compute-api=2.25 server migrate --live <server>
16:44:09 sean-k-mooney then it will just do the right thing
16:44:22 sean-k-mooney and we will leave libvirt copy the config drive i belive
16:45:04 sean-k-mooney ganso: there was a specific bug with libvirt and live migrating with iso files attached as a cdrom
16:45:11 sean-k-mooney that requried us to copy it first
16:45:21 sean-k-mooney but that was in like ubuntu 16.04
16:45:54 sean-k-mooney resolved in libvirt v1.2.17.
16:46:30 sean-k-mooney https://bugs.launchpad.net/nova/+bug/1246201
16:46:53 dougszu It's this code block running on dest (contains both config drive and glance image copy): https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L10892
16:47:21 dougszu I'm not seeing where it's gated on --block-migration, but could have missed something
16:47:29 sean-k-mooney right that was added by https://review.opendev.org/c/openstack/nova/+/365420/1/nova/virt/libvirt/driver.py
16:47:51 sean-k-mooney that code is not needed anymore and has not been for litrally years
16:48:27 dougszu For config drive yeah, but not the Glance image?
16:48:31 sean-k-mooney ya so that code https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L10899-L10911
16:48:34 sean-k-mooney can be removed
16:48:53 sean-k-mooney for the backing file
16:49:10 sean-k-mooney we cant remvoe it but its not a bug that that does nto use the live
16:49:24 sean-k-mooney *live_migrateion_inboud_adr
16:50:16 sean-k-mooney im not sure if that will now work with the new migration migration_inbound_addr
16:50:38 sean-k-mooney i belive it should but it it does not it would be a bug in that new feature
16:51:42 ganso sean-k-mooney: ok I will ask my customer to re-test that with the --os-compute-api=2.25 --live and without --block-migration and see if that solves the issue

Earlier   Later