Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2024-02-14
15:21:52 sean-k-mooney ack we proably should chat about this either in the channel or in the team meetign to get import form stable folks in general
15:22:02 elodilles +1
15:30:07 opendevreview Merged openstack/nova master: HyperV: Add todo to remove HyperVLiveMigrateData object https://review.opendev.org/c/openstack/nova/+/906636
16:01:57 dougszu sean-k-mooney: Perhaps I missed something, but for https://bugs.launchpad.net/nova/+bug/1939869, i'm not sure how https://review.opendev.org/c/openstack/nova/+/908995 is going to 'fix' it. Since the source HV is calling the target HV by RPC, which then needs to get the IP of the source HV (not the hostname)
16:04:58 sean-k-mooney dougszu: it wont because it should not
16:05:20 sean-k-mooney howeve if you set the other config option then those calls shoudl use that for the data transfer
16:06:28 sean-k-mooney dougszu: defiening live_migration_inbound_addr should not cause the config driver or other data to be copied via that api
16:06:43 sean-k-mooney its a miss understanding fo what the config option currently is for
16:07:35 sean-k-mooney the new config option https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.migration_inbound_addr
16:07:51 sean-k-mooney states """ metadata is then used by the source of the migration traffic to construct the commands used to copy data (e.g. disk image) to the destination."""
16:08:28 sean-k-mooney so migration_inbound_addr should modify the adress used for the ssh/rsync
16:08:39 dougszu Yeah, no problem with rejecting the bug report.
16:08:50 sean-k-mooney but live_migration_inbound_addr should only modify the libvirt traffic
16:09:06 dougszu This bit, from i've seen happens the other way though: "metadata is then used by the source of the migration traffic to construct the commands used to copy data (e.g. disk image) to the destination"
16:09:17 dougszu I.e. the copy is initiated from the destination to the source
16:12:18 dougszu Eg. this runs on the dest: https://review.opendev.org/c/openstack/nova/+/906053/4/nova/virt/libvirt/driver.py#10917
16:12:49 dougszu So there has to be some way of transferring the source IP/FQDN etc to the dest
16:13:07 dougszu Apologies if I've missed something :D
16:19:11 ganso dougszu: I see your point, the custom IP config set in the source would need to be propagated to the destination somehow, as it wouldn't matter that the destination has a custom migration IP too because that's not the relevant one, it is the source one
16:23:53 dougszu ganso: yeah, that's it. Wondering if it's worth converting https://review.opendev.org/c/openstack/nova/+/906053 to a feature request
16:25:57 sean-k-mooney thats not how this work
16:26:19 sean-k-mooney the info is passed form the dest back to the souce
16:26:35 sean-k-mooney as the migration is started form the soucrce
16:26:43 sean-k-mooney and we copy the data form the source to test
16:26:45 sean-k-mooney *dest
16:28:13 ganso sean-k-mooney: the migration of VM data is, the disk copy no. In the logs that I have I see the dest ssh'ing to the src to scp the disk
16:29:43 dougszu +1, I definitely see the scp initiated from the dest. Eg. for fetching config drive during live migrate.
16:30:05 ganso dougszu: to be honest, if the patch already merged doesn't address the issue, then it becomes even more important that the new one does, and being a non-backportable feature prevents customers from accessing the fix unless they upgrade to caracal, which for some is a long time in the future
16:30:07 sean-k-mooney hum really
16:30:16 dougszu As in the bug report at the top: https://bugs.launchpad.net/nova/+bug/1939869
16:30:23 sean-k-mooney i would have to look at that again but for config dirve specificaly
16:30:23 dougszu Happy to share more logs
16:30:31 sean-k-mooney we dont actully need to copy it manually
16:30:36 sean-k-mooney libvirt can and will do that now
16:30:45 sean-k-mooney there was a bug in libvirt several year ago
16:30:54 sean-k-mooney that required us to do that but i tought we removed that code
16:30:57 dougszu It's also the glance image (if it's no longer available in glance)
16:31:32 sean-k-mooney so that should be done form the souce to dest i tought but perhaps its on the dest
16:31:53 sean-k-mooney ok lets do this
16:32:02 sean-k-mooney our normall request for any complex bug
16:32:21 sean-k-mooney is to first create a reproducer fucntional test showing the buggy behavior
16:32:54 sean-k-mooney so it woudl be good to try and simulate this with a fuctional test to show that
16:38:00 ganso sean-k-mooney: hmmmmm if it was an older libvirt bug that may not longer be valid or that version no longer used since a given version that matches recent nova version we would backport to (for example focal-zed), so maybe instead of backporting the new config option, or even going with the new patch that bumps the object version, we could perhaps shift things around and move that from the source as you say and use the
16:38:00 ganso live_migration_inbound_addr instead
16:38:57 ganso actually I don't think zed is used on focal, it is from jammy onwards, and in jammy it may be that every libvirt version available does not have that bug
16:39:12 sean-k-mooney so there are a few issue at play here.
16:39:22 sean-k-mooney first you should not be doing live migation by doing "openstack server migrate --live-migration --block-migration`"
16:39:38 sean-k-mooney --block-migation was depercated about 6+ years ago
16:40:28 ganso sean-k-mooney: oh I didn't know that, the customer is using --block-migration because that's the only way they can get the config drive moved. Is there a better way to do this?
16:42:04 sean-k-mooney in mitaka we added auto to do the right thing
16:42:08 sean-k-mooney https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#maximum-in-mitaka
16:43:26 ganso sean-k-mooney: by "auto" you mean, not specifying anything? in that case the code complains that it is not in shared storage
16:44:00 sean-k-mooney so if you do openstack --os-compute-api=2.25 server migrate --live <server>
16:44:09 sean-k-mooney then it will just do the right thing
16:44:22 sean-k-mooney and we will leave libvirt copy the config drive i belive
16:45:04 sean-k-mooney ganso: there was a specific bug with libvirt and live migrating with iso files attached as a cdrom
16:45:11 sean-k-mooney that requried us to copy it first
16:45:21 sean-k-mooney but that was in like ubuntu 16.04
16:45:54 sean-k-mooney resolved in libvirt v1.2.17.
16:46:30 sean-k-mooney https://bugs.launchpad.net/nova/+bug/1246201
16:46:53 dougszu It's this code block running on dest (contains both config drive and glance image copy): https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L10892
16:47:21 dougszu I'm not seeing where it's gated on --block-migration, but could have missed something
16:47:29 sean-k-mooney right that was added by https://review.opendev.org/c/openstack/nova/+/365420/1/nova/virt/libvirt/driver.py
16:47:51 sean-k-mooney that code is not needed anymore and has not been for litrally years
16:48:27 dougszu For config drive yeah, but not the Glance image?
16:48:31 sean-k-mooney ya so that code https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L10899-L10911
16:48:34 sean-k-mooney can be removed
16:48:53 sean-k-mooney for the backing file
16:49:10 sean-k-mooney we cant remvoe it but its not a bug that that does nto use the live
16:49:24 sean-k-mooney *live_migrateion_inboud_adr
16:50:16 sean-k-mooney im not sure if that will now work with the new migration migration_inbound_addr
16:50:38 sean-k-mooney i belive it should but it it does not it would be a bug in that new feature
16:51:42 ganso sean-k-mooney: ok I will ask my customer to re-test that with the --os-compute-api=2.25 --live and without --block-migration and see if that solves the issue
16:52:00 sean-k-mooney ack
16:52:02 dougszu Thanks, I can do some more testing. Good to know the config drive code can be removed.
16:52:03 ganso sean-k-mooney: btw I thought --live was deprecated in favor of --live-migration, --live was the one that specified the dest and --live-migration leaves it to the scheduler
16:52:22 sean-k-mooney oh right
16:52:42 sean-k-mooney ya in either case the impoatnt point it to use a microversion >= 2.25
16:52:48 sean-k-mooney to get the auto behvior
16:55:08 ganso sean-k-mooney: thanks for the explanation! :)
16:56:07 sean-k-mooney a straignt revert of https://review.opendev.org/c/openstack/nova/+/365420 wont work because of merge conflcits
16:56:32 opendevreview sean mooney proposed openstack/nova stable/mitaka: Revert "[libvirt] Live migration fails when config_drive_format=iso9660" https://review.opendev.org/c/openstack/nova/+/908937
16:56:46 sean-k-mooney actully never mind apprently it will work
16:57:09 opendevreview sean mooney proposed openstack/nova stable/mitaka: Revert "[libvirt] Live migration fails when config_drive_format=iso9660" https://review.opendev.org/c/openstack/nova/+/908937
16:58:51 sean-k-mooney actully the striagt revert is wrong for other reason but ill see if i can remove that code properly oh also its agaisnt the wrong branch
16:58:59 sean-k-mooney so ignore ^
16:59:52 sean-k-mooney dougszu: ganso what i was going to say is if either of ye feel like reverting the config drive bit on master i can review
17:00:08 sean-k-mooney if not ill add this to my todo list and se if we can remove that
17:00:34 dougszu I'll try and have a go tomorrow
18:13:40 ganso sean-k-mooney: I rather have my customer test first to be sure there will be a code path that addresses their config drive migration. At the moment I am not sure as I am not familiar with that part of the code. It should go "auto" as you say but each case could be different
#openstack-nova - 2024-02-15
05:43:19 opendevreview Amit Uniyal proposed openstack/nova master: Added context manager for instance lock https://review.opendev.org/c/openstack/nova/+/873648
05:43:20 opendevreview Amit Uniyal proposed openstack/nova master: Separate OSError with ValueError https://review.opendev.org/c/openstack/nova/+/908825
05:43:20 opendevreview Amit Uniyal proposed openstack/nova master: Disconnecting volume from the compute host https://review.opendev.org/c/openstack/nova/+/877446
05:43:21 opendevreview Amit Uniyal proposed openstack/nova master: Removed explicit call to delete attachment https://review.opendev.org/c/openstack/nova/+/891289
08:47:59 opendevreview Amit Uniyal proposed openstack/nova master: Separate OSError with ValueError https://review.opendev.org/c/openstack/nova/+/908825
08:47:59 opendevreview Amit Uniyal proposed openstack/nova master: Disconnecting volume from the compute host https://review.opendev.org/c/openstack/nova/+/877446
08:48:00 opendevreview Amit Uniyal proposed openstack/nova master: Removed explicit call to delete attachment https://review.opendev.org/c/openstack/nova/+/891289
08:49:49 tkajinam sean-k-mooney, hi. thanks for your feedback in libvirt related patches ! I've replied to your comment in https://review.opendev.org/c/openstack/nova/+/908546/ so please check it when you have time
08:50:21 tkajinam in short, swtpm_ioctl has been required since swtpm support was initially added to libvirt.
10:22:43 opendevreview Merged openstack/nova master: libvirt: Stop unconditionally enabling evmcs https://review.opendev.org/c/openstack/nova/+/899776

Earlier   Later