Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2025-11-04
19:10:42 sean-k-mooney to allow pre-copy to make progress
19:10:44 Zhan[m] we have auto converge on and I do see VIR_DOMAIN_EVENT_SUSPENDED_MIGRATED gets generated
19:11:08 sean-k-mooney right but my point is its only safe to use VIR_DOMAIN_EVENT_SUSPENDED_MIGRATED
19:11:09 Zhan[m] the VM needs to paused before the swtich-over, regardless of converge or not, right?
19:11:14 sean-k-mooney if its only sent once
19:11:26 sean-k-mooney when we are never going to resume on this host again
19:11:47 Zhan[m] good point, I believe it is only sent once, I'll double check that
19:12:29 sean-k-mooney if VIR_DOMAIN_EVENT_SUSPENDED_MIGRATED is a point of no return (we will never resume on the source after that) then
19:12:38 sean-k-mooney i think we can move to useign that
19:13:12 sean-k-mooney but we will obviuly need to test hat
19:13:16 sean-k-mooney *that
19:13:46 Zhan[m] but even if it's resumed later (likely due to the migration fails), libvirt will report error and we can still roll back the port bindings to source?
19:14:05 sean-k-mooney Zhan[m]: since your using calico your proably also interest in https://bugs.launchpad.net/nova/+bug/2033681
19:14:26 sean-k-mooney Zhan[m]: not if we have invoked post-live-migration
19:14:40 sean-k-mooney which was my point before if we are not adding a speciall addtional handeler
19:14:53 sean-k-mooney and jsut calling post-live-migration like we do for post-copy
19:15:19 sean-k-mooney then we need to be 100% sure it cant role back and that its safe to start cleaning up the souce on the nova side
19:15:31 sean-k-mooney if we only do the port bindign activation it probaly ok
19:16:21 sean-k-mooney what we might do is move the port bidning activation logic to its own method and have that tirger on either VIR_DOMAIN_EVENT_SUSPENDED_POSTCOPY or VIR_DOMAIN_EVENT_SUSPENDED_POSTCOPY
19:16:31 sean-k-mooney *VIR_DOMAIN_EVENT_SUSPENDED_POSTCOPY or VIR_DOMAIN_EVENT_SUSPENDED_MIGRATED
19:16:31 Zhan[m] If we have invoked post live migration then it must mean the migration succeeded, so we are ok to move everything to dest
19:16:54 Zhan[m] right, that's what I'm thinking
19:17:31 Zhan[m] I do have some code in hand and will be starting testing while writing the spec, I can link it there
19:19:12 sean-k-mooney https://docs.openstack.org/nova/latest/reference/live-migration.html
19:19:12 Zhan[m] ok I think we are on the same page, lemme write that spec and put details there
19:19:23 sean-k-mooney the main concen i have is you are changing the current workflow
19:19:45 sean-k-mooney by adding another pahse for network activation
19:19:49 Zhan[m] yes, because before, for precopy, we only activate bindings when we see the migration is good
19:20:02 sean-k-mooney well when its complete
19:20:43 Zhan[m] complete and good (Ig :P)
19:20:48 sean-k-mooney the lbivrt part where we block should typeicly be very very short
19:21:03 sean-k-mooney unelss you are activly copying memry for the guest
19:21:24 Zhan[m] no... in our environment we saw some crz numbers like 20s+
19:21:27 Zhan[m] or even a minute
19:21:35 Zhan[m] between PAUSED to job completed event
19:21:56 Zhan[m] because libvirt waits for the source QEMU process to be cleaned up before saying job is completed
19:21:57 sean-k-mooney Zhan[m]: https://docs.openstack.org/nova/latest/configuration/config.html#workarounds.enable_qemu_monitor_announce_self
19:22:02 sean-k-mooney by the way do you enable ^
19:22:30 sean-k-mooney Zhan[m]: right but that shoudl only happen if qemu is copying memory
19:22:55 sean-k-mooney i guess the qemu instance clean up coudl take a long time if it waithign for io to complete
19:23:26 sean-k-mooney but libvirt shoudl not unpasue the dest instance until the source is stoped.
19:23:32 Zhan[m] what I found was that the kernel was spinning to clean up the memory resource
19:23:41 Zhan[m] which took quite a lot of time
19:24:09 Zhan[m] I dont think we enabled that, lemme take a look
19:24:51 Zhan[m] the memory resource -> the memory space that the VM uses, not by migration. if the VM is using a huge number of pages then the cleanup can take longer.
19:24:55 sean-k-mooney Zhan[m]: does the guest use hugepages or anonoums 4k pages out fo interest
19:25:22 sean-k-mooney im just wonderign why that takes so long in your case
19:25:55 Zhan[m] we use THP, but we figured that KSM and this bug(https://lore.kernel.org/all/20240626191830.3819324-1-yang@os.amperecomputing.com/) was causing it to not use hugepages as expected.
19:26:09 sean-k-mooney ah ok
19:26:29 sean-k-mooney THP is awsome if it works and terible otherwise
19:26:36 sean-k-mooney espically when combind iwht ksm
19:26:44 sean-k-mooney it causes a lot of non determinism
19:27:03 Zhan[m] exactly lol, but yeah, that's why we also wanna tweak nova to just skip the cleanup
19:28:55 sean-k-mooney hum i was not aware of that bug but reading it that definly going to slow things donw
19:29:12 sean-k-mooney is that fixed upstream?
19:29:19 Zhan[m] yes, what QEMU does in migration is that it will read the page first and then write
19:29:24 Zhan[m] it's fixed in 6.13 IIRC
19:29:46 Zhan[m] so all of the following writes done by the VM will be regular pages
19:29:57 Zhan[m] which is painful....
19:30:31 sean-k-mooney ah ok
19:30:34 Zhan[m] https://github.com/torvalds/linux/commit/5c00ff742bf5caf85f60e1c73999f99376fb865d yeah 6.13
19:30:57 sean-k-mooney 6.13 is resonably new so that proably not in a lot of disto kernels
19:31:39 sean-k-mooney ubutnu has 6.14 in the HWE kernsl but it default to 6.11 for 24.04
19:31:59 sean-k-mooney rhel 10 is 6.12 + backports
19:31:59 Zhan[m] yeah but anyways, nova doesn't really need to wait for the cleanup to be done.
19:32:09 Zhan[m] if it's doing cleanup, then it means the VM has been resumed on dest
19:32:19 sean-k-mooney ya
19:32:29 Zhan[m] which we can start at least setting up the network
19:32:37 sean-k-mooney if we can trigger form that point we likely cna just go direclty ot post-live-migrate
19:33:08 sean-k-mooney but we cant relibly tirgger on the cleanup starting then we can expore just doing the network activation
19:34:08 Zhan[m] yeah just the network I feel is fine, everything else can be pushed back
19:34:26 Zhan[m] what VM users ultimately care about is just network connectivity and if the VM is running or not
19:34:49 Zhan[m] which in that case the VM is running so we should setup the network
19:36:01 sean-k-mooney what i woudl suggest is push a patch to hack https://github.com/openstack/nova/blob/master/nova/virt/libvirt/host.py#L274-L321
19:36:21 sean-k-mooney oh
19:36:23 sean-k-mooney https://github.com/openstack/nova/blob/master/nova/virt/libvirt/host.py#L295-L298
19:36:56 Zhan[m] right, I'm thinking if the migration fails, we rollback the port bindings back to source
19:37:11 sean-k-mooney Zhan[m]: so based on that comment we cant realy on that only beign sent on sucssess
19:37:20 sean-k-mooney whgich mean we have to call guest.get_job_info()
19:37:42 Zhan[m] right but that get_job_info will hang until cleanup is done
19:37:48 sean-k-mooney right
19:38:11 sean-k-mooney but we cant just assume its ok to activate the netork on the dest based on VIR_DOMAIN_EVENT_SUSPENDED_MIGRATED
19:38:49 Zhan[m] right, but if migration fails at last, then we can just rollback the port binding change back to source
19:39:11 sean-k-mooney no becaue if the migtion failed that does not mean the guest crashed
19:39:20 sean-k-mooney it can unpase on the souce
19:39:35 sean-k-mooney so we dont want to have to rool back the prot bidning in that case
19:40:38 Zhan[m] hmm why we can't?
19:41:23 sean-k-mooney i was goign tot setted modfiyin tghis to just tirger post-live-migration and see what happesn in ci
19:41:48 sean-k-mooney https://github.com/openstack/nova/commit/aa87b9c288d316b85079e681e0df24354ec1912c
19:42:09 sean-k-mooney well because of https://bugs.launchpad.net/nova/+bug/1788014
19:42:31 Zhan[m] right that is what I put in that bug report
19:43:06 Zhan[m] the question is "As a result, failed live migrations will inadvertantly trigger activation of the port bindings on the destination host, which deactivates the source host port bindings, and then _rollback_live_migration will delete those activated dest host port bindings and leave the source host port bindings deactivated."
19:43:23 Zhan[m] so let's just rollback the port binding changes in _rollback_live_migration?
19:43:56 Zhan[m] for failure cases like this.
19:44:13 sean-k-mooney if the migration failure does nto kill the vm and it resulmes activating the port binding on the dest will deactivate tehm on the souce and some network backend will stop delivering packet to the soruce
19:44:38 sean-k-mooney so we woudl be addigng nework downtiem in the failure case that did nto exist before
19:44:52 sean-k-mooney however that may or may not be accpatable
19:46:44 sean-k-mooney im going to quickly try hacking somethign to see how it behviaves one sec
19:49:31 Zhan[m] I tested this (rollback when failure) in my environment, it works properly, like I see port binding gets updated on dest, then I wrote a function to roll back (basically delete, create, and re-activate bindings on source), then the VM regained network connectivity and is still on source.
19:50:58 Zhan[m] yes we would add more downtime, but I would think that failure doesn't really happen that often (correct me if I'm wrong), so we would be trading benefits for something happens less frequent for benefits for something happens more frequent (migration success case).

Earlier   Later