| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-11-04 | |||
| 19:08:17 | sean-k-mooney | ya so your suggestin using VIR_DOMAIN_EVENT_SUSPENDED_MIGRATED right | |
| 19:08:24 | Zhan[m] | so even though it is saying "VIR_DOMAIN_JOB_COMPLETED -> job finished, but cleanup is not finished", it is very likely that the queryJobInfo by nova will be blocked until the cleanup it done | |
| 19:08:25 | sean-k-mooney | similar to VIR_DOMAIN_EVENT_SUSPENDED_POSTCOPY | |
| 19:08:29 | Zhan[m] | yes | |
| 19:08:54 | sean-k-mooney | so VIR_DOMAIN_EVENT_SUSPENDED_MIGRATED woudl be sent when the vm is paused in pre-copy or when you do force-complete via the api | |
| 19:09:06 | sean-k-mooney | or the migrtion timeout action does force-complete | |
| 19:09:39 | sean-k-mooney | what we woudl need to confirm is that VIR_DOMAIN_EVENT_SUSPENDED_MIGRATED is not emitted if auto converge is enabled | |
| 19:10:18 | sean-k-mooney | if you set live_migration_permit_auto_converge | |
| 19:10:30 | Zhan[m] | why not? auto converge is just to make sure that the migration will eventually finish right? | |
| 19:10:34 | sean-k-mooney | it allows qemu to pause the vcpu temporally | |
| 19:10:42 | sean-k-mooney | to allow pre-copy to make progress | |
| 19:10:44 | Zhan[m] | we have auto converge on and I do see VIR_DOMAIN_EVENT_SUSPENDED_MIGRATED gets generated | |
| 19:11:08 | sean-k-mooney | right but my point is its only safe to use VIR_DOMAIN_EVENT_SUSPENDED_MIGRATED | |
| 19:11:09 | Zhan[m] | the VM needs to paused before the swtich-over, regardless of converge or not, right? | |
| 19:11:14 | sean-k-mooney | if its only sent once | |
| 19:11:26 | sean-k-mooney | when we are never going to resume on this host again | |
| 19:11:47 | Zhan[m] | good point, I believe it is only sent once, I'll double check that | |
| 19:12:29 | sean-k-mooney | if VIR_DOMAIN_EVENT_SUSPENDED_MIGRATED is a point of no return (we will never resume on the source after that) then | |
| 19:12:38 | sean-k-mooney | i think we can move to useign that | |
| 19:13:12 | sean-k-mooney | but we will obviuly need to test hat | |
| 19:13:16 | sean-k-mooney | *that | |
| 19:13:46 | Zhan[m] | but even if it's resumed later (likely due to the migration fails), libvirt will report error and we can still roll back the port bindings to source? | |
| 19:14:05 | sean-k-mooney | Zhan[m]: since your using calico your proably also interest in https://bugs.launchpad.net/nova/+bug/2033681 | |
| 19:14:26 | sean-k-mooney | Zhan[m]: not if we have invoked post-live-migration | |
| 19:14:40 | sean-k-mooney | which was my point before if we are not adding a speciall addtional handeler | |
| 19:14:53 | sean-k-mooney | and jsut calling post-live-migration like we do for post-copy | |
| 19:15:19 | sean-k-mooney | then we need to be 100% sure it cant role back and that its safe to start cleaning up the souce on the nova side | |
| 19:15:31 | sean-k-mooney | if we only do the port bindign activation it probaly ok | |
| 19:16:21 | sean-k-mooney | what we might do is move the port bidning activation logic to its own method and have that tirger on either VIR_DOMAIN_EVENT_SUSPENDED_POSTCOPY or VIR_DOMAIN_EVENT_SUSPENDED_POSTCOPY | |
| 19:16:31 | sean-k-mooney | *VIR_DOMAIN_EVENT_SUSPENDED_POSTCOPY or VIR_DOMAIN_EVENT_SUSPENDED_MIGRATED | |
| 19:16:31 | Zhan[m] | If we have invoked post live migration then it must mean the migration succeeded, so we are ok to move everything to dest | |
| 19:16:54 | Zhan[m] | right, that's what I'm thinking | |
| 19:17:31 | Zhan[m] | I do have some code in hand and will be starting testing while writing the spec, I can link it there | |
| 19:19:12 | sean-k-mooney | https://docs.openstack.org/nova/latest/reference/live-migration.html | |
| 19:19:12 | Zhan[m] | ok I think we are on the same page, lemme write that spec and put details there | |
| 19:19:23 | sean-k-mooney | the main concen i have is you are changing the current workflow | |
| 19:19:45 | sean-k-mooney | by adding another pahse for network activation | |
| 19:19:49 | Zhan[m] | yes, because before, for precopy, we only activate bindings when we see the migration is good | |
| 19:20:02 | sean-k-mooney | well when its complete | |
| 19:20:43 | Zhan[m] | complete and good (Ig :P) | |
| 19:20:48 | sean-k-mooney | the lbivrt part where we block should typeicly be very very short | |
| 19:21:03 | sean-k-mooney | unelss you are activly copying memry for the guest | |
| 19:21:24 | Zhan[m] | no... in our environment we saw some crz numbers like 20s+ | |
| 19:21:27 | Zhan[m] | or even a minute | |
| 19:21:35 | Zhan[m] | between PAUSED to job completed event | |
| 19:21:56 | Zhan[m] | because libvirt waits for the source QEMU process to be cleaned up before saying job is completed | |
| 19:21:57 | sean-k-mooney | Zhan[m]: https://docs.openstack.org/nova/latest/configuration/config.html#workarounds.enable_qemu_monitor_announce_self | |
| 19:22:02 | sean-k-mooney | by the way do you enable ^ | |
| 19:22:30 | sean-k-mooney | Zhan[m]: right but that shoudl only happen if qemu is copying memory | |
| 19:22:55 | sean-k-mooney | i guess the qemu instance clean up coudl take a long time if it waithign for io to complete | |
| 19:23:26 | sean-k-mooney | but libvirt shoudl not unpasue the dest instance until the source is stoped. | |
| 19:23:32 | Zhan[m] | what I found was that the kernel was spinning to clean up the memory resource | |
| 19:23:41 | Zhan[m] | which took quite a lot of time | |
| 19:24:09 | Zhan[m] | I dont think we enabled that, lemme take a look | |
| 19:24:51 | Zhan[m] | the memory resource -> the memory space that the VM uses, not by migration. if the VM is using a huge number of pages then the cleanup can take longer. | |
| 19:24:55 | sean-k-mooney | Zhan[m]: does the guest use hugepages or anonoums 4k pages out fo interest | |
| 19:25:22 | sean-k-mooney | im just wonderign why that takes so long in your case | |
| 19:25:55 | Zhan[m] | we use THP, but we figured that KSM and this bug(https://lore.kernel.org/all/20240626191830.3819324-1-yang@os.amperecomputing.com/) was causing it to not use hugepages as expected. | |
| 19:26:09 | sean-k-mooney | ah ok | |
| 19:26:29 | sean-k-mooney | THP is awsome if it works and terible otherwise | |
| 19:26:36 | sean-k-mooney | espically when combind iwht ksm | |
| 19:26:44 | sean-k-mooney | it causes a lot of non determinism | |
| 19:27:03 | Zhan[m] | exactly lol, but yeah, that's why we also wanna tweak nova to just skip the cleanup | |
| 19:28:55 | sean-k-mooney | hum i was not aware of that bug but reading it that definly going to slow things donw | |
| 19:29:12 | sean-k-mooney | is that fixed upstream? | |
| 19:29:19 | Zhan[m] | yes, what QEMU does in migration is that it will read the page first and then write | |
| 19:29:24 | Zhan[m] | it's fixed in 6.13 IIRC | |
| 19:29:46 | Zhan[m] | so all of the following writes done by the VM will be regular pages | |
| 19:29:57 | Zhan[m] | which is painful.... | |
| 19:30:31 | sean-k-mooney | ah ok | |
| 19:30:34 | Zhan[m] | https://github.com/torvalds/linux/commit/5c00ff742bf5caf85f60e1c73999f99376fb865d yeah 6.13 | |
| 19:30:57 | sean-k-mooney | 6.13 is resonably new so that proably not in a lot of disto kernels | |
| 19:31:39 | sean-k-mooney | ubutnu has 6.14 in the HWE kernsl but it default to 6.11 for 24.04 | |
| 19:31:59 | sean-k-mooney | rhel 10 is 6.12 + backports | |
| 19:31:59 | Zhan[m] | yeah but anyways, nova doesn't really need to wait for the cleanup to be done. | |
| 19:32:09 | Zhan[m] | if it's doing cleanup, then it means the VM has been resumed on dest | |
| 19:32:19 | sean-k-mooney | ya | |
| 19:32:29 | Zhan[m] | which we can start at least setting up the network | |
| 19:32:37 | sean-k-mooney | if we can trigger form that point we likely cna just go direclty ot post-live-migrate | |
| 19:33:08 | sean-k-mooney | but we cant relibly tirgger on the cleanup starting then we can expore just doing the network activation | |
| 19:34:08 | Zhan[m] | yeah just the network I feel is fine, everything else can be pushed back | |
| 19:34:26 | Zhan[m] | what VM users ultimately care about is just network connectivity and if the VM is running or not | |
| 19:34:49 | Zhan[m] | which in that case the VM is running so we should setup the network | |
| 19:36:01 | sean-k-mooney | what i woudl suggest is push a patch to hack https://github.com/openstack/nova/blob/master/nova/virt/libvirt/host.py#L274-L321 | |
| 19:36:21 | sean-k-mooney | oh | |
| 19:36:23 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/host.py#L295-L298 | |
| 19:36:56 | Zhan[m] | right, I'm thinking if the migration fails, we rollback the port bindings back to source | |
| 19:37:11 | sean-k-mooney | Zhan[m]: so based on that comment we cant realy on that only beign sent on sucssess | |
| 19:37:20 | sean-k-mooney | whgich mean we have to call guest.get_job_info() | |
| 19:37:42 | Zhan[m] | right but that get_job_info will hang until cleanup is done | |
| 19:37:48 | sean-k-mooney | right | |
| 19:38:11 | sean-k-mooney | but we cant just assume its ok to activate the netork on the dest based on VIR_DOMAIN_EVENT_SUSPENDED_MIGRATED | |
| 19:38:49 | Zhan[m] | right, but if migration fails at last, then we can just rollback the port binding change back to source | |
| 19:39:11 | sean-k-mooney | no becaue if the migtion failed that does not mean the guest crashed | |
| 19:39:20 | sean-k-mooney | it can unpase on the souce | |
| 19:39:35 | sean-k-mooney | so we dont want to have to rool back the prot bidning in that case | |
| 19:40:38 | Zhan[m] | hmm why we can't? | |
| 19:41:23 | sean-k-mooney | i was goign tot setted modfiyin tghis to just tirger post-live-migration and see what happesn in ci | |
| 19:41:48 | sean-k-mooney | https://github.com/openstack/nova/commit/aa87b9c288d316b85079e681e0df24354ec1912c | |
| 19:42:09 | sean-k-mooney | well because of https://bugs.launchpad.net/nova/+bug/1788014 | |