| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-11-04 | |||
| 19:17:31 | Zhan[m] | I do have some code in hand and will be starting testing while writing the spec, I can link it there | |
| 19:19:12 | Zhan[m] | ok I think we are on the same page, lemme write that spec and put details there | |
| 19:19:12 | sean-k-mooney | https://docs.openstack.org/nova/latest/reference/live-migration.html | |
| 19:19:23 | sean-k-mooney | the main concen i have is you are changing the current workflow | |
| 19:19:45 | sean-k-mooney | by adding another pahse for network activation | |
| 19:19:49 | Zhan[m] | yes, because before, for precopy, we only activate bindings when we see the migration is good | |
| 19:20:02 | sean-k-mooney | well when its complete | |
| 19:20:43 | Zhan[m] | complete and good (Ig :P) | |
| 19:20:48 | sean-k-mooney | the lbivrt part where we block should typeicly be very very short | |
| 19:21:03 | sean-k-mooney | unelss you are activly copying memry for the guest | |
| 19:21:24 | Zhan[m] | no... in our environment we saw some crz numbers like 20s+ | |
| 19:21:27 | Zhan[m] | or even a minute | |
| 19:21:35 | Zhan[m] | between PAUSED to job completed event | |
| 19:21:56 | Zhan[m] | because libvirt waits for the source QEMU process to be cleaned up before saying job is completed | |
| 19:21:57 | sean-k-mooney | Zhan[m]: https://docs.openstack.org/nova/latest/configuration/config.html#workarounds.enable_qemu_monitor_announce_self | |
| 19:22:02 | sean-k-mooney | by the way do you enable ^ | |
| 19:22:30 | sean-k-mooney | Zhan[m]: right but that shoudl only happen if qemu is copying memory | |
| 19:22:55 | sean-k-mooney | i guess the qemu instance clean up coudl take a long time if it waithign for io to complete | |
| 19:23:26 | sean-k-mooney | but libvirt shoudl not unpasue the dest instance until the source is stoped. | |
| 19:23:32 | Zhan[m] | what I found was that the kernel was spinning to clean up the memory resource | |
| 19:23:41 | Zhan[m] | which took quite a lot of time | |
| 19:24:09 | Zhan[m] | I dont think we enabled that, lemme take a look | |
| 19:24:51 | Zhan[m] | the memory resource -> the memory space that the VM uses, not by migration. if the VM is using a huge number of pages then the cleanup can take longer. | |
| 19:24:55 | sean-k-mooney | Zhan[m]: does the guest use hugepages or anonoums 4k pages out fo interest | |
| 19:25:22 | sean-k-mooney | im just wonderign why that takes so long in your case | |
| 19:25:55 | Zhan[m] | we use THP, but we figured that KSM and this bug(https://lore.kernel.org/all/20240626191830.3819324-1-yang@os.amperecomputing.com/) was causing it to not use hugepages as expected. | |
| 19:26:09 | sean-k-mooney | ah ok | |
| 19:26:29 | sean-k-mooney | THP is awsome if it works and terible otherwise | |
| 19:26:36 | sean-k-mooney | espically when combind iwht ksm | |
| 19:26:44 | sean-k-mooney | it causes a lot of non determinism | |
| 19:27:03 | Zhan[m] | exactly lol, but yeah, that's why we also wanna tweak nova to just skip the cleanup | |
| 19:28:55 | sean-k-mooney | hum i was not aware of that bug but reading it that definly going to slow things donw | |
| 19:29:12 | sean-k-mooney | is that fixed upstream? | |
| 19:29:19 | Zhan[m] | yes, what QEMU does in migration is that it will read the page first and then write | |
| 19:29:24 | Zhan[m] | it's fixed in 6.13 IIRC | |
| 19:29:46 | Zhan[m] | so all of the following writes done by the VM will be regular pages | |
| 19:29:57 | Zhan[m] | which is painful.... | |
| 19:30:31 | sean-k-mooney | ah ok | |
| 19:30:34 | Zhan[m] | https://github.com/torvalds/linux/commit/5c00ff742bf5caf85f60e1c73999f99376fb865d yeah 6.13 | |
| 19:30:57 | sean-k-mooney | 6.13 is resonably new so that proably not in a lot of disto kernels | |
| 19:31:39 | sean-k-mooney | ubutnu has 6.14 in the HWE kernsl but it default to 6.11 for 24.04 | |
| 19:31:59 | Zhan[m] | yeah but anyways, nova doesn't really need to wait for the cleanup to be done. | |
| 19:31:59 | sean-k-mooney | rhel 10 is 6.12 + backports | |
| 19:32:09 | Zhan[m] | if it's doing cleanup, then it means the VM has been resumed on dest | |
| 19:32:19 | sean-k-mooney | ya | |
| 19:32:29 | Zhan[m] | which we can start at least setting up the network | |
| 19:32:37 | sean-k-mooney | if we can trigger form that point we likely cna just go direclty ot post-live-migrate | |
| 19:33:08 | sean-k-mooney | but we cant relibly tirgger on the cleanup starting then we can expore just doing the network activation | |
| 19:34:08 | Zhan[m] | yeah just the network I feel is fine, everything else can be pushed back | |
| 19:34:26 | Zhan[m] | what VM users ultimately care about is just network connectivity and if the VM is running or not | |
| 19:34:49 | Zhan[m] | which in that case the VM is running so we should setup the network | |
| 19:36:01 | sean-k-mooney | what i woudl suggest is push a patch to hack https://github.com/openstack/nova/blob/master/nova/virt/libvirt/host.py#L274-L321 | |
| 19:36:21 | sean-k-mooney | oh | |
| 19:36:23 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/host.py#L295-L298 | |
| 19:36:56 | Zhan[m] | right, I'm thinking if the migration fails, we rollback the port bindings back to source | |
| 19:37:11 | sean-k-mooney | Zhan[m]: so based on that comment we cant realy on that only beign sent on sucssess | |
| 19:37:20 | sean-k-mooney | whgich mean we have to call guest.get_job_info() | |
| 19:37:42 | Zhan[m] | right but that get_job_info will hang until cleanup is done | |
| 19:37:48 | sean-k-mooney | right | |
| 19:38:11 | sean-k-mooney | but we cant just assume its ok to activate the netork on the dest based on VIR_DOMAIN_EVENT_SUSPENDED_MIGRATED | |
| 19:38:49 | Zhan[m] | right, but if migration fails at last, then we can just rollback the port binding change back to source | |
| 19:39:11 | sean-k-mooney | no becaue if the migtion failed that does not mean the guest crashed | |
| 19:39:20 | sean-k-mooney | it can unpase on the souce | |
| 19:39:35 | sean-k-mooney | so we dont want to have to rool back the prot bidning in that case | |
| 19:40:38 | Zhan[m] | hmm why we can't? | |
| 19:41:23 | sean-k-mooney | i was goign tot setted modfiyin tghis to just tirger post-live-migration and see what happesn in ci | |
| 19:41:48 | sean-k-mooney | https://github.com/openstack/nova/commit/aa87b9c288d316b85079e681e0df24354ec1912c | |
| 19:42:09 | sean-k-mooney | well because of https://bugs.launchpad.net/nova/+bug/1788014 | |
| 19:42:31 | Zhan[m] | right that is what I put in that bug report | |
| 19:43:06 | Zhan[m] | the question is "As a result, failed live migrations will inadvertantly trigger activation of the port bindings on the destination host, which deactivates the source host port bindings, and then _rollback_live_migration will delete those activated dest host port bindings and leave the source host port bindings deactivated." | |
| 19:43:23 | Zhan[m] | so let's just rollback the port binding changes in _rollback_live_migration? | |
| 19:43:56 | Zhan[m] | for failure cases like this. | |
| 19:44:13 | sean-k-mooney | if the migration failure does nto kill the vm and it resulmes activating the port binding on the dest will deactivate tehm on the souce and some network backend will stop delivering packet to the soruce | |
| 19:44:38 | sean-k-mooney | so we woudl be addigng nework downtiem in the failure case that did nto exist before | |
| 19:44:52 | sean-k-mooney | however that may or may not be accpatable | |
| 19:46:44 | sean-k-mooney | im going to quickly try hacking somethign to see how it behviaves one sec | |
| 19:49:31 | Zhan[m] | I tested this (rollback when failure) in my environment, it works properly, like I see port binding gets updated on dest, then I wrote a function to roll back (basically delete, create, and re-activate bindings on source), then the VM regained network connectivity and is still on source. | |
| 19:50:58 | Zhan[m] | yes we would add more downtime, but I would think that failure doesn't really happen that often (correct me if I'm wrong), so we would be trading benefits for something happens less frequent for benefits for something happens more frequent (migration success case). | |
| 19:51:12 | Zhan[m] | *downtime for failure cases | |
| 19:52:31 | opendevreview | sean mooney proposed openstack/nova master: [DNM] testing early activation https://review.opendev.org/c/openstack/nova/+/966106 | |
| 19:53:39 | sean-k-mooney | Zhan[m]: in the happy path i think that is effectivly what you want to do | |
| 19:53:53 | Zhan[m] | yes | |
| 19:54:15 | Zhan[m] | the only concern I think is the unhappy path :) | |
| 19:56:08 | sean-k-mooney | ya, so the reason im debating this in my head is this is already complciated and im not sure we want to make it more complicted by adding aother event handeler | |
| 19:57:47 | Zhan[m] | To ease the process a little bit, so basically: 1. activate port bindings when paused 2. If migration ok later, all is well 3. If migration not ok later, delete, recreate, reactivate the port binding on the source side | |
| 19:58:22 | sean-k-mooney | well you just activate them again and delete the dest ones | |
| 19:58:24 | Zhan[m] | It is complicated :( I spent quite some time digging all these too | |
| 19:58:49 | sean-k-mooney | but you also have to cater for neturon backend that dont supprot multipel port bidnign | |
| 19:58:59 | sean-k-mooney | because we have not made those unsupproted yet | |
| 19:59:42 | sean-k-mooney | there are some network backend that dont supprot the multipe prot bidnging exteion and in those case we only have one and we jsut update the bindings:host field | |
| 19:59:42 | Zhan[m] | I'm not sure on that, I got exceptions when I try to re-enable the source bindings and I saw some fields are missing in DB, so that's why I recreated the bindings, but this is not important, the idea is the same. | |
| 20:00:09 | sean-k-mooney | i gues it depend on if they are delete or not | |
| 20:00:22 | Zhan[m] | right, if backend doesn't support I think the port binding extension will not be used right? | |
| 20:00:33 | sean-k-mooney | but ya we have to recrate them if they are i though activating them only deactived teh souce one but didnt delete tehm | |
| 20:00:37 | Zhan[m] | yeah in that case migrate_instance_start won't do anything, so | |
| 20:01:38 | Zhan[m] | they are not deleted, but as far as what I'm seeing in the DB, some metadata are deleted so hmmm not very sure on that | |
| 20:01:46 | sean-k-mooney | so what i woudl like is if we do have a dedicated port biding activation phase we use it for both pre and post copy | |
| 20:01:55 | sean-k-mooney | that is how it works today | |
| 20:02:36 | sean-k-mooney | what i dont wnat to have is 1 way to activate them for pre-copy, 1 way for post copy adn 1 way for backend that dont suprot it | |
| 20:02:38 | Zhan[m] | ok so like extracting it out of post live migration and event handler function? | |