| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-11-04 | |||
| 16:20:06 | sean-k-mooney | thanks | |
| 16:29:37 | Zhan[m] | hi friends, I'm trying to work on some improvements for pre-copy live migrations - to reduce VMs' network connectivity loss time. I put my ideas and research in https://bugs.launchpad.net/nova/+bug/2128665 and would love some feedback before I start coding and making changes. does anyone mind taking a look? I can create a blueprint if it's needed. thanks!! | |
| 16:45:04 | sean-k-mooney | Zhan[m]: you are effectivly askign to trigger post-live-migration very slightly earlier | |
| 16:45:22 | sean-k-mooney | when pre-copy triggers a pause of the guest. | |
| 16:45:39 | sean-k-mooney | that only happens if you have set force-complete as the timeout action or if you manually do that | |
| 16:46:06 | sean-k-mooney | we do this for post-copy migration already when it enter the post copy phase | |
| 16:46:32 | sean-k-mooney | im not sure how safe ti woudl be to do that in this case vs the current migration completion event | |
| 16:46:54 | sean-k-mooney | Zhan[m]: this likely need more then a bug ie. perhaps a spec | |
| 16:47:26 | sean-k-mooney | or atleast more condierdation the a quick review of your propal to think though the implciiton for upgrades and the diffenent failure modes | |
| 16:49:15 | Zhan[m] | understood, I will go ahead and create a spec. wanna get some early feedback to see if this idea make sense or not. | |
| 16:49:48 | Zhan[m] | we are not using post-copy (I know that we'll avoid this with post-copy) so hence this is just for pre-copy | |
| 16:49:50 | sean-k-mooney | we might be able to use a slightly earlier event | |
| 16:50:40 | Zhan[m] | from my understanding, the paused event should be the earliest? (basically when the process is paused for switch-over) | |
| 16:51:24 | sean-k-mooney | yes but we need to think about the recover cases if there si a failure after pause | |
| 16:51:39 | sean-k-mooney | we currently do it in complete because we know we cant rollback at that point | |
| 16:52:20 | sean-k-mooney | Zhan[m]: just to confirm you are currenlty using pre-copy not post-copy | |
| 16:52:27 | sean-k-mooney | is there a reason you are not using post-copy | |
| 16:55:56 | Zhan[m] | > if there si a failure after pause... (full message at <https://matrix.org/oftc/media/v1/media/download/AQcXzaU3yYgbieAMs5LdQxhfkMQjWLQ8n98jM_lIB8gwWH9ApwZ_vdawDM9LNz0brOl_UevcPCkv3f-67_yYcUZCeal-T3mAAG1hdHJpeC5vcmcvRFZtSG9LWGxZaFNCZ0hrYnpXbVh0WnNi>) | |
| 16:56:15 | Zhan[m] | * - if there si a failure after pause... (full message at <https://matrix.org/oftc/media/v1/media/download/AYPX5olphEwNM-oVQEAeW1n4bjBMOCL7JcBa2--xwCdXhT0nA_sPuR_jnKT0C0oDEmSyzPWDZpPKsZ1wc-FdO-dCeal-VAbgAG1hdHJpeC5vcmcvRUtYVUxDZVBiekRLck1wZ0hpSW1ZVWNn>) | |
| 16:56:51 | Zhan[m] | * - if there si a failure after pause... (full message at <https://matrix.org/oftc/media/v1/media/download/AfGmfWXp6q289_-pAg_3r2CNmHk8GLA3AAnfqOAWmBViUGQMwTDwe8p12QyJHYlvI7HoFg0OieVLTgc2FNFdS1dCeal-XNrgAG1hdHJpeC5vcmcvZkprTUVOeWlRR0htWmlYc2hLZVhYZ0ha>) | |
| 16:58:00 | Zhan[m] | * - if there si a failure after pause... (full message at <https://matrix.org/oftc/media/v1/media/download/AQbdtBpPhkIEZJLqRgxshCfnuFsmdpJyW0XBuScJnchhklfJJ0L11j5bwq4dTpkoh88wJMokcThxBRTCLw1ECLJCeal-bZagAG1hdHJpeC5vcmcvQXNCTUJqVWtqUWF0cmJKRG1XRURxbmF1>) | |
| 16:58:37 | Zhan[m] | I think this change can help regardless, maybe in some environments, pre-copy migrations are preferred. | |
| 17:05:50 | Zhan[m] | We've also seen some crazy wait time (20+ seconds) between when the VM is paused and when libvirt says the migration is done (the completion event), so that's where our motivation comes from. | |
| 17:48:08 | gibi | I saw working live migration with native threaded compute \o/ | |
| 17:48:11 | gibi | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_071/openstack/07142d96394445fc9d4d5ad58b963223/testr_results.html | |
| 18:12:30 | sean-k-mooney | gibi: nice | |
| 18:14:10 | sean-k-mooney | gibi: do you know what is causing the volume issues | |
| 18:14:17 | gibi | not yet | |
| 18:14:41 | sean-k-mooney | well its less broken then it could have been | |
| 18:14:54 | gibi | I'm still not convinced that the lifecycle events are actually working between libvirt and nova in the gate. Locally it is working for me so I need to look more tomorrow | |
| 18:14:57 | sean-k-mooney | i assum this is lvm? | |
| 18:15:57 | sean-k-mooney | ok ya it is cidner lvm | |
| 18:16:57 | gibi | yeah it is not ceph I'm sure | |
| 18:17:08 | gibi | but I drop now. Continue from here tomorrow | |
| 18:17:26 | sean-k-mooney | gibi: looksl ilke there are are a buch of multipahd error bug i cant tell at a glance if that brick being brick or actully imporant | |
| 18:19:18 | sean-k-mooney | gibi: when we stop monkeypatching nova-comptue | |
| 18:19:28 | sean-k-mooney | we ar ealso unmonkeypatchign os-brick | |
| 18:19:40 | sean-k-mooney | i hope that is not related | |
| 18:20:16 | sean-k-mooney | i do see that some of the iscsiadm commadns failed os hopefuly it a unrealted ci faliure and a recheck will jsut work | |
| 18:20:42 | sean-k-mooney | most of the time the iscsiadm dicovery is workign so it coudl be load/timing related | |
| 18:49:28 | sean-k-mooney | Zhan[m]: multi line matix message are hard to read on the irc side as we only get the first line and link to the rest of the message | |
| 18:49:52 | sean-k-mooney | Zhan[m]: but no i htink the issue you are hitting is sepcific to pre-copy migration | |
| 18:50:17 | Zhan[m] | ah apologies, first time using this | |
| 18:50:42 | sean-k-mooney | thats ok | |
| 18:50:51 | sean-k-mooney | we also get a new message everytime you edit a message | |
| 18:51:28 | Zhan[m] | yes this is specific to pre-copy. When I was reading https://docs.openstack.org/nova/2024.1/configuration/config.html#libvirt.live_migration_permit_post_copy, I think it's saying that post-copy won't be used until we hit timeout right? | |
| 18:52:35 | sean-k-mooney | no post-copy will be enabeld by libvirt without nova's intervention or contol | |
| 18:52:57 | sean-k-mooney | the way post-copy works is qemu does an inital precoppy pahse whiel the instnace is running | |
| 18:53:17 | sean-k-mooney | it then internally decied if it is making enough forward progress to proceed with a standar pre-copy migration | |
| 18:54:14 | sean-k-mooney | if not after the 2nd pre-copy compeltes it will swap to post copy mode where the vm will be resumed on teh dest | |
| 18:54:34 | sean-k-mooney | we trigger off the suspend call in post-copy case | |
| 18:56:54 | sean-k-mooney | Zhan[m]: effectivly what nova does is tell qemu it is allwoed to use post-copy and it will make the decion of when to do it | |
| 18:56:56 | Zhan[m] | thanks, that's interesting to know, I was thinking we are controlling this with Nova when reading the config | |
| 18:57:29 | sean-k-mooney | the config option just tell nova to pass the flag to librit to give it permision to use post-copy | |
| 18:58:12 | Zhan[m] | but talking back, do you think this is something that can be improved on for pre-copy? | |
| 18:58:24 | Zhan[m] | I'll definitely play around with post-copy too. | |
| 18:59:51 | sean-k-mooney | im unsure https://specs.openstack.org/openstack/nova-specs/specs/rocky/implemented/neutron-new-port-binding-api.html is the spec where we defien dthe current sematicn for prot activation during live migration | |
| 19:00:57 | sean-k-mooney | if we change when we do the activateion (it may be ok) we need ot review the libvirt docuemtion and lifecycle fo migrton evnet | |
| 19:02:07 | Zhan[m] | yes I will list relevant events, job stats, and related stuff in the spec, as well as any edge cases I can think of. | |
| 19:02:16 | sean-k-mooney | Zhan[m]: today the only way to do the activation sooner is to do post_live_migration sooner which may or may not be safe | |
| 19:02:39 | sean-k-mooney | failing that we need to register a specific addtional event handeler just for the port activation case | |
| 19:03:19 | sean-k-mooney | that shoudl be possible but post_live_migration woudl have to handel the fact that the bining is actived earlier and they clean up code for error/abort woudl too | |
| 19:03:27 | Zhan[m] | Currently, I'm thinking we just do the networking part - activate port bindings earlier like what's done previously | |
| 19:03:58 | sean-k-mooney | well when we activate it on the test network backend are ment to start routing traffic to the dest as well | |
| 19:04:23 | sean-k-mooney | in the past with ovn that would have resulting in packet beign rescived on the source host being drop | |
| 19:04:56 | Zhan[m] | I'm thinking adding addition functions in the rollback function in case the live migration fails later. | |
| 19:04:56 | sean-k-mooney | today we replciat the packt to the inactive port bidnign sepcified in the migrating_to filed in the port bidning profile | |
| 19:05:14 | Zhan[m] | right, we're not using ovn so that might be something to consider too... | |
| 19:05:32 | sean-k-mooney | ya so this is why im sayin gwe might need a sepc for this | |
| 19:05:44 | sean-k-mooney | because what ever we do has to work correctly for all neutorn network backends | |
| 19:06:10 | Zhan[m] | no I'm not objecting a writing a spec, I'll work on that. I'm just asking what do we feel about this (like if it worth a try at all | |
| 19:07:03 | sean-k-mooney | by the way to beclear the VIR_DOMAIN_JOB_COMPLETED is waht tirggers post live migration wehn doing precoppy today | |
| 19:07:09 | Zhan[m] | yeah indeed, lemme get that spec out, as it seems possible as long as we handle the cases correctly | |
| 19:07:36 | Zhan[m] | yeah,but what I found is that the query of job status will hang when libvirt is cleaning up the domain | |
| 19:08:17 | sean-k-mooney | ya so your suggestin using VIR_DOMAIN_EVENT_SUSPENDED_MIGRATED right | |
| 19:08:24 | Zhan[m] | so even though it is saying "VIR_DOMAIN_JOB_COMPLETED -> job finished, but cleanup is not finished", it is very likely that the queryJobInfo by nova will be blocked until the cleanup it done | |
| 19:08:25 | sean-k-mooney | similar to VIR_DOMAIN_EVENT_SUSPENDED_POSTCOPY | |
| 19:08:29 | Zhan[m] | yes | |
| 19:08:54 | sean-k-mooney | so VIR_DOMAIN_EVENT_SUSPENDED_MIGRATED woudl be sent when the vm is paused in pre-copy or when you do force-complete via the api | |
| 19:09:06 | sean-k-mooney | or the migrtion timeout action does force-complete | |
| 19:09:39 | sean-k-mooney | what we woudl need to confirm is that VIR_DOMAIN_EVENT_SUSPENDED_MIGRATED is not emitted if auto converge is enabled | |
| 19:10:18 | sean-k-mooney | if you set live_migration_permit_auto_converge | |
| 19:10:30 | Zhan[m] | why not? auto converge is just to make sure that the migration will eventually finish right? | |
| 19:10:34 | sean-k-mooney | it allows qemu to pause the vcpu temporally | |
| 19:10:42 | sean-k-mooney | to allow pre-copy to make progress | |
| 19:10:44 | Zhan[m] | we have auto converge on and I do see VIR_DOMAIN_EVENT_SUSPENDED_MIGRATED gets generated | |
| 19:11:08 | sean-k-mooney | right but my point is its only safe to use VIR_DOMAIN_EVENT_SUSPENDED_MIGRATED | |
| 19:11:09 | Zhan[m] | the VM needs to paused before the swtich-over, regardless of converge or not, right? | |
| 19:11:14 | sean-k-mooney | if its only sent once | |
| 19:11:26 | sean-k-mooney | when we are never going to resume on this host again | |
| 19:11:47 | Zhan[m] | good point, I believe it is only sent once, I'll double check that | |
| 19:12:29 | sean-k-mooney | if VIR_DOMAIN_EVENT_SUSPENDED_MIGRATED is a point of no return (we will never resume on the source after that) then | |
| 19:12:38 | sean-k-mooney | i think we can move to useign that | |
| 19:13:12 | sean-k-mooney | but we will obviuly need to test hat | |
| 19:13:16 | sean-k-mooney | *that | |
| 19:13:46 | Zhan[m] | but even if it's resumed later (likely due to the migration fails), libvirt will report error and we can still roll back the port bindings to source? | |
| 19:14:05 | sean-k-mooney | Zhan[m]: since your using calico your proably also interest in https://bugs.launchpad.net/nova/+bug/2033681 | |
| 19:14:26 | sean-k-mooney | Zhan[m]: not if we have invoked post-live-migration | |
| 19:14:40 | sean-k-mooney | which was my point before if we are not adding a speciall addtional handeler | |
| 19:14:53 | sean-k-mooney | and jsut calling post-live-migration like we do for post-copy | |