| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-11-04 | |||
| 18:17:26 | sean-k-mooney | gibi: looksl ilke there are are a buch of multipahd error bug i cant tell at a glance if that brick being brick or actully imporant | |
| 18:19:18 | sean-k-mooney | gibi: when we stop monkeypatching nova-comptue | |
| 18:19:28 | sean-k-mooney | we ar ealso unmonkeypatchign os-brick | |
| 18:19:40 | sean-k-mooney | i hope that is not related | |
| 18:20:16 | sean-k-mooney | i do see that some of the iscsiadm commadns failed os hopefuly it a unrealted ci faliure and a recheck will jsut work | |
| 18:20:42 | sean-k-mooney | most of the time the iscsiadm dicovery is workign so it coudl be load/timing related | |
| 18:49:28 | sean-k-mooney | Zhan[m]: multi line matix message are hard to read on the irc side as we only get the first line and link to the rest of the message | |
| 18:49:52 | sean-k-mooney | Zhan[m]: but no i htink the issue you are hitting is sepcific to pre-copy migration | |
| 18:50:17 | Zhan[m] | ah apologies, first time using this | |
| 18:50:42 | sean-k-mooney | thats ok | |
| 18:50:51 | sean-k-mooney | we also get a new message everytime you edit a message | |
| 18:51:28 | Zhan[m] | yes this is specific to pre-copy. When I was reading https://docs.openstack.org/nova/2024.1/configuration/config.html#libvirt.live_migration_permit_post_copy, I think it's saying that post-copy won't be used until we hit timeout right? | |
| 18:52:35 | sean-k-mooney | no post-copy will be enabeld by libvirt without nova's intervention or contol | |
| 18:52:57 | sean-k-mooney | the way post-copy works is qemu does an inital precoppy pahse whiel the instnace is running | |
| 18:53:17 | sean-k-mooney | it then internally decied if it is making enough forward progress to proceed with a standar pre-copy migration | |
| 18:54:14 | sean-k-mooney | if not after the 2nd pre-copy compeltes it will swap to post copy mode where the vm will be resumed on teh dest | |
| 18:54:34 | sean-k-mooney | we trigger off the suspend call in post-copy case | |
| 18:56:54 | sean-k-mooney | Zhan[m]: effectivly what nova does is tell qemu it is allwoed to use post-copy and it will make the decion of when to do it | |
| 18:56:56 | Zhan[m] | thanks, that's interesting to know, I was thinking we are controlling this with Nova when reading the config | |
| 18:57:29 | sean-k-mooney | the config option just tell nova to pass the flag to librit to give it permision to use post-copy | |
| 18:58:12 | Zhan[m] | but talking back, do you think this is something that can be improved on for pre-copy? | |
| 18:58:24 | Zhan[m] | I'll definitely play around with post-copy too. | |
| 18:59:51 | sean-k-mooney | im unsure https://specs.openstack.org/openstack/nova-specs/specs/rocky/implemented/neutron-new-port-binding-api.html is the spec where we defien dthe current sematicn for prot activation during live migration | |
| 19:00:57 | sean-k-mooney | if we change when we do the activateion (it may be ok) we need ot review the libvirt docuemtion and lifecycle fo migrton evnet | |
| 19:02:07 | Zhan[m] | yes I will list relevant events, job stats, and related stuff in the spec, as well as any edge cases I can think of. | |
| 19:02:16 | sean-k-mooney | Zhan[m]: today the only way to do the activation sooner is to do post_live_migration sooner which may or may not be safe | |
| 19:02:39 | sean-k-mooney | failing that we need to register a specific addtional event handeler just for the port activation case | |
| 19:03:19 | sean-k-mooney | that shoudl be possible but post_live_migration woudl have to handel the fact that the bining is actived earlier and they clean up code for error/abort woudl too | |
| 19:03:27 | Zhan[m] | Currently, I'm thinking we just do the networking part - activate port bindings earlier like what's done previously | |
| 19:03:58 | sean-k-mooney | well when we activate it on the test network backend are ment to start routing traffic to the dest as well | |
| 19:04:23 | sean-k-mooney | in the past with ovn that would have resulting in packet beign rescived on the source host being drop | |
| 19:04:56 | sean-k-mooney | today we replciat the packt to the inactive port bidnign sepcified in the migrating_to filed in the port bidning profile | |
| 19:04:56 | Zhan[m] | I'm thinking adding addition functions in the rollback function in case the live migration fails later. | |
| 19:05:14 | Zhan[m] | right, we're not using ovn so that might be something to consider too... | |
| 19:05:32 | sean-k-mooney | ya so this is why im sayin gwe might need a sepc for this | |
| 19:05:44 | sean-k-mooney | because what ever we do has to work correctly for all neutorn network backends | |
| 19:06:10 | Zhan[m] | no I'm not objecting a writing a spec, I'll work on that. I'm just asking what do we feel about this (like if it worth a try at all | |
| 19:07:03 | sean-k-mooney | by the way to beclear the VIR_DOMAIN_JOB_COMPLETED is waht tirggers post live migration wehn doing precoppy today | |
| 19:07:09 | Zhan[m] | yeah indeed, lemme get that spec out, as it seems possible as long as we handle the cases correctly | |
| 19:07:36 | Zhan[m] | yeah,but what I found is that the query of job status will hang when libvirt is cleaning up the domain | |
| 19:08:17 | sean-k-mooney | ya so your suggestin using VIR_DOMAIN_EVENT_SUSPENDED_MIGRATED right | |
| 19:08:24 | Zhan[m] | so even though it is saying "VIR_DOMAIN_JOB_COMPLETED -> job finished, but cleanup is not finished", it is very likely that the queryJobInfo by nova will be blocked until the cleanup it done | |
| 19:08:25 | sean-k-mooney | similar to VIR_DOMAIN_EVENT_SUSPENDED_POSTCOPY | |
| 19:08:29 | Zhan[m] | yes | |
| 19:08:54 | sean-k-mooney | so VIR_DOMAIN_EVENT_SUSPENDED_MIGRATED woudl be sent when the vm is paused in pre-copy or when you do force-complete via the api | |
| 19:09:06 | sean-k-mooney | or the migrtion timeout action does force-complete | |
| 19:09:39 | sean-k-mooney | what we woudl need to confirm is that VIR_DOMAIN_EVENT_SUSPENDED_MIGRATED is not emitted if auto converge is enabled | |
| 19:10:18 | sean-k-mooney | if you set live_migration_permit_auto_converge | |
| 19:10:30 | Zhan[m] | why not? auto converge is just to make sure that the migration will eventually finish right? | |
| 19:10:34 | sean-k-mooney | it allows qemu to pause the vcpu temporally | |
| 19:10:42 | sean-k-mooney | to allow pre-copy to make progress | |
| 19:10:44 | Zhan[m] | we have auto converge on and I do see VIR_DOMAIN_EVENT_SUSPENDED_MIGRATED gets generated | |
| 19:11:08 | sean-k-mooney | right but my point is its only safe to use VIR_DOMAIN_EVENT_SUSPENDED_MIGRATED | |
| 19:11:09 | Zhan[m] | the VM needs to paused before the swtich-over, regardless of converge or not, right? | |
| 19:11:14 | sean-k-mooney | if its only sent once | |
| 19:11:26 | sean-k-mooney | when we are never going to resume on this host again | |
| 19:11:47 | Zhan[m] | good point, I believe it is only sent once, I'll double check that | |
| 19:12:29 | sean-k-mooney | if VIR_DOMAIN_EVENT_SUSPENDED_MIGRATED is a point of no return (we will never resume on the source after that) then | |
| 19:12:38 | sean-k-mooney | i think we can move to useign that | |
| 19:13:12 | sean-k-mooney | but we will obviuly need to test hat | |
| 19:13:16 | sean-k-mooney | *that | |
| 19:13:46 | Zhan[m] | but even if it's resumed later (likely due to the migration fails), libvirt will report error and we can still roll back the port bindings to source? | |
| 19:14:05 | sean-k-mooney | Zhan[m]: since your using calico your proably also interest in https://bugs.launchpad.net/nova/+bug/2033681 | |
| 19:14:26 | sean-k-mooney | Zhan[m]: not if we have invoked post-live-migration | |
| 19:14:40 | sean-k-mooney | which was my point before if we are not adding a speciall addtional handeler | |
| 19:14:53 | sean-k-mooney | and jsut calling post-live-migration like we do for post-copy | |
| 19:15:19 | sean-k-mooney | then we need to be 100% sure it cant role back and that its safe to start cleaning up the souce on the nova side | |
| 19:15:31 | sean-k-mooney | if we only do the port bindign activation it probaly ok | |
| 19:16:21 | sean-k-mooney | what we might do is move the port bidning activation logic to its own method and have that tirger on either VIR_DOMAIN_EVENT_SUSPENDED_POSTCOPY or VIR_DOMAIN_EVENT_SUSPENDED_POSTCOPY | |
| 19:16:31 | sean-k-mooney | *VIR_DOMAIN_EVENT_SUSPENDED_POSTCOPY or VIR_DOMAIN_EVENT_SUSPENDED_MIGRATED | |
| 19:16:31 | Zhan[m] | If we have invoked post live migration then it must mean the migration succeeded, so we are ok to move everything to dest | |
| 19:16:54 | Zhan[m] | right, that's what I'm thinking | |
| 19:17:31 | Zhan[m] | I do have some code in hand and will be starting testing while writing the spec, I can link it there | |
| 19:19:12 | sean-k-mooney | https://docs.openstack.org/nova/latest/reference/live-migration.html | |
| 19:19:12 | Zhan[m] | ok I think we are on the same page, lemme write that spec and put details there | |
| 19:19:23 | sean-k-mooney | the main concen i have is you are changing the current workflow | |
| 19:19:45 | sean-k-mooney | by adding another pahse for network activation | |
| 19:19:49 | Zhan[m] | yes, because before, for precopy, we only activate bindings when we see the migration is good | |
| 19:20:02 | sean-k-mooney | well when its complete | |
| 19:20:43 | Zhan[m] | complete and good (Ig :P) | |
| 19:20:48 | sean-k-mooney | the lbivrt part where we block should typeicly be very very short | |
| 19:21:03 | sean-k-mooney | unelss you are activly copying memry for the guest | |
| 19:21:24 | Zhan[m] | no... in our environment we saw some crz numbers like 20s+ | |
| 19:21:27 | Zhan[m] | or even a minute | |
| 19:21:35 | Zhan[m] | between PAUSED to job completed event | |
| 19:21:56 | Zhan[m] | because libvirt waits for the source QEMU process to be cleaned up before saying job is completed | |
| 19:21:57 | sean-k-mooney | Zhan[m]: https://docs.openstack.org/nova/latest/configuration/config.html#workarounds.enable_qemu_monitor_announce_self | |
| 19:22:02 | sean-k-mooney | by the way do you enable ^ | |
| 19:22:30 | sean-k-mooney | Zhan[m]: right but that shoudl only happen if qemu is copying memory | |
| 19:22:55 | sean-k-mooney | i guess the qemu instance clean up coudl take a long time if it waithign for io to complete | |
| 19:23:26 | sean-k-mooney | but libvirt shoudl not unpasue the dest instance until the source is stoped. | |
| 19:23:32 | Zhan[m] | what I found was that the kernel was spinning to clean up the memory resource | |
| 19:23:41 | Zhan[m] | which took quite a lot of time | |
| 19:24:09 | Zhan[m] | I dont think we enabled that, lemme take a look | |
| 19:24:51 | Zhan[m] | the memory resource -> the memory space that the VM uses, not by migration. if the VM is using a huge number of pages then the cleanup can take longer. | |
| 19:24:55 | sean-k-mooney | Zhan[m]: does the guest use hugepages or anonoums 4k pages out fo interest | |
| 19:25:22 | sean-k-mooney | im just wonderign why that takes so long in your case | |
| 19:25:55 | Zhan[m] | we use THP, but we figured that KSM and this bug(https://lore.kernel.org/all/20240626191830.3819324-1-yang@os.amperecomputing.com/) was causing it to not use hugepages as expected. | |
| 19:26:09 | sean-k-mooney | ah ok | |
| 19:26:29 | sean-k-mooney | THP is awsome if it works and terible otherwise | |