| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-06-10 | |||
| 18:13:13 | sean-k-mooney | "nova live-migration-abort 580e624f-d1c9-496c-a591-a36c2de92e62 384" | |
| 18:13:21 | sean-k-mooney | mp Id | |
| 18:13:25 | sean-k-mooney | *no | |
| 18:13:32 | sean-k-mooney | it the first column | |
| 18:13:42 | sean-k-mooney | the UUID column is the instance uuid | |
| 18:14:10 | spatel | to abort it i need migration ID right ( that i not able to find :( ) | |
| 18:14:39 | sean-k-mooney | spatel: so as i said its call Id in the output | |
| 18:14:57 | sean-k-mooney | so in this case it 384 and the command would be "nova live-migration-abort 580e624f-d1c9-496c-a591-a36c2de92e62 384" | |
| 18:15:24 | spatel | 384 | |
| 18:15:38 | sean-k-mooney | yep does that make sense | |
| 18:18:00 | sean-k-mooney | the migration has a UUID also but which is the firt UUID column i miss typed earlirer but the id colume "384" is what you were looking for. let me kno if that works for you :) | |
| 18:19:47 | spatel | http://paste.openstack.org/show/752698/ | |
| 18:19:49 | spatel | sean-k-mooney: ^^ | |
| 18:20:49 | sean-k-mooney | ok so it looks like we dont allow you to abort if its already copying the guest memory... that seams weird | |
| 18:21:40 | sean-k-mooney | spatel: are you doing a live migration or a cold migration | |
| 18:22:03 | spatel | My compute nodes attached to shared storage. | |
| 18:22:16 | spatel | so live migration isn't possible it has to be block migration | |
| 18:22:30 | sean-k-mooney | live migration is unrealted to shared storage | |
| 18:22:37 | spatel | my compute nodes **NOT** attached to shared storage | |
| 18:22:42 | sean-k-mooney | and if you are on shared storage you do not use block migration | |
| 18:22:52 | sean-k-mooney | ah | |
| 18:23:08 | sean-k-mooney | what command did you use to start the migration | |
| 18:23:10 | spatel | Oh! wait.. i though live migration only possible when you are using shared storage like ceph | |
| 18:23:33 | sean-k-mooney | no you can do live migration with block storage too | |
| 18:23:48 | sean-k-mooney | we just rsync the disk image to the destaion | |
| 18:23:48 | spatel | I didn't do anything, someone else just drop down in horizon and said "live migrate" | |
| 18:23:52 | sean-k-mooney | or rather libvirt does | |
| 18:24:12 | sean-k-mooney | ah ok just making sure its a live migration | |
| 18:24:20 | spatel | I meant "Migrate instance" option in GUI | |
| 18:24:27 | sean-k-mooney | ok so that is different | |
| 18:24:36 | spatel | what is the difference between | |
| 18:24:37 | sean-k-mooney | the instance is shutdown http://paste.openstack.org/show/752693/ | |
| 18:24:48 | sean-k-mooney | so its doing a cold or offline migration | |
| 18:24:52 | spatel | "Migrate instance" vs "Live migration" in GUI drop down? | |
| 18:25:16 | sean-k-mooney | migrate instance in horizon i think is a cold migration | |
| 18:25:22 | sean-k-mooney | i would have to double check. | |
| 18:25:28 | spatel | We powered down machine before migration (its safe) | |
| 18:25:57 | sean-k-mooney | well live migration are only done if the vm is running | |
| 18:26:04 | sean-k-mooney | that what makes the a live migration | |
| 18:26:26 | sean-k-mooney | so the api i asked you to try is only used for live migration but you are doing a cold migration | |
| 18:26:26 | spatel | I talled to you couple month back NOVA has bug which doesn't copy full disk and that cause instance never get full migrate... | |
| 18:26:52 | spatel | I am worried about that bug... (someone did migration without checking with me) now i want to stop this :( | |
| 18:26:54 | sean-k-mooney | i vaguly rememeber that yes | |
| 18:27:26 | sean-k-mooney | mriedem: is there any way to stop a cold migration that is in flight? | |
| 18:27:34 | sean-k-mooney | mriedem: can we revert it | |
| 18:27:50 | mriedem | nope | |
| 18:27:51 | spatel | nova start block migration and then kill it source disk without verifying full disk migration.. that was the bug | |
| 18:27:52 | sean-k-mooney | when the instance is in task_sate resize_migrating | |
| 18:28:32 | sean-k-mooney | mriedem: what about reseting the state on the vm? | |
| 18:29:49 | mriedem | sean-k-mooney: that would reset the task_state to None so you'd maybe fail somewhere along the way in compute if it tries to update the task_state and is using instance.save(expected_task_state=...) | |
| 18:30:04 | mriedem | but clearly that's not any kind of supported way to interrupt a cold migration | |
| 18:30:22 | sean-k-mooney | mriedem: yes i know | |
| 18:30:54 | spatel | cold migration = when machine is shutdown it does cold migration ? | |
| 18:31:00 | spatel | is that true? | |
| 18:31:17 | sean-k-mooney | spatel: by machine you mean the vm? | |
| 18:31:21 | mriedem | NTT (i think it was NTT) had a spec up for several releases to add support to abort a cold migration, with the main reason being "we shove logs into a 100GB+ local disk and migrating those VMs takes a long time so we'd maybe like to abort the cold migration" | |
| 18:31:27 | spatel | yes machine = VM | |
| 18:31:52 | sean-k-mooney | mriedem: ya i vaguly rememebr that | |
| 18:32:16 | sean-k-mooney | spatel: do you have nova configured to auto confim migrations/resize | |
| 18:32:37 | sean-k-mooney | spatel: if not then nova will not delete the souce disk unless you confim the resize. | |
| 18:33:11 | spatel | NO i don't have auto confim | |
| 18:33:13 | sean-k-mooney | at which point you can either boot up the vm and check its ok or you can do "nova resize-revert <server uuid>" | |
| 18:33:53 | spatel | oh! can i boot instance without confirm ? | |
| 18:34:20 | sean-k-mooney | i think so. i have not tried this in quite a while. | |
| 18:35:48 | sean-k-mooney | maybe we cant but i know that has also been asked for in the past | |
| 18:36:28 | sean-k-mooney | spatel: in anycase when the migration eventually complete you can revert it which will be quick as teh data will not have been delete on the souce node. | |
| 18:36:47 | openstackgerrit | Adam Spiers proposed openstack/nova master: Add extra spec parameter and image property for memory encryption https://review.opendev.org/664420 | |
| 18:37:02 | aspiers | efried: is this roughly along the right lines? ^^^ | |
| 18:38:01 | spatel | i will revert it without confirm it | |
| 18:38:03 | efried | aspiers: I'm going to have to look a bit later, I'm afraid. | |
| 18:38:27 | aspiers | efried: np. Don't need a full review yet, just confirmation I'm barking up the right tree | |
| 18:55:45 | spatel | sean-k-mooney: revert works! | |
| 18:55:53 | spatel | what a pain :) | |
| 18:56:20 | spatel | sean-k-mooney: thank you again for saving my a... :) | |
| 19:14:01 | openstackgerrit | Matt Riedemann proposed openstack/nova-specs master: Spec to pre-filter disabled computes with placement https://review.opendev.org/657884 | |
| 19:53:07 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: WIP: Revert resize: wait for events according to hybrid plug https://review.opendev.org/644881 | |
| 19:53:08 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Remove finish_revert_migration TypeError try/except https://review.opendev.org/664013 | |
| 19:53:09 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: WIP: Include direct-physical in compute manager events check https://review.opendev.org/664431 | |
| 19:53:48 | artom | sean-k-mooney, mriedem, ^^ did a thing. Still lots of cleanup to do, but maybe check the compute manager/libvirt driver parts to see if the general approach is palatable? | |
| 19:59:44 | sean-k-mooney | that likely wont be backportable. due to the chagne in the signiture of finish_revert_migration or at least its more invasive then it needs to be | |
| 19:59:47 | sean-k-mooney | still reading it | |
| 20:00:48 | artom | finish_revert_migration was changed with the previous approach as well (was adding vifs_already_plugged) | |
| 20:01:19 | sean-k-mooney | yes but we can actully make this work without adding any new parmater to finish_revert_migration | |
| 20:01:57 | sean-k-mooney | if we take the approch i was suggesting. we actully did not need to pass vif_already_plugged | |
| 20:02:16 | artom | Why is adding a parameter bad though? That's not RPC, don't think it'll affect backportability... | |
| 20:02:40 | artom | It's compute manager talking to its virt driver, both are updated at the same time | |
| 20:02:40 | sean-k-mooney | its chaning the virt dirver api right | |
| 20:02:49 | artom | Yeah | |
| 20:02:55 | sean-k-mooney | so it would break out of tree drivers without the try catch | |
| 20:03:10 | artom | Well, yeah | |
| 20:03:13 | sean-k-mooney | * try excpet | |
| 20:04:43 | artom | I don't see a way around passing the virt driver *something* | |
| 20:05:27 | sean-k-mooney | we have all the info we need in the network_info object since that is a list of vifs | |
| 20:05:31 | sean-k-mooney | so we can calualte if we need to wait in the driver or manager form that without passing anything | |
| 20:05:55 | sean-k-mooney | which is part of the reason i suggested adding the function in the vif model class | |
| 20:07:17 | artom | The problem is that finish_revert_migration isn't always called for revert resize... | |
| 20:07:43 | artom | So even if network_info has all it needs in terms of event "types" | |
| 20:07:55 | artom | We still need a way to tell the virt driver when to actually use that information | |
| 20:08:42 | sean-k-mooney | im not following | |
| 20:09:19 | artom | All of this stuff, it's only applicable in the revert resize case, right? | |