| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-06-10 | |||
| 18:00:02 | spatel | nova server-migration-list <server id> this is showing nothing about migration except saying | OS-EXT-STS:task_state | resize_migrating | |
| 18:00:36 | spatel | i am not seeing any ID... | |
| 18:04:02 | sean-k-mooney | spatel: am can you past the output to paste.openstack.org. feel free to remove anything sensitive | |
| 18:04:46 | sean-k-mooney | nova server-migration list shoudl vice you an output that start with Id | Source Node | Dest Node .... | |
| 18:05:01 | sean-k-mooney | the Id colum is the migration id | |
| 18:06:21 | spatel | hmm | |
| 18:09:00 | spatel | http://paste.openstack.org/show/752693/ | |
| 18:09:03 | spatel | sean-k-mooney: ^^ | |
| 18:09:27 | sean-k-mooney | spatel: openstack server show not migration list | |
| 18:10:08 | sean-k-mooney | do "nova migration-list --instance-uuid 580e624f-d1c9-496c-a591-a36c2de92e62 | |
| 18:10:12 | sean-k-mooney | " | |
| 18:11:02 | spatel | I can see lots of info npw | |
| 18:11:04 | spatel | now | |
| 18:11:19 | sean-k-mooney | ok do you have more then 1 row | |
| 18:11:33 | sean-k-mooney | if not you just need the value of the ID colum | |
| 18:12:22 | sean-k-mooney | then you can do "nova live-migration-abort 580e624f-d1c9-496c-a591-a36c2de92e62 <id> | |
| 18:12:26 | sean-k-mooney | " | |
| 18:12:46 | spatel | http://paste.openstack.org/show/752695/ | |
| 18:12:56 | spatel | sean-k-mooney: ^6 | |
| 18:13:03 | sean-k-mooney | ya so it would be | |
| 18:13:12 | spatel | is that UUID is migration ID? | |
| 18:13:13 | sean-k-mooney | "nova live-migration-abort 580e624f-d1c9-496c-a591-a36c2de92e62 384" | |
| 18:13:21 | sean-k-mooney | mp Id | |
| 18:13:25 | sean-k-mooney | *no | |
| 18:13:32 | sean-k-mooney | it the first column | |
| 18:13:42 | sean-k-mooney | the UUID column is the instance uuid | |
| 18:14:10 | spatel | to abort it i need migration ID right ( that i not able to find :( ) | |
| 18:14:39 | sean-k-mooney | spatel: so as i said its call Id in the output | |
| 18:14:57 | sean-k-mooney | so in this case it 384 and the command would be "nova live-migration-abort 580e624f-d1c9-496c-a591-a36c2de92e62 384" | |
| 18:15:24 | spatel | 384 | |
| 18:15:38 | sean-k-mooney | yep does that make sense | |
| 18:18:00 | sean-k-mooney | the migration has a UUID also but which is the firt UUID column i miss typed earlirer but the id colume "384" is what you were looking for. let me kno if that works for you :) | |
| 18:19:47 | spatel | http://paste.openstack.org/show/752698/ | |
| 18:19:49 | spatel | sean-k-mooney: ^^ | |
| 18:20:49 | sean-k-mooney | ok so it looks like we dont allow you to abort if its already copying the guest memory... that seams weird | |
| 18:21:40 | sean-k-mooney | spatel: are you doing a live migration or a cold migration | |
| 18:22:03 | spatel | My compute nodes attached to shared storage. | |
| 18:22:16 | spatel | so live migration isn't possible it has to be block migration | |
| 18:22:30 | sean-k-mooney | live migration is unrealted to shared storage | |
| 18:22:37 | spatel | my compute nodes **NOT** attached to shared storage | |
| 18:22:42 | sean-k-mooney | and if you are on shared storage you do not use block migration | |
| 18:22:52 | sean-k-mooney | ah | |
| 18:23:08 | sean-k-mooney | what command did you use to start the migration | |
| 18:23:10 | spatel | Oh! wait.. i though live migration only possible when you are using shared storage like ceph | |
| 18:23:33 | sean-k-mooney | no you can do live migration with block storage too | |
| 18:23:48 | spatel | I didn't do anything, someone else just drop down in horizon and said "live migrate" | |
| 18:23:48 | sean-k-mooney | we just rsync the disk image to the destaion | |
| 18:23:52 | sean-k-mooney | or rather libvirt does | |
| 18:24:12 | sean-k-mooney | ah ok just making sure its a live migration | |
| 18:24:20 | spatel | I meant "Migrate instance" option in GUI | |
| 18:24:27 | sean-k-mooney | ok so that is different | |
| 18:24:36 | spatel | what is the difference between | |
| 18:24:37 | sean-k-mooney | the instance is shutdown http://paste.openstack.org/show/752693/ | |
| 18:24:48 | sean-k-mooney | so its doing a cold or offline migration | |
| 18:24:52 | spatel | "Migrate instance" vs "Live migration" in GUI drop down? | |
| 18:25:16 | sean-k-mooney | migrate instance in horizon i think is a cold migration | |
| 18:25:22 | sean-k-mooney | i would have to double check. | |
| 18:25:28 | spatel | We powered down machine before migration (its safe) | |
| 18:25:57 | sean-k-mooney | well live migration are only done if the vm is running | |
| 18:26:04 | sean-k-mooney | that what makes the a live migration | |
| 18:26:26 | spatel | I talled to you couple month back NOVA has bug which doesn't copy full disk and that cause instance never get full migrate... | |
| 18:26:26 | sean-k-mooney | so the api i asked you to try is only used for live migration but you are doing a cold migration | |
| 18:26:52 | spatel | I am worried about that bug... (someone did migration without checking with me) now i want to stop this :( | |
| 18:26:54 | sean-k-mooney | i vaguly rememeber that yes | |
| 18:27:26 | sean-k-mooney | mriedem: is there any way to stop a cold migration that is in flight? | |
| 18:27:34 | sean-k-mooney | mriedem: can we revert it | |
| 18:27:50 | mriedem | nope | |
| 18:27:51 | spatel | nova start block migration and then kill it source disk without verifying full disk migration.. that was the bug | |
| 18:27:52 | sean-k-mooney | when the instance is in task_sate resize_migrating | |
| 18:28:32 | sean-k-mooney | mriedem: what about reseting the state on the vm? | |
| 18:29:49 | mriedem | sean-k-mooney: that would reset the task_state to None so you'd maybe fail somewhere along the way in compute if it tries to update the task_state and is using instance.save(expected_task_state=...) | |
| 18:30:04 | mriedem | but clearly that's not any kind of supported way to interrupt a cold migration | |
| 18:30:22 | sean-k-mooney | mriedem: yes i know | |
| 18:30:54 | spatel | cold migration = when machine is shutdown it does cold migration ? | |
| 18:31:00 | spatel | is that true? | |
| 18:31:17 | sean-k-mooney | spatel: by machine you mean the vm? | |
| 18:31:21 | mriedem | NTT (i think it was NTT) had a spec up for several releases to add support to abort a cold migration, with the main reason being "we shove logs into a 100GB+ local disk and migrating those VMs takes a long time so we'd maybe like to abort the cold migration" | |
| 18:31:27 | spatel | yes machine = VM | |
| 18:31:52 | sean-k-mooney | mriedem: ya i vaguly rememebr that | |
| 18:32:16 | sean-k-mooney | spatel: do you have nova configured to auto confim migrations/resize | |
| 18:32:37 | sean-k-mooney | spatel: if not then nova will not delete the souce disk unless you confim the resize. | |
| 18:33:11 | spatel | NO i don't have auto confim | |
| 18:33:13 | sean-k-mooney | at which point you can either boot up the vm and check its ok or you can do "nova resize-revert <server uuid>" | |
| 18:33:53 | spatel | oh! can i boot instance without confirm ? | |
| 18:34:20 | sean-k-mooney | i think so. i have not tried this in quite a while. | |
| 18:35:48 | sean-k-mooney | maybe we cant but i know that has also been asked for in the past | |
| 18:36:28 | sean-k-mooney | spatel: in anycase when the migration eventually complete you can revert it which will be quick as teh data will not have been delete on the souce node. | |
| 18:36:47 | openstackgerrit | Adam Spiers proposed openstack/nova master: Add extra spec parameter and image property for memory encryption https://review.opendev.org/664420 | |
| 18:37:02 | aspiers | efried: is this roughly along the right lines? ^^^ | |
| 18:38:01 | spatel | i will revert it without confirm it | |
| 18:38:03 | efried | aspiers: I'm going to have to look a bit later, I'm afraid. | |
| 18:38:27 | aspiers | efried: np. Don't need a full review yet, just confirmation I'm barking up the right tree | |
| 18:55:45 | spatel | sean-k-mooney: revert works! | |
| 18:55:53 | spatel | what a pain :) | |
| 18:56:20 | spatel | sean-k-mooney: thank you again for saving my a... :) | |
| 19:14:01 | openstackgerrit | Matt Riedemann proposed openstack/nova-specs master: Spec to pre-filter disabled computes with placement https://review.opendev.org/657884 | |
| 19:53:07 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: WIP: Revert resize: wait for events according to hybrid plug https://review.opendev.org/644881 | |
| 19:53:08 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Remove finish_revert_migration TypeError try/except https://review.opendev.org/664013 | |
| 19:53:09 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: WIP: Include direct-physical in compute manager events check https://review.opendev.org/664431 | |
| 19:53:48 | artom | sean-k-mooney, mriedem, ^^ did a thing. Still lots of cleanup to do, but maybe check the compute manager/libvirt driver parts to see if the general approach is palatable? | |