| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-06-10 | |||
| 17:49:57 | sean-k-mooney | spatel: you can abort it via libvirt using virsh | |
| 17:50:58 | sean-k-mooney | spatel: so ya it looke like you can use the old nova cli | |
| 17:51:11 | sean-k-mooney | nova live-migration-abort INSTANCE_ID MIGRATION_ID or nova live-migration-force-complete INSTANCE_ID MIGRATION_ID | |
| 17:51:31 | sean-k-mooney | spatel: https://docs.openstack.org/nova/pike/admin/live-migration-usage.html#addressing-migration-timeouts | |
| 17:52:34 | sean-k-mooney | spatel: nova live-migration-abort is what you are looking for. | |
| 17:52:54 | spatel | let me see | |
| 17:52:55 | sean-k-mooney | you need the instance id + the migration_id for the currently running migration | |
| 17:53:26 | sean-k-mooney | you get the migration id by doing "nova server-migration-list <server/instance uuid>" | |
| 17:54:10 | spatel | how do i find MIGRATION_ID? | |
| 17:54:25 | sean-k-mooney | spatel: ^ :) | |
| 17:54:58 | sean-k-mooney | spatel: its covered in the monitoring migration part of that doc https://docs.openstack.org/nova/pike/admin/live-migration-usage.html#monitoring-the-migration | |
| 17:55:08 | spatel | reading it.. | |
| 17:57:26 | sean-k-mooney | does that make sense you do "nova server-migration-list <server id>" | |
| 17:57:38 | sean-k-mooney | then that give you the list of migrations. | |
| 17:58:20 | sean-k-mooney | then if you have more then one you use nova server-migration-show <server uuid > <migration id> | |
| 17:58:32 | sean-k-mooney | to see which one was the most recent | |
| 17:58:42 | sean-k-mooney | and then that one is the migration you want to canchel with | |
| 17:58:53 | sean-k-mooney | nova live-migration-abort INSTANCE_ID MIGRATION_ID | |
| 18:00:02 | spatel | nova server-migration-list <server id> this is showing nothing about migration except saying | OS-EXT-STS:task_state | resize_migrating | |
| 18:00:36 | spatel | i am not seeing any ID... | |
| 18:04:02 | sean-k-mooney | spatel: am can you past the output to paste.openstack.org. feel free to remove anything sensitive | |
| 18:04:46 | sean-k-mooney | nova server-migration list shoudl vice you an output that start with Id | Source Node | Dest Node .... | |
| 18:05:01 | sean-k-mooney | the Id colum is the migration id | |
| 18:06:21 | spatel | hmm | |
| 18:09:00 | spatel | http://paste.openstack.org/show/752693/ | |
| 18:09:03 | spatel | sean-k-mooney: ^^ | |
| 18:09:27 | sean-k-mooney | spatel: openstack server show not migration list | |
| 18:10:08 | sean-k-mooney | do "nova migration-list --instance-uuid 580e624f-d1c9-496c-a591-a36c2de92e62 | |
| 18:10:12 | sean-k-mooney | " | |
| 18:11:02 | spatel | I can see lots of info npw | |
| 18:11:04 | spatel | now | |
| 18:11:19 | sean-k-mooney | ok do you have more then 1 row | |
| 18:11:33 | sean-k-mooney | if not you just need the value of the ID colum | |
| 18:12:22 | sean-k-mooney | then you can do "nova live-migration-abort 580e624f-d1c9-496c-a591-a36c2de92e62 <id> | |
| 18:12:26 | sean-k-mooney | " | |
| 18:12:46 | spatel | http://paste.openstack.org/show/752695/ | |
| 18:12:56 | spatel | sean-k-mooney: ^6 | |
| 18:13:03 | sean-k-mooney | ya so it would be | |
| 18:13:12 | spatel | is that UUID is migration ID? | |
| 18:13:13 | sean-k-mooney | "nova live-migration-abort 580e624f-d1c9-496c-a591-a36c2de92e62 384" | |
| 18:13:21 | sean-k-mooney | mp Id | |
| 18:13:25 | sean-k-mooney | *no | |
| 18:13:32 | sean-k-mooney | it the first column | |
| 18:13:42 | sean-k-mooney | the UUID column is the instance uuid | |
| 18:14:10 | spatel | to abort it i need migration ID right ( that i not able to find :( ) | |
| 18:14:39 | sean-k-mooney | spatel: so as i said its call Id in the output | |
| 18:14:57 | sean-k-mooney | so in this case it 384 and the command would be "nova live-migration-abort 580e624f-d1c9-496c-a591-a36c2de92e62 384" | |
| 18:15:24 | spatel | 384 | |
| 18:15:38 | sean-k-mooney | yep does that make sense | |
| 18:18:00 | sean-k-mooney | the migration has a UUID also but which is the firt UUID column i miss typed earlirer but the id colume "384" is what you were looking for. let me kno if that works for you :) | |
| 18:19:47 | spatel | http://paste.openstack.org/show/752698/ | |
| 18:19:49 | spatel | sean-k-mooney: ^^ | |
| 18:20:49 | sean-k-mooney | ok so it looks like we dont allow you to abort if its already copying the guest memory... that seams weird | |
| 18:21:40 | sean-k-mooney | spatel: are you doing a live migration or a cold migration | |
| 18:22:03 | spatel | My compute nodes attached to shared storage. | |
| 18:22:16 | spatel | so live migration isn't possible it has to be block migration | |
| 18:22:30 | sean-k-mooney | live migration is unrealted to shared storage | |
| 18:22:37 | spatel | my compute nodes **NOT** attached to shared storage | |
| 18:22:42 | sean-k-mooney | and if you are on shared storage you do not use block migration | |
| 18:22:52 | sean-k-mooney | ah | |
| 18:23:08 | sean-k-mooney | what command did you use to start the migration | |
| 18:23:10 | spatel | Oh! wait.. i though live migration only possible when you are using shared storage like ceph | |
| 18:23:33 | sean-k-mooney | no you can do live migration with block storage too | |
| 18:23:48 | sean-k-mooney | we just rsync the disk image to the destaion | |
| 18:23:48 | spatel | I didn't do anything, someone else just drop down in horizon and said "live migrate" | |
| 18:23:52 | sean-k-mooney | or rather libvirt does | |
| 18:24:12 | sean-k-mooney | ah ok just making sure its a live migration | |
| 18:24:20 | spatel | I meant "Migrate instance" option in GUI | |
| 18:24:27 | sean-k-mooney | ok so that is different | |
| 18:24:36 | spatel | what is the difference between | |
| 18:24:37 | sean-k-mooney | the instance is shutdown http://paste.openstack.org/show/752693/ | |
| 18:24:48 | sean-k-mooney | so its doing a cold or offline migration | |
| 18:24:52 | spatel | "Migrate instance" vs "Live migration" in GUI drop down? | |
| 18:25:16 | sean-k-mooney | migrate instance in horizon i think is a cold migration | |
| 18:25:22 | sean-k-mooney | i would have to double check. | |
| 18:25:28 | spatel | We powered down machine before migration (its safe) | |
| 18:25:57 | sean-k-mooney | well live migration are only done if the vm is running | |
| 18:26:04 | sean-k-mooney | that what makes the a live migration | |
| 18:26:26 | sean-k-mooney | so the api i asked you to try is only used for live migration but you are doing a cold migration | |
| 18:26:26 | spatel | I talled to you couple month back NOVA has bug which doesn't copy full disk and that cause instance never get full migrate... | |
| 18:26:52 | spatel | I am worried about that bug... (someone did migration without checking with me) now i want to stop this :( | |
| 18:26:54 | sean-k-mooney | i vaguly rememeber that yes | |
| 18:27:26 | sean-k-mooney | mriedem: is there any way to stop a cold migration that is in flight? | |
| 18:27:34 | sean-k-mooney | mriedem: can we revert it | |
| 18:27:50 | mriedem | nope | |
| 18:27:51 | spatel | nova start block migration and then kill it source disk without verifying full disk migration.. that was the bug | |
| 18:27:52 | sean-k-mooney | when the instance is in task_sate resize_migrating | |
| 18:28:32 | sean-k-mooney | mriedem: what about reseting the state on the vm? | |
| 18:29:49 | mriedem | sean-k-mooney: that would reset the task_state to None so you'd maybe fail somewhere along the way in compute if it tries to update the task_state and is using instance.save(expected_task_state=...) | |
| 18:30:04 | mriedem | but clearly that's not any kind of supported way to interrupt a cold migration | |
| 18:30:22 | sean-k-mooney | mriedem: yes i know | |
| 18:30:54 | spatel | cold migration = when machine is shutdown it does cold migration ? | |
| 18:31:00 | spatel | is that true? | |
| 18:31:17 | sean-k-mooney | spatel: by machine you mean the vm? | |
| 18:31:21 | mriedem | NTT (i think it was NTT) had a spec up for several releases to add support to abort a cold migration, with the main reason being "we shove logs into a 100GB+ local disk and migrating those VMs takes a long time so we'd maybe like to abort the cold migration" | |
| 18:31:27 | spatel | yes machine = VM | |
| 18:31:52 | sean-k-mooney | mriedem: ya i vaguly rememebr that | |
| 18:32:16 | sean-k-mooney | spatel: do you have nova configured to auto confim migrations/resize | |
| 18:32:37 | sean-k-mooney | spatel: if not then nova will not delete the souce disk unless you confim the resize. | |
| 18:33:11 | spatel | NO i don't have auto confim | |