Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-11-05
15:45:58 artom mriedem, yep - as I said in my first comment, I can't speak to the raison d'etre of the patch
15:46:00 artom Just the mechanics (assuming we need it in the first place)
15:46:00 mriedem so i think tl;dr on live migration rollback since rocky we leave the source host port bindings inactive and you likely need to do something to resolve that, i'm not sure that the heal instance periodic fixes it
15:46:14 sean-k-mooney yep
15:46:50 sean-k-mooney well we have two patach to resovle to related edgecases already
15:47:05 sean-k-mooney artom: what bug are you looking at
15:47:17 artom sean-k-mooney, https://bugs.launchpad.net/nova/+bug/1788014
15:47:17 openstack Launchpad bug 1788014 in OpenStack Compute (nova) "when live migration fails due to a internal error rollback is not handeled correctly." [Medium,In progress] - Assigned to Matt Riedemann (mriedem)
15:47:36 watersj mriedem, vm status is pre-migrating
15:47:40 mriedem https://review.opendev.org/#/c/603844/ landed in stein but i'm not sure that heals the port binding on the source host either
15:48:15 mriedem watersj: i wanted to know the migration record status, not the server vm_state
15:48:24 mriedem or maybe that's what you meant
15:48:39 sean-k-mooney so that is a duplicate of https://bugs.launchpad.net/nova/+bug/1822884
15:48:39 openstack Launchpad bug 1822884 in OpenStack Compute (nova) "live migration fails due to port binding duplicate key entry in post_live_migrate" [Medium,In progress] - Assigned to sean mooney (sean-k-mooney)
15:48:42 watersj mriedem, output from server-migration-list is empty
15:49:03 mriedem i guess because the api doesn't consider pre-migrating as in-progress...i think i found that out recently
15:49:33 artom mriedem, wait, that rings a bell
15:49:35 mriedem anyway, https://docs.openstack.org/python-novaclient/latest/cli/nova.html#nova-live-migration-abort won't do anything for a pre-migrating migration, only running/queued/preparing
15:49:51 mriedem artom: yeah i noticed it when reviewing your numa live migration series
15:50:14 sean-k-mooney there was a seperate issue with evacuate and pre-migrationg that artom might be thinking of
15:50:16 artom mriedem, oh right - https://review.opendev.org/#/c/684409/
15:50:21 watersj this is stein that I am running
15:50:23 mriedem because in the long ago live migration records didn't have pre-migrating since that was only done during the RT claim
15:50:42 artom But my patch only changes the non-API version of that query
15:51:24 artom Which is your comment on https://review.opendev.org/#/c/684409/1/nova/db/sqlalchemy/api.py@4431
15:52:14 mriedem right, so the api only shows in-progress live migrations when they have status "'queued', 'preparing', 'running'"
15:52:18 mriedem which watersj just confirmed
15:52:37 mriedem even though arguably 'accepted' and 'pre-migrating' could be considered in-progress
15:52:44 mriedem or as in-progress as queued
15:53:07 mriedem watersj: did you trace down where the migration request failed and got stuck?
15:53:43 artom mriedem, I think that has to do with when they get put in the queue
15:53:50 artom IOW, you can't abort it if it's not in the queue yet
15:53:58 mriedem watersj: if the guest is still on the source host then nothing has really changed. you would likely have to manually change the migration record status to 'error' and reset the instance vm_state to 'active' and task_state to None
15:54:07 watersj no, looking at nova-compute logs did not give me much info. Currently vm is down and I trying to get it back on
15:54:35 watersj i did change its state ti error
15:54:39 watersj and then active
15:54:42 mriedem watersj: ok, so you'll have to set the migration record status to 'error' and hard reboot the guest i think
15:54:48 sean-k-mooney artom: mriedem by the way i though when we got to post-copy stage we cant rollback anymore
15:54:53 mriedem watersj: the instances table record or the migrations table record?
15:54:57 mriedem those are different records
15:55:06 watersj 1sec
15:56:05 artom sean-k-mooney, yeah, but... is that relevant here?
15:56:14 watersj i ran nova reset-state <name>
15:56:17 artom Also, which "here"? watersj's issue, or https://review.opendev.org/#/c/594139/1 ?
15:56:20 watersj i ran nova reset-state --active <name>
15:56:33 mriedem watersj: that just updates the server's vm/task states, it doesn't do anything with a stopped guest
15:56:35 sean-k-mooney artom: for mriedem commit "If we've reached the point of post-copy during live migration on the source host and activated the destination host port bindings, but then something fails and we have to rollback,"
15:56:46 watersj the guest is not running
15:56:51 watersj that is confirmed
15:57:02 mriedem right, that's why i said you'll have to hard reboot it
15:57:08 mriedem to get it back to active/running for real
15:57:12 mriedem bye!
15:57:22 artom You're welcome?
15:57:26 artom sean-k-mooney, huh, good spot
15:57:40 artom Once we're in post-copy, we can't rollback, as the VM is now running on the dest
15:57:52 sean-k-mooney yes
15:57:53 artom https://wiki.qemu.org/Features/PostCopyLiveMigration#Summary
15:58:06 sean-k-mooney and libvirt cant rollback at that point
15:58:53 artom Although I think that's just the commit message
15:59:21 artom Your bug (https://bugs.launchpad.net/nova/+bug/1788014) doesn't mention post-copy
15:59:21 openstack Launchpad bug 1788014 in OpenStack Compute (nova) "when live migration fails due to a internal error rollback is not handeled correctly." [Medium,In progress] - Assigned to Matt Riedemann (mriedem)
16:00:01 artom And the new code is being called from the existing _rollback_live_migration() method, so it's not like we're attempting to rollback a post-copy migration
16:00:03 mriedem i likely mentioned post-copy in the commit message b/c the lifecycle handler callback thing will activate the dest host port binding if the migration switches to post-copy
16:00:46 mriedem this code https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1252
16:01:10 watersj Instance not resizing, skipping migration, and looks to be staying there
16:01:21 watersj from nova-compute logs
16:01:29 mriedem but it doesn't have to be post-copy to switch bindings https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1203
16:01:49 mriedem watersj: yes, i realize b/c the resource tracker ignores live migrations in stein
16:02:07 mriedem watersj: like i said, you'll have to find and manually change the migrations table record to 'error' status and then hard reboot the server
16:02:19 watersj ah, k
16:02:30 mriedem the former takes it out of the RT loop which is why you're seeing "Instance not resizing, skipping migration" and the latter is to get the guest running agian
16:02:31 mriedem *again
16:02:48 watersj ty, give it try
16:03:08 mriedem something like this should find the migration:
16:03:08 artom mriedem, so EVENT_LIFECYCLE_POSTCOPY_STARTED and EVENT_LIFECYCLE_MIGRATION_COMPLETED mean the same thing in that context: the VM is now irreversibly running on the dest
16:03:20 artom mriedem, so it makes sense to update the port bindings at that point
16:03:39 mriedem select * from nova_cell1.migrations where instance_uuid=<server id> and deleted=0 and status='pre-migrating';
16:03:44 sean-k-mooney artom: not nessisarally
16:03:51 sean-k-mooney EVENT_LIFECYCLE_MIGRATION_COMPLETED i think can be sent for failure
16:04:04 mriedem yup
16:04:07 mriedem which is another patch of mine
16:04:26 mriedem https://review.opendev.org/#/c/594527/
16:05:02 openstackgerrit Matt Riedemann proposed openstack/nova master: libvirt: check job status for VIR_DOMAIN_EVENT_SUSPENDED_MIGRATED event https://review.opendev.org/594527
16:05:09 artom Aha, TIL
16:05:24 artom (Actually, I think I already checked this when faking it for my NUMA LM func tests, but forgot)
16:06:00 artom (Which, btw, are waiting for review - but given the last thing was Stephen's -1 which I addressed, maybe wait for him to get pack from PTG)
16:06:20 artom We've long passed any sense of urgency, at this point
16:06:58 sean-k-mooney oh actully i think i know how to repoduce the failure by the way
16:07:32 sean-k-mooney i think i found this issue while trying to live migrate between kerenl ovs and ovs dpdk were the virtio feature flags change
16:07:43 sean-k-mooney due to setting the mtu in one and not in the other
16:08:42 sean-k-mooney strangely enough i set up a dpdk env to work on that and and another dpdk thing on friday
16:10:14 sean-k-mooney artom: this was part of the orginal error http://paste.openstack.org/show/728539/
16:11:56 artom sean-k-mooney, ah, so you hit this because the source sent some XML the dest couldn't handle, and QEMU died
16:12:06 sean-k-mooney yep
16:12:11 sean-k-mooney libvirt was fine with it
16:12:28 sean-k-mooney well you can see the error
16:12:38 artom Right, the XML itself was "fine"
16:12:49 artom But the generated qemu command line exploded it
16:12:55 sean-k-mooney ya but not for the migration
16:13:11 sean-k-mooney qemu also acccpeted teh commandline

Earlier   Later