Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-11-05
15:30:17 mriedem tl;dr the neutronv2/api code is a gd mess when it comes to migrations
15:30:27 mriedem there are like at least 6 methods involved in mucking with port bindings
15:37:32 artom mriedem, I guess it comes down to "are we backporting this?"
15:37:41 artom If not, we can invest in a more thorough refactor/rethink
15:38:21 mriedem it probably depends on if sean-k-mooney can recreate the failure in a real env and confirm the bug and the fix
15:38:28 mriedem on...rocky
15:38:45 mriedem like i said in the change over a year ago, "Waiting for Sean Mooney to test this out in his environment since he was able to generate the issue and report the bug, so I want to know if this fixes it. Also, I want Miguel to take a look at this given my assumptions about how I thought neutron would handle the source host binding when the dest binding is deleted."
15:39:46 mriedem even if we did backport, i could do either of my 1 or 2 options and make it backportable
15:40:11 mriedem and add a wrapper/convenience method which doesn't get backported that calls both the activate and teardown methods under the covers
15:41:07 sean-k-mooney am what do i need to repoduce? reads back
15:41:17 mriedem https://review.opendev.org/#/c/594139/
15:42:14 artom I kinda wonder what ovs-dpdk and/or hugepages have to do with it
15:42:43 sean-k-mooney mriedem: that sounds like https://review.opendev.org/#/c/653506/
15:42:45 artom Seems like we could repro it in func tests with an instance with some ports
15:43:16 sean-k-mooney although i need to reload the context for both back into my head
15:43:51 mriedem artom: the neutron fixture would have to start tracking port bindings and do it correctly, which i'm not entirely confident in how that api works - which is why we have this bug in the first place
15:44:04 mriedem i.e. deleting an inactive port binding does not automatically activate the other port binding
15:44:22 mriedem but activating an inactive port binding automatically deactivates the previously active port binding
15:44:26 artom mriedem, ah, right
15:44:47 mriedem from my commit message, "I had assumed neutron would automatically activate the old remaining port binding when the dest host port binding was deleted, like how neutron automatically deactivates the source host binding when the dest host binding is activated, but apparently that is not the case, so we have to do it ourselves on rollback."
15:45:58 artom mriedem, yep - as I said in my first comment, I can't speak to the raison d'etre of the patch
15:46:00 mriedem so i think tl;dr on live migration rollback since rocky we leave the source host port bindings inactive and you likely need to do something to resolve that, i'm not sure that the heal instance periodic fixes it
15:46:00 artom Just the mechanics (assuming we need it in the first place)
15:46:14 sean-k-mooney yep
15:46:50 sean-k-mooney well we have two patach to resovle to related edgecases already
15:47:05 sean-k-mooney artom: what bug are you looking at
15:47:17 openstack Launchpad bug 1788014 in OpenStack Compute (nova) "when live migration fails due to a internal error rollback is not handeled correctly." [Medium,In progress] - Assigned to Matt Riedemann (mriedem)
15:47:17 artom sean-k-mooney, https://bugs.launchpad.net/nova/+bug/1788014
15:47:36 watersj mriedem, vm status is pre-migrating
15:47:40 mriedem https://review.opendev.org/#/c/603844/ landed in stein but i'm not sure that heals the port binding on the source host either
15:48:15 mriedem watersj: i wanted to know the migration record status, not the server vm_state
15:48:24 mriedem or maybe that's what you meant
15:48:39 openstack Launchpad bug 1822884 in OpenStack Compute (nova) "live migration fails due to port binding duplicate key entry in post_live_migrate" [Medium,In progress] - Assigned to sean mooney (sean-k-mooney)
15:48:39 sean-k-mooney so that is a duplicate of https://bugs.launchpad.net/nova/+bug/1822884
15:48:42 watersj mriedem, output from server-migration-list is empty
15:49:03 mriedem i guess because the api doesn't consider pre-migrating as in-progress...i think i found that out recently
15:49:33 artom mriedem, wait, that rings a bell
15:49:35 mriedem anyway, https://docs.openstack.org/python-novaclient/latest/cli/nova.html#nova-live-migration-abort won't do anything for a pre-migrating migration, only running/queued/preparing
15:49:51 mriedem artom: yeah i noticed it when reviewing your numa live migration series
15:50:14 sean-k-mooney there was a seperate issue with evacuate and pre-migrationg that artom might be thinking of
15:50:16 artom mriedem, oh right - https://review.opendev.org/#/c/684409/
15:50:21 watersj this is stein that I am running
15:50:23 mriedem because in the long ago live migration records didn't have pre-migrating since that was only done during the RT claim
15:50:42 artom But my patch only changes the non-API version of that query
15:51:24 artom Which is your comment on https://review.opendev.org/#/c/684409/1/nova/db/sqlalchemy/api.py@4431
15:52:14 mriedem right, so the api only shows in-progress live migrations when they have status "'queued', 'preparing', 'running'"
15:52:18 mriedem which watersj just confirmed
15:52:37 mriedem even though arguably 'accepted' and 'pre-migrating' could be considered in-progress
15:52:44 mriedem or as in-progress as queued
15:53:07 mriedem watersj: did you trace down where the migration request failed and got stuck?
15:53:43 artom mriedem, I think that has to do with when they get put in the queue
15:53:50 artom IOW, you can't abort it if it's not in the queue yet
15:53:58 mriedem watersj: if the guest is still on the source host then nothing has really changed. you would likely have to manually change the migration record status to 'error' and reset the instance vm_state to 'active' and task_state to None
15:54:07 watersj no, looking at nova-compute logs did not give me much info. Currently vm is down and I trying to get it back on
15:54:35 watersj i did change its state ti error
15:54:39 watersj and then active
15:54:42 mriedem watersj: ok, so you'll have to set the migration record status to 'error' and hard reboot the guest i think
15:54:48 sean-k-mooney artom: mriedem by the way i though when we got to post-copy stage we cant rollback anymore
15:54:53 mriedem watersj: the instances table record or the migrations table record?
15:54:57 mriedem those are different records
15:55:06 watersj 1sec
15:56:05 artom sean-k-mooney, yeah, but... is that relevant here?
15:56:14 watersj i ran nova reset-state <name>
15:56:17 artom Also, which "here"? watersj's issue, or https://review.opendev.org/#/c/594139/1 ?
15:56:20 watersj i ran nova reset-state --active <name>
15:56:33 mriedem watersj: that just updates the server's vm/task states, it doesn't do anything with a stopped guest
15:56:35 sean-k-mooney artom: for mriedem commit "If we've reached the point of post-copy during live migration on the source host and activated the destination host port bindings, but then something fails and we have to rollback,"
15:56:46 watersj the guest is not running
15:56:51 watersj that is confirmed
15:57:02 mriedem right, that's why i said you'll have to hard reboot it
15:57:08 mriedem to get it back to active/running for real
15:57:12 mriedem bye!
15:57:22 artom You're welcome?
15:57:26 artom sean-k-mooney, huh, good spot
15:57:40 artom Once we're in post-copy, we can't rollback, as the VM is now running on the dest
15:57:52 sean-k-mooney yes
15:57:53 artom https://wiki.qemu.org/Features/PostCopyLiveMigration#Summary
15:58:06 sean-k-mooney and libvirt cant rollback at that point
15:58:53 artom Although I think that's just the commit message
15:59:21 openstack Launchpad bug 1788014 in OpenStack Compute (nova) "when live migration fails due to a internal error rollback is not handeled correctly." [Medium,In progress] - Assigned to Matt Riedemann (mriedem)
15:59:21 artom Your bug (https://bugs.launchpad.net/nova/+bug/1788014) doesn't mention post-copy
16:00:01 artom And the new code is being called from the existing _rollback_live_migration() method, so it's not like we're attempting to rollback a post-copy migration
16:00:03 mriedem i likely mentioned post-copy in the commit message b/c the lifecycle handler callback thing will activate the dest host port binding if the migration switches to post-copy
16:00:46 mriedem this code https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1252
16:01:10 watersj Instance not resizing, skipping migration, and looks to be staying there
16:01:21 watersj from nova-compute logs
16:01:29 mriedem but it doesn't have to be post-copy to switch bindings https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1203
16:01:49 mriedem watersj: yes, i realize b/c the resource tracker ignores live migrations in stein
16:02:07 mriedem watersj: like i said, you'll have to find and manually change the migrations table record to 'error' status and then hard reboot the server
16:02:19 watersj ah, k
16:02:30 mriedem the former takes it out of the RT loop which is why you're seeing "Instance not resizing, skipping migration" and the latter is to get the guest running agian
16:02:31 mriedem *again
16:02:48 watersj ty, give it try
16:03:08 artom mriedem, so EVENT_LIFECYCLE_POSTCOPY_STARTED and EVENT_LIFECYCLE_MIGRATION_COMPLETED mean the same thing in that context: the VM is now irreversibly running on the dest
16:03:08 mriedem something like this should find the migration:
16:03:20 artom mriedem, so it makes sense to update the port bindings at that point
16:03:39 mriedem select * from nova_cell1.migrations where instance_uuid=<server id> and deleted=0 and status='pre-migrating';
16:03:44 sean-k-mooney artom: not nessisarally
16:03:51 sean-k-mooney EVENT_LIFECYCLE_MIGRATION_COMPLETED i think can be sent for failure
16:04:04 mriedem yup

Earlier   Later