| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-11-05 | |||
| 15:53:43 | artom | mriedem, I think that has to do with when they get put in the queue | |
| 15:53:50 | artom | IOW, you can't abort it if it's not in the queue yet | |
| 15:53:58 | mriedem | watersj: if the guest is still on the source host then nothing has really changed. you would likely have to manually change the migration record status to 'error' and reset the instance vm_state to 'active' and task_state to None | |
| 15:54:07 | watersj | no, looking at nova-compute logs did not give me much info. Currently vm is down and I trying to get it back on | |
| 15:54:35 | watersj | i did change its state ti error | |
| 15:54:39 | watersj | and then active | |
| 15:54:42 | mriedem | watersj: ok, so you'll have to set the migration record status to 'error' and hard reboot the guest i think | |
| 15:54:48 | sean-k-mooney | artom: mriedem by the way i though when we got to post-copy stage we cant rollback anymore | |
| 15:54:53 | mriedem | watersj: the instances table record or the migrations table record? | |
| 15:54:57 | mriedem | those are different records | |
| 15:55:06 | watersj | 1sec | |
| 15:56:05 | artom | sean-k-mooney, yeah, but... is that relevant here? | |
| 15:56:14 | watersj | i ran nova reset-state <name> | |
| 15:56:17 | artom | Also, which "here"? watersj's issue, or https://review.opendev.org/#/c/594139/1 ? | |
| 15:56:20 | watersj | i ran nova reset-state --active <name> | |
| 15:56:33 | mriedem | watersj: that just updates the server's vm/task states, it doesn't do anything with a stopped guest | |
| 15:56:35 | sean-k-mooney | artom: for mriedem commit "If we've reached the point of post-copy during live migration on the source host and activated the destination host port bindings, but then something fails and we have to rollback," | |
| 15:56:46 | watersj | the guest is not running | |
| 15:56:51 | watersj | that is confirmed | |
| 15:57:02 | mriedem | right, that's why i said you'll have to hard reboot it | |
| 15:57:08 | mriedem | to get it back to active/running for real | |
| 15:57:12 | mriedem | bye! | |
| 15:57:22 | artom | You're welcome? | |
| 15:57:26 | artom | sean-k-mooney, huh, good spot | |
| 15:57:40 | artom | Once we're in post-copy, we can't rollback, as the VM is now running on the dest | |
| 15:57:52 | sean-k-mooney | yes | |
| 15:57:53 | artom | https://wiki.qemu.org/Features/PostCopyLiveMigration#Summary | |
| 15:58:06 | sean-k-mooney | and libvirt cant rollback at that point | |
| 15:58:53 | artom | Although I think that's just the commit message | |
| 15:59:21 | artom | Your bug (https://bugs.launchpad.net/nova/+bug/1788014) doesn't mention post-copy | |
| 15:59:21 | openstack | Launchpad bug 1788014 in OpenStack Compute (nova) "when live migration fails due to a internal error rollback is not handeled correctly." [Medium,In progress] - Assigned to Matt Riedemann (mriedem) | |
| 16:00:01 | artom | And the new code is being called from the existing _rollback_live_migration() method, so it's not like we're attempting to rollback a post-copy migration | |
| 16:00:03 | mriedem | i likely mentioned post-copy in the commit message b/c the lifecycle handler callback thing will activate the dest host port binding if the migration switches to post-copy | |
| 16:00:46 | mriedem | this code https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1252 | |
| 16:01:10 | watersj | Instance not resizing, skipping migration, and looks to be staying there | |
| 16:01:21 | watersj | from nova-compute logs | |
| 16:01:29 | mriedem | but it doesn't have to be post-copy to switch bindings https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1203 | |
| 16:01:49 | mriedem | watersj: yes, i realize b/c the resource tracker ignores live migrations in stein | |
| 16:02:07 | mriedem | watersj: like i said, you'll have to find and manually change the migrations table record to 'error' status and then hard reboot the server | |
| 16:02:19 | watersj | ah, k | |
| 16:02:30 | mriedem | the former takes it out of the RT loop which is why you're seeing "Instance not resizing, skipping migration" and the latter is to get the guest running agian | |
| 16:02:31 | mriedem | *again | |
| 16:02:48 | watersj | ty, give it try | |
| 16:03:08 | mriedem | something like this should find the migration: | |
| 16:03:08 | artom | mriedem, so EVENT_LIFECYCLE_POSTCOPY_STARTED and EVENT_LIFECYCLE_MIGRATION_COMPLETED mean the same thing in that context: the VM is now irreversibly running on the dest | |
| 16:03:20 | artom | mriedem, so it makes sense to update the port bindings at that point | |
| 16:03:39 | mriedem | select * from nova_cell1.migrations where instance_uuid=<server id> and deleted=0 and status='pre-migrating'; | |
| 16:03:44 | sean-k-mooney | artom: not nessisarally | |
| 16:03:51 | sean-k-mooney | EVENT_LIFECYCLE_MIGRATION_COMPLETED i think can be sent for failure | |
| 16:04:04 | mriedem | yup | |
| 16:04:07 | mriedem | which is another patch of mine | |
| 16:04:26 | mriedem | https://review.opendev.org/#/c/594527/ | |
| 16:05:02 | openstackgerrit | Matt Riedemann proposed openstack/nova master: libvirt: check job status for VIR_DOMAIN_EVENT_SUSPENDED_MIGRATED event https://review.opendev.org/594527 | |
| 16:05:09 | artom | Aha, TIL | |
| 16:05:24 | artom | (Actually, I think I already checked this when faking it for my NUMA LM func tests, but forgot) | |
| 16:06:00 | artom | (Which, btw, are waiting for review - but given the last thing was Stephen's -1 which I addressed, maybe wait for him to get pack from PTG) | |
| 16:06:20 | artom | We've long passed any sense of urgency, at this point | |
| 16:06:58 | sean-k-mooney | oh actully i think i know how to repoduce the failure by the way | |
| 16:07:32 | sean-k-mooney | i think i found this issue while trying to live migrate between kerenl ovs and ovs dpdk were the virtio feature flags change | |
| 16:07:43 | sean-k-mooney | due to setting the mtu in one and not in the other | |
| 16:08:42 | sean-k-mooney | strangely enough i set up a dpdk env to work on that and and another dpdk thing on friday | |
| 16:10:14 | sean-k-mooney | artom: this was part of the orginal error http://paste.openstack.org/show/728539/ | |
| 16:11:56 | artom | sean-k-mooney, ah, so you hit this because the source sent some XML the dest couldn't handle, and QEMU died | |
| 16:12:06 | sean-k-mooney | yep | |
| 16:12:11 | sean-k-mooney | libvirt was fine with it | |
| 16:12:28 | sean-k-mooney | well you can see the error | |
| 16:12:38 | artom | Right, the XML itself was "fine" | |
| 16:12:49 | artom | But the generated qemu command line exploded it | |
| 16:12:55 | sean-k-mooney | ya but not for the migration | |
| 16:13:11 | sean-k-mooney | qemu also acccpeted teh commandline | |
| 16:13:18 | sean-k-mooney | it was only an issue when the migration started | |
| 16:13:24 | mriedem | so if https://review.opendev.org/#/c/594527/ fixes that to not activate the dest host port bindings, then we don't need to rollback and maybe i can just abandon https://review.opendev.org/594139 | |
| 16:14:03 | sean-k-mooney | ya maybe | |
| 16:14:20 | artom | I wonder if we could reproduce that in the gate, with a DNM patch that borks the LM on purpose | |
| 16:14:23 | sean-k-mooney | i can try and recreate teh excat error i had and do some testing i guess | |
| 16:14:48 | sean-k-mooney | am i might be able to do that yes... | |
| 16:15:06 | sean-k-mooney | it woudl be quicker to do it locally first | |
| 16:15:17 | sean-k-mooney | i have to go fix my dpdk job | |
| 16:15:47 | artom | (Apologies to infra and the nodepool donors) | |
| 16:16:01 | sean-k-mooney | if i make it work again i could have 1 node be kernel ovs and the other dpdk and then replicte this with a simple live migrate test | |
| 16:16:42 | artom | I'm assuming hacking Nova to just generate broken XML for the dest won't be enough? | |
| 16:16:59 | sean-k-mooney | well yes | |
| 16:17:19 | sean-k-mooney | to get teh conflcit one of the xmls shoudl have the mtu set and the udpated one not | |
| 16:17:36 | watersj | mriedem, just to verify update migrations set status = 'error' where id = 76; | |
| 16:17:51 | sean-k-mooney | that will change between the 32bit feature flags and teh 64bit feature flags and will cause the error | |
| 16:18:01 | watersj | id 76 is record in question | |
| 16:18:02 | artom | sean-k-mooney, so why not do that? DNM patch that adds a "break_xml()" to _update_xml() on the source | |
| 16:18:11 | artom | Remove all the non-LM testing from Zuul | |
| 16:18:14 | artom | And play around with that | |
| 16:18:33 | sean-k-mooney | artom: so your going to write that patch ya :) | |
| 16:18:46 | artom | sean-k-mooney, I don't mind | |
| 16:19:49 | mriedem | watersj: yeah | |
| 16:20:34 | sean-k-mooney | well it shoudl be pretty easy to do i guess. am i need to look at that downstream bug breifly but i can also maybe look at this with you after i figure out if we can backport that fix | |
| 16:20:52 | artom | sean-k-mooney, there's a downstream bug for that? | |
| 16:20:55 | sean-k-mooney | that fix bing the binding_failded thing | |
| 16:21:00 | sean-k-mooney | no a different thing | |
| 16:21:06 | artom | Ah | |
| 16:21:17 | artom | Because one of the things we were discussing is are we backporting the fix | |
| 16:21:29 | artom | Which influences how invasive it can be | |
| 16:21:38 | mriedem | artom: (10:13:25 AM) mriedem: so if https://review.opendev.org/#/c/594527/ fixes that to not activate the dest host port bindings, then we don't need to rollback and maybe i can just abandon https://review.opendev.org/594139 | |