| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-11-05 | |||
| 15:56:14 | watersj | i ran nova reset-state <name> | |
| 15:56:17 | artom | Also, which "here"? watersj's issue, or https://review.opendev.org/#/c/594139/1 ? | |
| 15:56:20 | watersj | i ran nova reset-state --active <name> | |
| 15:56:33 | mriedem | watersj: that just updates the server's vm/task states, it doesn't do anything with a stopped guest | |
| 15:56:35 | sean-k-mooney | artom: for mriedem commit "If we've reached the point of post-copy during live migration on the source host and activated the destination host port bindings, but then something fails and we have to rollback," | |
| 15:56:46 | watersj | the guest is not running | |
| 15:56:51 | watersj | that is confirmed | |
| 15:57:02 | mriedem | right, that's why i said you'll have to hard reboot it | |
| 15:57:08 | mriedem | to get it back to active/running for real | |
| 15:57:12 | mriedem | bye! | |
| 15:57:22 | artom | You're welcome? | |
| 15:57:26 | artom | sean-k-mooney, huh, good spot | |
| 15:57:40 | artom | Once we're in post-copy, we can't rollback, as the VM is now running on the dest | |
| 15:57:52 | sean-k-mooney | yes | |
| 15:57:53 | artom | https://wiki.qemu.org/Features/PostCopyLiveMigration#Summary | |
| 15:58:06 | sean-k-mooney | and libvirt cant rollback at that point | |
| 15:58:53 | artom | Although I think that's just the commit message | |
| 15:59:21 | artom | Your bug (https://bugs.launchpad.net/nova/+bug/1788014) doesn't mention post-copy | |
| 15:59:21 | openstack | Launchpad bug 1788014 in OpenStack Compute (nova) "when live migration fails due to a internal error rollback is not handeled correctly." [Medium,In progress] - Assigned to Matt Riedemann (mriedem) | |
| 16:00:01 | artom | And the new code is being called from the existing _rollback_live_migration() method, so it's not like we're attempting to rollback a post-copy migration | |
| 16:00:03 | mriedem | i likely mentioned post-copy in the commit message b/c the lifecycle handler callback thing will activate the dest host port binding if the migration switches to post-copy | |
| 16:00:46 | mriedem | this code https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1252 | |
| 16:01:10 | watersj | Instance not resizing, skipping migration, and looks to be staying there | |
| 16:01:21 | watersj | from nova-compute logs | |
| 16:01:29 | mriedem | but it doesn't have to be post-copy to switch bindings https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1203 | |
| 16:01:49 | mriedem | watersj: yes, i realize b/c the resource tracker ignores live migrations in stein | |
| 16:02:07 | mriedem | watersj: like i said, you'll have to find and manually change the migrations table record to 'error' status and then hard reboot the server | |
| 16:02:19 | watersj | ah, k | |
| 16:02:30 | mriedem | the former takes it out of the RT loop which is why you're seeing "Instance not resizing, skipping migration" and the latter is to get the guest running agian | |
| 16:02:31 | mriedem | *again | |
| 16:02:48 | watersj | ty, give it try | |
| 16:03:08 | mriedem | something like this should find the migration: | |
| 16:03:08 | artom | mriedem, so EVENT_LIFECYCLE_POSTCOPY_STARTED and EVENT_LIFECYCLE_MIGRATION_COMPLETED mean the same thing in that context: the VM is now irreversibly running on the dest | |
| 16:03:20 | artom | mriedem, so it makes sense to update the port bindings at that point | |
| 16:03:39 | mriedem | select * from nova_cell1.migrations where instance_uuid=<server id> and deleted=0 and status='pre-migrating'; | |
| 16:03:44 | sean-k-mooney | artom: not nessisarally | |
| 16:03:51 | sean-k-mooney | EVENT_LIFECYCLE_MIGRATION_COMPLETED i think can be sent for failure | |
| 16:04:04 | mriedem | yup | |
| 16:04:07 | mriedem | which is another patch of mine | |
| 16:04:26 | mriedem | https://review.opendev.org/#/c/594527/ | |
| 16:05:02 | openstackgerrit | Matt Riedemann proposed openstack/nova master: libvirt: check job status for VIR_DOMAIN_EVENT_SUSPENDED_MIGRATED event https://review.opendev.org/594527 | |
| 16:05:09 | artom | Aha, TIL | |
| 16:05:24 | artom | (Actually, I think I already checked this when faking it for my NUMA LM func tests, but forgot) | |
| 16:06:00 | artom | (Which, btw, are waiting for review - but given the last thing was Stephen's -1 which I addressed, maybe wait for him to get pack from PTG) | |
| 16:06:20 | artom | We've long passed any sense of urgency, at this point | |
| 16:06:58 | sean-k-mooney | oh actully i think i know how to repoduce the failure by the way | |
| 16:07:32 | sean-k-mooney | i think i found this issue while trying to live migrate between kerenl ovs and ovs dpdk were the virtio feature flags change | |
| 16:07:43 | sean-k-mooney | due to setting the mtu in one and not in the other | |
| 16:08:42 | sean-k-mooney | strangely enough i set up a dpdk env to work on that and and another dpdk thing on friday | |
| 16:10:14 | sean-k-mooney | artom: this was part of the orginal error http://paste.openstack.org/show/728539/ | |
| 16:11:56 | artom | sean-k-mooney, ah, so you hit this because the source sent some XML the dest couldn't handle, and QEMU died | |
| 16:12:06 | sean-k-mooney | yep | |
| 16:12:11 | sean-k-mooney | libvirt was fine with it | |
| 16:12:28 | sean-k-mooney | well you can see the error | |
| 16:12:38 | artom | Right, the XML itself was "fine" | |
| 16:12:49 | artom | But the generated qemu command line exploded it | |
| 16:12:55 | sean-k-mooney | ya but not for the migration | |
| 16:13:11 | sean-k-mooney | qemu also acccpeted teh commandline | |
| 16:13:18 | sean-k-mooney | it was only an issue when the migration started | |
| 16:13:24 | mriedem | so if https://review.opendev.org/#/c/594527/ fixes that to not activate the dest host port bindings, then we don't need to rollback and maybe i can just abandon https://review.opendev.org/594139 | |
| 16:14:03 | sean-k-mooney | ya maybe | |
| 16:14:20 | artom | I wonder if we could reproduce that in the gate, with a DNM patch that borks the LM on purpose | |
| 16:14:23 | sean-k-mooney | i can try and recreate teh excat error i had and do some testing i guess | |
| 16:14:48 | sean-k-mooney | am i might be able to do that yes... | |
| 16:15:06 | sean-k-mooney | it woudl be quicker to do it locally first | |
| 16:15:17 | sean-k-mooney | i have to go fix my dpdk job | |
| 16:15:47 | artom | (Apologies to infra and the nodepool donors) | |
| 16:16:01 | sean-k-mooney | if i make it work again i could have 1 node be kernel ovs and the other dpdk and then replicte this with a simple live migrate test | |
| 16:16:42 | artom | I'm assuming hacking Nova to just generate broken XML for the dest won't be enough? | |
| 16:16:59 | sean-k-mooney | well yes | |
| 16:17:19 | sean-k-mooney | to get teh conflcit one of the xmls shoudl have the mtu set and the udpated one not | |
| 16:17:36 | watersj | mriedem, just to verify update migrations set status = 'error' where id = 76; | |
| 16:17:51 | sean-k-mooney | that will change between the 32bit feature flags and teh 64bit feature flags and will cause the error | |
| 16:18:01 | watersj | id 76 is record in question | |
| 16:18:02 | artom | sean-k-mooney, so why not do that? DNM patch that adds a "break_xml()" to _update_xml() on the source | |
| 16:18:11 | artom | Remove all the non-LM testing from Zuul | |
| 16:18:14 | artom | And play around with that | |
| 16:18:33 | sean-k-mooney | artom: so your going to write that patch ya :) | |
| 16:18:46 | artom | sean-k-mooney, I don't mind | |
| 16:19:49 | mriedem | watersj: yeah | |
| 16:20:34 | sean-k-mooney | well it shoudl be pretty easy to do i guess. am i need to look at that downstream bug breifly but i can also maybe look at this with you after i figure out if we can backport that fix | |
| 16:20:52 | artom | sean-k-mooney, there's a downstream bug for that? | |
| 16:20:55 | sean-k-mooney | that fix bing the binding_failded thing | |
| 16:21:00 | sean-k-mooney | no a different thing | |
| 16:21:06 | artom | Ah | |
| 16:21:17 | artom | Because one of the things we were discussing is are we backporting the fix | |
| 16:21:29 | artom | Which influences how invasive it can be | |
| 16:21:38 | mriedem | artom: (10:13:25 AM) mriedem: so if https://review.opendev.org/#/c/594527/ fixes that to not activate the dest host port bindings, then we don't need to rollback and maybe i can just abandon https://review.opendev.org/594139 | |
| 16:22:15 | sean-k-mooney | mriedem: if i can recreate it ill apply that patch and see if it fixes it | |
| 16:22:46 | mriedem | ack | |
| 16:22:49 | artom | mriedem, ah, yeah, it we never update the port bindings for a failed LM, we don't need to roll them back | |
| 16:23:04 | artom | So, would there be value in a DNM to repro that in the gate? | |
| 16:23:07 | artom | *DNM patch | |
| 16:26:41 | watersj | I'm getting, There are allocations remaining against the source host that might need to be removed | |
| 16:27:39 | sean-k-mooney | artom: if you can repro it in the gate we can use that to test both of mriedem patches so yes i think there is still value | |
| 16:28:01 | sean-k-mooney | although long term a functional regression test would be optimal | |
| 16:28:13 | artom | sean-k-mooney, mriedem was saying we need to fix the Neutron fixture for that | |
| 16:28:24 | sean-k-mooney | proably | |
| 16:28:59 | artom | sean-k-mooney, so what was that thing about the MTU? Removing it should trigger thing? Seems weird | |
| 16:29:05 | sean-k-mooney | it does not really do the multiple port bindings workflow right | |