Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-11-05
16:03:39 mriedem select * from nova_cell1.migrations where instance_uuid=<server id> and deleted=0 and status='pre-migrating';
16:03:44 sean-k-mooney artom: not nessisarally
16:03:51 sean-k-mooney EVENT_LIFECYCLE_MIGRATION_COMPLETED i think can be sent for failure
16:04:04 mriedem yup
16:04:07 mriedem which is another patch of mine
16:04:26 mriedem https://review.opendev.org/#/c/594527/
16:05:02 openstackgerrit Matt Riedemann proposed openstack/nova master: libvirt: check job status for VIR_DOMAIN_EVENT_SUSPENDED_MIGRATED event https://review.opendev.org/594527
16:05:09 artom Aha, TIL
16:05:24 artom (Actually, I think I already checked this when faking it for my NUMA LM func tests, but forgot)
16:06:00 artom (Which, btw, are waiting for review - but given the last thing was Stephen's -1 which I addressed, maybe wait for him to get pack from PTG)
16:06:20 artom We've long passed any sense of urgency, at this point
16:06:58 sean-k-mooney oh actully i think i know how to repoduce the failure by the way
16:07:32 sean-k-mooney i think i found this issue while trying to live migrate between kerenl ovs and ovs dpdk were the virtio feature flags change
16:07:43 sean-k-mooney due to setting the mtu in one and not in the other
16:08:42 sean-k-mooney strangely enough i set up a dpdk env to work on that and and another dpdk thing on friday
16:10:14 sean-k-mooney artom: this was part of the orginal error http://paste.openstack.org/show/728539/
16:11:56 artom sean-k-mooney, ah, so you hit this because the source sent some XML the dest couldn't handle, and QEMU died
16:12:06 sean-k-mooney yep
16:12:11 sean-k-mooney libvirt was fine with it
16:12:28 sean-k-mooney well you can see the error
16:12:38 artom Right, the XML itself was "fine"
16:12:49 artom But the generated qemu command line exploded it
16:12:55 sean-k-mooney ya but not for the migration
16:13:11 sean-k-mooney qemu also acccpeted teh commandline
16:13:18 sean-k-mooney it was only an issue when the migration started
16:13:24 mriedem so if https://review.opendev.org/#/c/594527/ fixes that to not activate the dest host port bindings, then we don't need to rollback and maybe i can just abandon https://review.opendev.org/594139
16:14:03 sean-k-mooney ya maybe
16:14:20 artom I wonder if we could reproduce that in the gate, with a DNM patch that borks the LM on purpose
16:14:23 sean-k-mooney i can try and recreate teh excat error i had and do some testing i guess
16:14:48 sean-k-mooney am i might be able to do that yes...
16:15:06 sean-k-mooney it woudl be quicker to do it locally first
16:15:17 sean-k-mooney i have to go fix my dpdk job
16:15:47 artom (Apologies to infra and the nodepool donors)
16:16:01 sean-k-mooney if i make it work again i could have 1 node be kernel ovs and the other dpdk and then replicte this with a simple live migrate test
16:16:42 artom I'm assuming hacking Nova to just generate broken XML for the dest won't be enough?
16:16:59 sean-k-mooney well yes
16:17:19 sean-k-mooney to get teh conflcit one of the xmls shoudl have the mtu set and the udpated one not
16:17:36 watersj mriedem, just to verify update migrations set status = 'error' where id = 76;
16:17:51 sean-k-mooney that will change between the 32bit feature flags and teh 64bit feature flags and will cause the error
16:18:01 watersj id 76 is record in question
16:18:02 artom sean-k-mooney, so why not do that? DNM patch that adds a "break_xml()" to _update_xml() on the source
16:18:11 artom Remove all the non-LM testing from Zuul
16:18:14 artom And play around with that
16:18:33 sean-k-mooney artom: so your going to write that patch ya :)
16:18:46 artom sean-k-mooney, I don't mind
16:19:49 mriedem watersj: yeah
16:20:34 sean-k-mooney well it shoudl be pretty easy to do i guess. am i need to look at that downstream bug breifly but i can also maybe look at this with you after i figure out if we can backport that fix
16:20:52 artom sean-k-mooney, there's a downstream bug for that?
16:20:55 sean-k-mooney that fix bing the binding_failded thing
16:21:00 sean-k-mooney no a different thing
16:21:06 artom Ah
16:21:17 artom Because one of the things we were discussing is are we backporting the fix
16:21:29 artom Which influences how invasive it can be
16:21:38 mriedem artom: (10:13:25 AM) mriedem: so if https://review.opendev.org/#/c/594527/ fixes that to not activate the dest host port bindings, then we don't need to rollback and maybe i can just abandon https://review.opendev.org/594139
16:22:15 sean-k-mooney mriedem: if i can recreate it ill apply that patch and see if it fixes it
16:22:46 mriedem ack
16:22:49 artom mriedem, ah, yeah, it we never update the port bindings for a failed LM, we don't need to roll them back
16:23:04 artom So, would there be value in a DNM to repro that in the gate?
16:23:07 artom *DNM patch
16:26:41 watersj I'm getting, There are allocations remaining against the source host that might need to be removed
16:27:39 sean-k-mooney artom: if you can repro it in the gate we can use that to test both of mriedem patches so yes i think there is still value
16:28:01 sean-k-mooney although long term a functional regression test would be optimal
16:28:13 artom sean-k-mooney, mriedem was saying we need to fix the Neutron fixture for that
16:28:24 sean-k-mooney proably
16:28:59 artom sean-k-mooney, so what was that thing about the MTU? Removing it should trigger thing? Seems weird
16:29:05 sean-k-mooney it does not really do the multiple port bindings workflow right
16:29:35 sean-k-mooney artom: ya remove it form the port forces libvirt to change form teh 64bit encodeing to the 32bit encoding
16:29:59 sean-k-mooney well adding it forces it
16:31:04 sean-k-mooney anyway the root of that secific libvirt/qemu issue is they ran out of feature bits and adding and removing hte mtu is one way to chagne it
16:31:28 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Create BDM in API during attach volume https://review.opendev.org/692940
16:31:28 openstackgerrit Matt Riedemann proposed openstack/nova master: Provide a better error when _verify_response hits a TypeError https://review.opendev.org/693042
16:33:30 mriedem watersj: ah yeah...so in placement the migration record (id=76) holds the allocations for the source host and the instance holds allocations for the dest host, and you need to move the instance allocations back to the source host and drop the allocations from the dest host, which doesn't sound fun right
16:33:49 mriedem luckily i just wrote a guide on how to do this https://review.opendev.org/#/c/691427/
16:34:03 watersj lol
16:34:27 mriedem you might want to just skip straight to the heal_allocations part
16:34:44 mriedem i.e. delete the allocations for both the migration and instance consumer and then heal allocations for the instance
16:34:59 mriedem https://docs.openstack.org/osc-placement/latest/cli/index.html#resource-provider-allocation-delete
16:35:11 mriedem you'd pass the migration uuid and instance uuid to ^ to remove the allocations in placement
16:35:23 mriedem then run: nova-manage placemet heal_allocations
16:35:45 mriedem https://docs.openstack.org/nova/stein/cli/nova-manage.html#placement
16:36:28 mriedem if you want to target the specific instance, or do a dry-run heal_allocations, you'd need to setup a train nova instance in a vm/container/virtualenv https://docs.openstack.org/nova/train/cli/nova-manage.html#placement since --dry-run and --instance options aren't in stein
16:37:02 watersj have backup of volume
16:37:08 watersj ready really break it
16:37:16 mriedem i don't know what that means
16:37:27 mriedem oh you mean you do'nt need a dry run
16:37:52 mriedem heal allocations just recreates the missing allocation records in placement for the instance using the flavor data associated with the instance
16:37:58 watersj mean I ok to try without dryrun capabilities
16:37:59 mriedem doesn't touch the guest or anything like that
16:38:04 openstackgerrit Artom Lifshitz proposed openstack/nova master: DNM: break live migration to trigger bug 1788014 https://review.opendev.org/693045
16:38:04 openstack bug 1788014 in OpenStack Compute (nova) "when live migration fails due to a internal error rollback is not handeled correctly." [Medium,In progress] https://launchpad.net/bugs/1788014 - Assigned to Matt Riedemann (mriedem)
16:38:32 watersj k, ty for the help
16:39:15 openstackgerrit Merged openstack/nova stable/stein: Add integration testing for heal_allocations https://review.opendev.org/692923
16:57:10 mriedem dansmith: replied in https://review.opendev.org/#/c/635668/ - i'm not sure exactly what you're looking for so i gave some options
16:57:56 mriedem if you want to see a paste of what i'm talking about i can do that quick
16:59:07 openstackgerrit Artom Lifshitz proposed openstack/nova master: DNM: break live migration to trigger bug 1788014 https://review.opendev.org/693045
16:59:07 openstack bug 1788014 in OpenStack Compute (nova) "when live migration fails due to a internal error rollback is not handeled correctly." [Medium,In progress] https://launchpad.net/bugs/1788014 - Assigned to Matt Riedemann (mriedem)
17:01:58 dansmith mriedem: seems like you get it to me :)
17:02:14 mriedem ok
17:02:19 mriedem updating thusly
17:23:51 melwitt whoa, one of my things merged the first time

Earlier   Later