Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-11-05
16:15:17 sean-k-mooney i have to go fix my dpdk job
16:15:47 artom (Apologies to infra and the nodepool donors)
16:16:01 sean-k-mooney if i make it work again i could have 1 node be kernel ovs and the other dpdk and then replicte this with a simple live migrate test
16:16:42 artom I'm assuming hacking Nova to just generate broken XML for the dest won't be enough?
16:16:59 sean-k-mooney well yes
16:17:19 sean-k-mooney to get teh conflcit one of the xmls shoudl have the mtu set and the udpated one not
16:17:36 watersj mriedem, just to verify update migrations set status = 'error' where id = 76;
16:17:51 sean-k-mooney that will change between the 32bit feature flags and teh 64bit feature flags and will cause the error
16:18:01 watersj id 76 is record in question
16:18:02 artom sean-k-mooney, so why not do that? DNM patch that adds a "break_xml()" to _update_xml() on the source
16:18:11 artom Remove all the non-LM testing from Zuul
16:18:14 artom And play around with that
16:18:33 sean-k-mooney artom: so your going to write that patch ya :)
16:18:46 artom sean-k-mooney, I don't mind
16:19:49 mriedem watersj: yeah
16:20:34 sean-k-mooney well it shoudl be pretty easy to do i guess. am i need to look at that downstream bug breifly but i can also maybe look at this with you after i figure out if we can backport that fix
16:20:52 artom sean-k-mooney, there's a downstream bug for that?
16:20:55 sean-k-mooney that fix bing the binding_failded thing
16:21:00 sean-k-mooney no a different thing
16:21:06 artom Ah
16:21:17 artom Because one of the things we were discussing is are we backporting the fix
16:21:29 artom Which influences how invasive it can be
16:21:38 mriedem artom: (10:13:25 AM) mriedem: so if https://review.opendev.org/#/c/594527/ fixes that to not activate the dest host port bindings, then we don't need to rollback and maybe i can just abandon https://review.opendev.org/594139
16:22:15 sean-k-mooney mriedem: if i can recreate it ill apply that patch and see if it fixes it
16:22:46 mriedem ack
16:22:49 artom mriedem, ah, yeah, it we never update the port bindings for a failed LM, we don't need to roll them back
16:23:04 artom So, would there be value in a DNM to repro that in the gate?
16:23:07 artom *DNM patch
16:26:41 watersj I'm getting, There are allocations remaining against the source host that might need to be removed
16:27:39 sean-k-mooney artom: if you can repro it in the gate we can use that to test both of mriedem patches so yes i think there is still value
16:28:01 sean-k-mooney although long term a functional regression test would be optimal
16:28:13 artom sean-k-mooney, mriedem was saying we need to fix the Neutron fixture for that
16:28:24 sean-k-mooney proably
16:28:59 artom sean-k-mooney, so what was that thing about the MTU? Removing it should trigger thing? Seems weird
16:29:05 sean-k-mooney it does not really do the multiple port bindings workflow right
16:29:35 sean-k-mooney artom: ya remove it form the port forces libvirt to change form teh 64bit encodeing to the 32bit encoding
16:29:59 sean-k-mooney well adding it forces it
16:31:04 sean-k-mooney anyway the root of that secific libvirt/qemu issue is they ran out of feature bits and adding and removing hte mtu is one way to chagne it
16:31:28 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Create BDM in API during attach volume https://review.opendev.org/692940
16:31:28 openstackgerrit Matt Riedemann proposed openstack/nova master: Provide a better error when _verify_response hits a TypeError https://review.opendev.org/693042
16:33:30 mriedem watersj: ah yeah...so in placement the migration record (id=76) holds the allocations for the source host and the instance holds allocations for the dest host, and you need to move the instance allocations back to the source host and drop the allocations from the dest host, which doesn't sound fun right
16:33:49 mriedem luckily i just wrote a guide on how to do this https://review.opendev.org/#/c/691427/
16:34:03 watersj lol
16:34:27 mriedem you might want to just skip straight to the heal_allocations part
16:34:44 mriedem i.e. delete the allocations for both the migration and instance consumer and then heal allocations for the instance
16:34:59 mriedem https://docs.openstack.org/osc-placement/latest/cli/index.html#resource-provider-allocation-delete
16:35:11 mriedem you'd pass the migration uuid and instance uuid to ^ to remove the allocations in placement
16:35:23 mriedem then run: nova-manage placemet heal_allocations
16:35:45 mriedem https://docs.openstack.org/nova/stein/cli/nova-manage.html#placement
16:36:28 mriedem if you want to target the specific instance, or do a dry-run heal_allocations, you'd need to setup a train nova instance in a vm/container/virtualenv https://docs.openstack.org/nova/train/cli/nova-manage.html#placement since --dry-run and --instance options aren't in stein
16:37:02 watersj have backup of volume
16:37:08 watersj ready really break it
16:37:16 mriedem i don't know what that means
16:37:27 mriedem oh you mean you do'nt need a dry run
16:37:52 mriedem heal allocations just recreates the missing allocation records in placement for the instance using the flavor data associated with the instance
16:37:58 watersj mean I ok to try without dryrun capabilities
16:37:59 mriedem doesn't touch the guest or anything like that
16:38:04 openstackgerrit Artom Lifshitz proposed openstack/nova master: DNM: break live migration to trigger bug 1788014 https://review.opendev.org/693045
16:38:04 openstack bug 1788014 in OpenStack Compute (nova) "when live migration fails due to a internal error rollback is not handeled correctly." [Medium,In progress] https://launchpad.net/bugs/1788014 - Assigned to Matt Riedemann (mriedem)
16:38:32 watersj k, ty for the help
16:39:15 openstackgerrit Merged openstack/nova stable/stein: Add integration testing for heal_allocations https://review.opendev.org/692923
16:57:10 mriedem dansmith: replied in https://review.opendev.org/#/c/635668/ - i'm not sure exactly what you're looking for so i gave some options
16:57:56 mriedem if you want to see a paste of what i'm talking about i can do that quick
16:59:07 openstackgerrit Artom Lifshitz proposed openstack/nova master: DNM: break live migration to trigger bug 1788014 https://review.opendev.org/693045
16:59:07 openstack bug 1788014 in OpenStack Compute (nova) "when live migration fails due to a internal error rollback is not handeled correctly." [Medium,In progress] https://launchpad.net/bugs/1788014 - Assigned to Matt Riedemann (mriedem)
17:01:58 dansmith mriedem: seems like you get it to me :)
17:02:14 mriedem ok
17:02:19 mriedem updating thusly
17:23:51 melwitt whoa, one of my things merged the first time
17:26:08 dansmith you know
17:26:22 dansmith it has been said that on several providers we're our own noisy neighbor
17:26:32 dansmith if we see a massive drop in the IO-based failures this week,
17:26:45 dansmith that would probably just go to confirm that as a likely problem
17:26:51 dansmith less volume for samples for sure, but..
17:29:01 melwitt yeah, I was wondering that too. summit/ptg and if suddenly less gate failures
17:30:56 mriedem gate has been smooth has rich creamery butter this week
17:32:00 mriedem https://www.youtube.com/watch?v=3ds0vWfoTwU
17:33:51 melwitt lol
17:35:10 mriedem i didn't actually think of the simpsons when i said that, but now i realize it's just baked into my brain from years of watching those recorded episodes on vhs as a kid
17:35:35 mriedem "we didn't have streaming in my day..."
17:35:46 melwitt that's impressive
17:36:30 sean-k-mooney even with out streaming like friends the simsons is one of those shows that is on a loop on cable/satialite tv constantly
17:36:52 mriedem we didn't have cable until i was 17
17:36:54 mriedem hence the vhs
17:37:18 mriedem and then it was all WWE and Love Line years after my friends were sick of those shows
17:37:33 sean-k-mooney some what similar i was maybe 12/14 but that just becasue ireland and it not being a thing until the 90s
17:38:15 dansmith "This channel is for Nova development and mriedem's childhood nostalgia"
17:38:40 sean-k-mooney dansmith: and music references form the 80s
17:38:51 dansmith sean-k-mooney: same category
17:39:52 mriedem let it be so
17:40:01 sean-k-mooney well now that im donw backporting a patch we backported 6 months ago i think im going to get food then do something useful...
17:50:44 openstackgerrit melanie witt proposed openstack/nova stable/stein: Add regression test for bug 1824435 https://review.opendev.org/692906
17:50:44 openstack bug 1824435 in OpenStack Compute (nova) stein "fill_virtual_interface_list migration fails on second attempt" [Medium,In progress] https://launchpad.net/bugs/1824435 - Assigned to melanie witt (melwitt)
17:50:44 openstackgerrit melanie witt proposed openstack/nova stable/stein: Remove redundant call to get/create default security group https://review.opendev.org/692907
18:06:37 watersj is openstack resource provider a command available to stein? or am I supposed to replace "resource provider" with something else ?
18:07:37 melwitt watersj: you need to install osc-placement in order to get the placement cli commands
18:10:08 watersj melwitt, ty
18:17:08 openstackgerrit Matt Riedemann proposed openstack/nova master: Execute CrossCellMigrationTask from MigrationTask https://review.opendev.org/635668
18:17:09 openstackgerrit Matt Riedemann proposed openstack/nova master: Refresh instance in MigrationTask.execute Exception handler https://review.opendev.org/669012
18:17:09 openstackgerrit Matt Riedemann proposed openstack/nova master: Plumb allow_cross_cell_resize into compute API resize() https://review.opendev.org/635684

Earlier   Later