| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-11-05 | |||
| 16:12:06 | sean-k-mooney | yep | |
| 16:12:11 | sean-k-mooney | libvirt was fine with it | |
| 16:12:28 | sean-k-mooney | well you can see the error | |
| 16:12:38 | artom | Right, the XML itself was "fine" | |
| 16:12:49 | artom | But the generated qemu command line exploded it | |
| 16:12:55 | sean-k-mooney | ya but not for the migration | |
| 16:13:11 | sean-k-mooney | qemu also acccpeted teh commandline | |
| 16:13:18 | sean-k-mooney | it was only an issue when the migration started | |
| 16:13:24 | mriedem | so if https://review.opendev.org/#/c/594527/ fixes that to not activate the dest host port bindings, then we don't need to rollback and maybe i can just abandon https://review.opendev.org/594139 | |
| 16:14:03 | sean-k-mooney | ya maybe | |
| 16:14:20 | artom | I wonder if we could reproduce that in the gate, with a DNM patch that borks the LM on purpose | |
| 16:14:23 | sean-k-mooney | i can try and recreate teh excat error i had and do some testing i guess | |
| 16:14:48 | sean-k-mooney | am i might be able to do that yes... | |
| 16:15:06 | sean-k-mooney | it woudl be quicker to do it locally first | |
| 16:15:17 | sean-k-mooney | i have to go fix my dpdk job | |
| 16:15:47 | artom | (Apologies to infra and the nodepool donors) | |
| 16:16:01 | sean-k-mooney | if i make it work again i could have 1 node be kernel ovs and the other dpdk and then replicte this with a simple live migrate test | |
| 16:16:42 | artom | I'm assuming hacking Nova to just generate broken XML for the dest won't be enough? | |
| 16:16:59 | sean-k-mooney | well yes | |
| 16:17:19 | sean-k-mooney | to get teh conflcit one of the xmls shoudl have the mtu set and the udpated one not | |
| 16:17:36 | watersj | mriedem, just to verify update migrations set status = 'error' where id = 76; | |
| 16:17:51 | sean-k-mooney | that will change between the 32bit feature flags and teh 64bit feature flags and will cause the error | |
| 16:18:01 | watersj | id 76 is record in question | |
| 16:18:02 | artom | sean-k-mooney, so why not do that? DNM patch that adds a "break_xml()" to _update_xml() on the source | |
| 16:18:11 | artom | Remove all the non-LM testing from Zuul | |
| 16:18:14 | artom | And play around with that | |
| 16:18:33 | sean-k-mooney | artom: so your going to write that patch ya :) | |
| 16:18:46 | artom | sean-k-mooney, I don't mind | |
| 16:19:49 | mriedem | watersj: yeah | |
| 16:20:34 | sean-k-mooney | well it shoudl be pretty easy to do i guess. am i need to look at that downstream bug breifly but i can also maybe look at this with you after i figure out if we can backport that fix | |
| 16:20:52 | artom | sean-k-mooney, there's a downstream bug for that? | |
| 16:20:55 | sean-k-mooney | that fix bing the binding_failded thing | |
| 16:21:00 | sean-k-mooney | no a different thing | |
| 16:21:06 | artom | Ah | |
| 16:21:17 | artom | Because one of the things we were discussing is are we backporting the fix | |
| 16:21:29 | artom | Which influences how invasive it can be | |
| 16:21:38 | mriedem | artom: (10:13:25 AM) mriedem: so if https://review.opendev.org/#/c/594527/ fixes that to not activate the dest host port bindings, then we don't need to rollback and maybe i can just abandon https://review.opendev.org/594139 | |
| 16:22:15 | sean-k-mooney | mriedem: if i can recreate it ill apply that patch and see if it fixes it | |
| 16:22:46 | mriedem | ack | |
| 16:22:49 | artom | mriedem, ah, yeah, it we never update the port bindings for a failed LM, we don't need to roll them back | |
| 16:23:04 | artom | So, would there be value in a DNM to repro that in the gate? | |
| 16:23:07 | artom | *DNM patch | |
| 16:26:41 | watersj | I'm getting, There are allocations remaining against the source host that might need to be removed | |
| 16:27:39 | sean-k-mooney | artom: if you can repro it in the gate we can use that to test both of mriedem patches so yes i think there is still value | |
| 16:28:01 | sean-k-mooney | although long term a functional regression test would be optimal | |
| 16:28:13 | artom | sean-k-mooney, mriedem was saying we need to fix the Neutron fixture for that | |
| 16:28:24 | sean-k-mooney | proably | |
| 16:28:59 | artom | sean-k-mooney, so what was that thing about the MTU? Removing it should trigger thing? Seems weird | |
| 16:29:05 | sean-k-mooney | it does not really do the multiple port bindings workflow right | |
| 16:29:35 | sean-k-mooney | artom: ya remove it form the port forces libvirt to change form teh 64bit encodeing to the 32bit encoding | |
| 16:29:59 | sean-k-mooney | well adding it forces it | |
| 16:31:04 | sean-k-mooney | anyway the root of that secific libvirt/qemu issue is they ran out of feature bits and adding and removing hte mtu is one way to chagne it | |
| 16:31:28 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Create BDM in API during attach volume https://review.opendev.org/692940 | |
| 16:31:28 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Provide a better error when _verify_response hits a TypeError https://review.opendev.org/693042 | |
| 16:33:30 | mriedem | watersj: ah yeah...so in placement the migration record (id=76) holds the allocations for the source host and the instance holds allocations for the dest host, and you need to move the instance allocations back to the source host and drop the allocations from the dest host, which doesn't sound fun right | |
| 16:33:49 | mriedem | luckily i just wrote a guide on how to do this https://review.opendev.org/#/c/691427/ | |
| 16:34:03 | watersj | lol | |
| 16:34:27 | mriedem | you might want to just skip straight to the heal_allocations part | |
| 16:34:44 | mriedem | i.e. delete the allocations for both the migration and instance consumer and then heal allocations for the instance | |
| 16:34:59 | mriedem | https://docs.openstack.org/osc-placement/latest/cli/index.html#resource-provider-allocation-delete | |
| 16:35:11 | mriedem | you'd pass the migration uuid and instance uuid to ^ to remove the allocations in placement | |
| 16:35:23 | mriedem | then run: nova-manage placemet heal_allocations | |
| 16:35:45 | mriedem | https://docs.openstack.org/nova/stein/cli/nova-manage.html#placement | |
| 16:36:28 | mriedem | if you want to target the specific instance, or do a dry-run heal_allocations, you'd need to setup a train nova instance in a vm/container/virtualenv https://docs.openstack.org/nova/train/cli/nova-manage.html#placement since --dry-run and --instance options aren't in stein | |
| 16:37:02 | watersj | have backup of volume | |
| 16:37:08 | watersj | ready really break it | |
| 16:37:16 | mriedem | i don't know what that means | |
| 16:37:27 | mriedem | oh you mean you do'nt need a dry run | |
| 16:37:52 | mriedem | heal allocations just recreates the missing allocation records in placement for the instance using the flavor data associated with the instance | |
| 16:37:58 | watersj | mean I ok to try without dryrun capabilities | |
| 16:37:59 | mriedem | doesn't touch the guest or anything like that | |
| 16:38:04 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: DNM: break live migration to trigger bug 1788014 https://review.opendev.org/693045 | |
| 16:38:04 | openstack | bug 1788014 in OpenStack Compute (nova) "when live migration fails due to a internal error rollback is not handeled correctly." [Medium,In progress] https://launchpad.net/bugs/1788014 - Assigned to Matt Riedemann (mriedem) | |
| 16:38:32 | watersj | k, ty for the help | |
| 16:39:15 | openstackgerrit | Merged openstack/nova stable/stein: Add integration testing for heal_allocations https://review.opendev.org/692923 | |
| 16:57:10 | mriedem | dansmith: replied in https://review.opendev.org/#/c/635668/ - i'm not sure exactly what you're looking for so i gave some options | |
| 16:57:56 | mriedem | if you want to see a paste of what i'm talking about i can do that quick | |
| 16:59:07 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: DNM: break live migration to trigger bug 1788014 https://review.opendev.org/693045 | |
| 16:59:07 | openstack | bug 1788014 in OpenStack Compute (nova) "when live migration fails due to a internal error rollback is not handeled correctly." [Medium,In progress] https://launchpad.net/bugs/1788014 - Assigned to Matt Riedemann (mriedem) | |
| 17:01:58 | dansmith | mriedem: seems like you get it to me :) | |
| 17:02:14 | mriedem | ok | |
| 17:02:19 | mriedem | updating thusly | |
| 17:23:51 | melwitt | whoa, one of my things merged the first time | |
| 17:26:08 | dansmith | you know | |
| 17:26:22 | dansmith | it has been said that on several providers we're our own noisy neighbor | |
| 17:26:32 | dansmith | if we see a massive drop in the IO-based failures this week, | |
| 17:26:45 | dansmith | that would probably just go to confirm that as a likely problem | |
| 17:26:51 | dansmith | less volume for samples for sure, but.. | |
| 17:29:01 | melwitt | yeah, I was wondering that too. summit/ptg and if suddenly less gate failures | |
| 17:30:56 | mriedem | gate has been smooth has rich creamery butter this week | |
| 17:32:00 | mriedem | https://www.youtube.com/watch?v=3ds0vWfoTwU | |
| 17:33:51 | melwitt | lol | |
| 17:35:10 | mriedem | i didn't actually think of the simpsons when i said that, but now i realize it's just baked into my brain from years of watching those recorded episodes on vhs as a kid | |
| 17:35:35 | mriedem | "we didn't have streaming in my day..." | |
| 17:35:46 | melwitt | that's impressive | |
| 17:36:30 | sean-k-mooney | even with out streaming like friends the simsons is one of those shows that is on a loop on cable/satialite tv constantly | |
| 17:36:52 | mriedem | we didn't have cable until i was 17 | |
| 17:36:54 | mriedem | hence the vhs | |
| 17:37:18 | mriedem | and then it was all WWE and Love Line years after my friends were sick of those shows | |
| 17:37:33 | sean-k-mooney | some what similar i was maybe 12/14 but that just becasue ireland and it not being a thing until the 90s | |