| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-04-21 | |||
| 19:39:48 | mriedem | then around that time in syslog | |
| 19:39:48 | mriedem | Apr 21 13:54:08ubuntu-xenial-2-node-osic-cloud1-s3500-8527282 virtlogd[12188]: End of file while reading data: Input/output error | |
| 19:40:19 | kashyap | But _what_ caused the guest shutdown... | |
| 19:40:25 | mriedem | no idea | |
| 19:42:34 | kashyap | mriedem: Checking with Eric Blake & Max, both should know this code in libvirt / QEMU respectively | |
| 19:52:08 | cdent | thank you mriedem, someone had to respond to leakypipes | |
| 19:53:41 | kashyap | mriedem: Is this mig. failure blocking Gate? | |
| 19:55:22 | kashyap | At least, I found out (w/ help from Max) the commit that introduced it. | |
| 19:56:45 | mriedem | cdent: ha | |
| 19:56:53 | mriedem | kashyap: i think this is probably random failure | |
| 19:57:10 | kashyap | mriedem: Ah, okay, good. | |
| 19:57:21 | kashyap | The actual culprit failure is on the source host: | |
| 19:57:21 | kashyap | http://logs.openstack.org/43/458843/1/check/gate-tempest-dsvm-multinode-live-migration-ubuntu-xenial/697a501/logs/subnode-2/libvirt/qemu/instance-00000001.txt.gz | |
| 19:57:25 | mriedem | stable/ocata is blocked b/c of the live migration job, and to fix that i need to get this other change in, but i think this is just an intermittent failure | |
| 19:57:42 | mriedem | oh fun | |
| 19:57:43 | mriedem | qemu-system-x86_64: /build/qemu-5OJ39u/qemu-2.8+dfsg/block/io.c:1514: bdrv_co_pwritev: Assertion `!(bs->open_flags & BDRV_O_INACTIVE)' failed. | |
| 19:57:47 | kashyap | Yep | |
| 19:57:50 | mriedem | i have no idea what that means | |
| 20:02:44 | kashyap | mriedem: It means (according to the QEMU commit that introduced it): | |
| 20:03:08 | kashyap | Something tried to write to the image file on the source, _while_ it is being migrated | |
| 20:03:40 | kashyap | Anyway, it's unreasonably late here, I'll get to this on Monday | |
| 20:04:11 | kashyap | Cool; Gerrit is also offline on time. Everything is in harmony, and the earth keeps spinning | |
| 20:05:37 | mriedem | kashyap: ack, thanks for taking a look | |
| 20:06:12 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Do not log live migration success when it actually failed https://review.openstack.org/458958 | |
| 20:06:18 | mriedem | superdan: dig this ^ | |
| 20:06:33 | mriedem | sdague: ^ you too | |
| 20:06:39 | mriedem | classic terrible | |
| 20:08:30 | superdan | mriedem: does that cleanup flags thing tell us not to destroy disks, etc in the case of failure? | |
| 20:08:41 | superdan | like, I'm surprised that falling through is the right thing to do | |
| 20:09:12 | openstackgerrit | Steve Noyes proposed openstack/nova master: Add Cinder v3 detach to shutdown_instance https://review.openstack.org/456877 | |
| 20:09:29 | superdan | mriedem: doesn't look like it does to me | |
| 20:10:34 | superdan | mriedem: so if we fail, have not shared path, we'll do_cleanup=True and tell the driver to run cleanup right? | |
| 20:11:12 | mriedem | looks that way | |
| 20:11:13 | superdan | and libvirt goes right into deleting stuff | |
| 20:11:34 | mriedem | even though you should be rolling back | |
| 20:11:39 | mriedem | but this isn't where rollback is called from | |
| 20:12:20 | superdan | right, this is if post-at-destination fails, which presumably could fail for some mundane reason and then we eff things up by doing the cleanup | |
| 20:12:57 | mriedem | is live_migration started from the source or dest host? | |
| 20:13:02 | superdan | source | |
| 20:13:13 | mriedem | oh right it has the dest parameter | |
| 20:13:37 | mriedem | i lose track of the musical chairs game that is live migration | |
| 20:13:47 | superdan | yeah it's confusing | |
| 20:14:37 | mriedem | so the source side though it was done, and calls _post_live_migration (source compute stiffl) | |
| 20:14:39 | mriedem | *still | |
| 20:15:10 | mriedem | which does an rpc call to post_live_migration_at_destination on the dest host | |
| 20:15:20 | superdan | oh, we don't get here unless it actually migrated | |
| 20:15:33 | mriedem | that fails becuase the guest isn't found on the dest host | |
| 20:15:34 | superdan | and then we just failed in the post call, but whatever, we don't own it anymore | |
| 20:16:43 | mriedem | yeah post_live_migration_at_destination fails and returns the failure back through rpc | |
| 20:17:00 | mriedem | we get the failure, log the trace "Post live migration at destination %s failed" | |
| 20:17:03 | superdan | right, so I guess it's okay | |
| 20:17:12 | mriedem | then drop through and start cleaning up the source | |
| 20:17:12 | superdan | we don't get to this post method unless libvirt said it worked | |
| 20:17:16 | superdan | so we can clean up regardless | |
| 20:17:17 | mriedem | meanwhile, the guest is not on the dest either | |
| 20:17:20 | mriedem | yeah | |
| 20:17:25 | superdan | yeah, but nothing we can do about it | |
| 20:17:26 | mriedem | so basically, | |
| 20:17:30 | mriedem | your sh is f'ed | |
| 20:17:34 | mriedem | and you instance is gone | |
| 20:17:40 | mriedem | *your | |
| 20:17:45 | superdan | yeah, I just wanted to make sure we didn't delete the thing if migration failed | |
| 20:17:50 | superdan | that's a double kick in the pands | |
| 20:18:03 | mriedem | meanwhile we've deleted your disks | |
| 20:18:19 | mriedem | this is really like, well, it's all busted, oh and btw we deleted your instance | |
| 20:18:20 | mriedem | you're welcome | |
| 20:18:28 | superdan | whatever, libvirt told us to :) | |
| 20:18:37 | mriedem | wow | |
| 20:18:49 | mriedem | remind me to never run live migration if i'm in ops | |
| 20:18:51 | superdan | if libvirt told us it was done, then there should be disks on the other side too | |
| 20:19:00 | mriedem | yeah hopefully | |
| 20:19:21 | superdan | it's scary how many people depend on live migration | |
| 20:24:45 | coffee_cat | lyarwood: are you around? | |
| 20:27:18 | coffee_cat | or anyone who could/would answer stupid BDM questions | |
| 20:27:31 | mriedem | i can answer stupid questions | |
| 20:27:44 | mriedem | coffee_cat: are you in budapest right now? | |
| 20:27:48 | mriedem | or the US? | |
| 20:27:52 | openstackgerrit | Eric Fried proposed openstack/nova master: PowerVM Driver: spawn/destroy #4: full flavor https://review.openstack.org/391288 | |
| 20:28:05 | coffee_cat | mriedem: in the US until tomorrow evening | |
| 20:28:11 | mriedem | oh, lee is in the UK | |
| 20:28:17 | mriedem | so he's in a pub somewhere | |
| 20:28:18 | coffee_cat | mriedem: will fly back next Sunday though... | |
| 20:28:31 | coffee_cat | mriedem: ah, I forgot | |
| 20:29:13 | coffee_cat | mriedem: anyhow, I'm sure you can help me out here to understand what's going on :) | |
| 20:29:22 | coffee_cat | mriedem: so his comments are here: https://review.openstack.org/#/c/456971/2/nova/compute/manager.py | |
| 20:30:33 | coffee_cat | mriedem: and I started to wonder whether the BDM is host or instance specific? if this question even makes sense... | |
| 20:31:40 | mriedem | the bdm is instance + volume specific | |
| 20:31:50 | mriedem | so we get the bdm for the instance and existing (old) volume on L5011 | |
| 20:33:32 | mriedem | the connector is specific to the host | |
| 20:33:37 | mriedem | well, host + instance | |
| 20:35:23 | mriedem | so in the old way, after we swap the volume, we terminate the connection for the old volume, | |
| 20:36:02 | mriedem | and then update the bdm with the new connection_info for the new volume | |
| 20:41:23 | mriedem | coffee_cat: left comments inline | |
| 20:41:32 | mriedem | coffee_cat: this was also called out in john's spec i believe | |
| 20:41:52 | mriedem | https://specs.openstack.org/openstack/nova-specs/specs/pike/approved/cinder-new-attach-apis.html#swap-volume | |
| 20:42:26 | coffee_cat | mriedem: ok, cool, makes sense, so basically need to create a new attachment for the new volume | |
| 20:42:46 | coffee_cat | and we might not be able to consolidate the workflow that much, but oh well | |
| 20:42:52 | mriedem | yes | |
| 20:43:07 | mriedem | where we used to call initialize_connection for the new volume, now we'll need to create a new attachment | |
| 20:43:08 | mriedem | for the new volume | |