| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-04-21 | |||
| 20:09:29 | superdan | mriedem: doesn't look like it does to me | |
| 20:10:34 | superdan | mriedem: so if we fail, have not shared path, we'll do_cleanup=True and tell the driver to run cleanup right? | |
| 20:11:12 | mriedem | looks that way | |
| 20:11:13 | superdan | and libvirt goes right into deleting stuff | |
| 20:11:34 | mriedem | even though you should be rolling back | |
| 20:11:39 | mriedem | but this isn't where rollback is called from | |
| 20:12:20 | superdan | right, this is if post-at-destination fails, which presumably could fail for some mundane reason and then we eff things up by doing the cleanup | |
| 20:12:57 | mriedem | is live_migration started from the source or dest host? | |
| 20:13:02 | superdan | source | |
| 20:13:13 | mriedem | oh right it has the dest parameter | |
| 20:13:37 | mriedem | i lose track of the musical chairs game that is live migration | |
| 20:13:47 | superdan | yeah it's confusing | |
| 20:14:37 | mriedem | so the source side though it was done, and calls _post_live_migration (source compute stiffl) | |
| 20:14:39 | mriedem | *still | |
| 20:15:10 | mriedem | which does an rpc call to post_live_migration_at_destination on the dest host | |
| 20:15:20 | superdan | oh, we don't get here unless it actually migrated | |
| 20:15:33 | mriedem | that fails becuase the guest isn't found on the dest host | |
| 20:15:34 | superdan | and then we just failed in the post call, but whatever, we don't own it anymore | |
| 20:16:43 | mriedem | yeah post_live_migration_at_destination fails and returns the failure back through rpc | |
| 20:17:00 | mriedem | we get the failure, log the trace "Post live migration at destination %s failed" | |
| 20:17:03 | superdan | right, so I guess it's okay | |
| 20:17:12 | superdan | we don't get to this post method unless libvirt said it worked | |
| 20:17:12 | mriedem | then drop through and start cleaning up the source | |
| 20:17:16 | superdan | so we can clean up regardless | |
| 20:17:17 | mriedem | meanwhile, the guest is not on the dest either | |
| 20:17:20 | mriedem | yeah | |
| 20:17:25 | superdan | yeah, but nothing we can do about it | |
| 20:17:26 | mriedem | so basically, | |
| 20:17:30 | mriedem | your sh is f'ed | |
| 20:17:34 | mriedem | and you instance is gone | |
| 20:17:40 | mriedem | *your | |
| 20:17:45 | superdan | yeah, I just wanted to make sure we didn't delete the thing if migration failed | |
| 20:17:50 | superdan | that's a double kick in the pands | |
| 20:18:03 | mriedem | meanwhile we've deleted your disks | |
| 20:18:19 | mriedem | this is really like, well, it's all busted, oh and btw we deleted your instance | |
| 20:18:20 | mriedem | you're welcome | |
| 20:18:28 | superdan | whatever, libvirt told us to :) | |
| 20:18:37 | mriedem | wow | |
| 20:18:49 | mriedem | remind me to never run live migration if i'm in ops | |
| 20:18:51 | superdan | if libvirt told us it was done, then there should be disks on the other side too | |
| 20:19:00 | mriedem | yeah hopefully | |
| 20:19:21 | superdan | it's scary how many people depend on live migration | |
| 20:24:45 | coffee_cat | lyarwood: are you around? | |
| 20:27:18 | coffee_cat | or anyone who could/would answer stupid BDM questions | |
| 20:27:31 | mriedem | i can answer stupid questions | |
| 20:27:44 | mriedem | coffee_cat: are you in budapest right now? | |
| 20:27:48 | mriedem | or the US? | |
| 20:27:52 | openstackgerrit | Eric Fried proposed openstack/nova master: PowerVM Driver: spawn/destroy #4: full flavor https://review.openstack.org/391288 | |
| 20:28:05 | coffee_cat | mriedem: in the US until tomorrow evening | |
| 20:28:11 | mriedem | oh, lee is in the UK | |
| 20:28:17 | mriedem | so he's in a pub somewhere | |
| 20:28:18 | coffee_cat | mriedem: will fly back next Sunday though... | |
| 20:28:31 | coffee_cat | mriedem: ah, I forgot | |
| 20:29:13 | coffee_cat | mriedem: anyhow, I'm sure you can help me out here to understand what's going on :) | |
| 20:29:22 | coffee_cat | mriedem: so his comments are here: https://review.openstack.org/#/c/456971/2/nova/compute/manager.py | |
| 20:30:33 | coffee_cat | mriedem: and I started to wonder whether the BDM is host or instance specific? if this question even makes sense... | |
| 20:31:40 | mriedem | the bdm is instance + volume specific | |
| 20:31:50 | mriedem | so we get the bdm for the instance and existing (old) volume on L5011 | |
| 20:33:32 | mriedem | the connector is specific to the host | |
| 20:33:37 | mriedem | well, host + instance | |
| 20:35:23 | mriedem | so in the old way, after we swap the volume, we terminate the connection for the old volume, | |
| 20:36:02 | mriedem | and then update the bdm with the new connection_info for the new volume | |
| 20:41:23 | mriedem | coffee_cat: left comments inline | |
| 20:41:32 | mriedem | coffee_cat: this was also called out in john's spec i believe | |
| 20:41:52 | mriedem | https://specs.openstack.org/openstack/nova-specs/specs/pike/approved/cinder-new-attach-apis.html#swap-volume | |
| 20:42:26 | coffee_cat | mriedem: ok, cool, makes sense, so basically need to create a new attachment for the new volume | |
| 20:42:46 | coffee_cat | and we might not be able to consolidate the workflow that much, but oh well | |
| 20:42:52 | mriedem | yes | |
| 20:43:07 | mriedem | where we used to call initialize_connection for the new volume, now we'll need to create a new attachment | |
| 20:43:08 | mriedem | for the new volume | |
| 20:43:28 | coffee_cat | ok, that sounds easily doable | |
| 20:43:32 | mriedem | if everything is ok, we delete the old attachment for the old volume and update the bdm with the new attachment_id | |
| 20:44:22 | mriedem | john's spec is talking about doing things in the api too, which i'm not really following | |
| 20:44:36 | mriedem | (api) create new attachment for the volume uuid-new, fail API call if we can’t create that attachment | |
| 20:44:52 | mriedem | ^ is what i was saying we could do on the compute where initialize_connection is called for the new volume | |
| 20:45:03 | mriedem | we could create the attachment for the new volume in the api, but we don't have a bdm for that | |
| 20:45:18 | mriedem | so not sure how we get that in the compute, unless we lookup the new volume details and get it's attachment list | |
| 20:45:21 | coffee_cat | yeah, BDM is problematic :/ | |
| 20:46:25 | mriedem | maybe he just meant create the new attachment in the api so we can fail fast | |
| 20:46:39 | mriedem | but, | |
| 20:46:42 | coffee_cat | probably as that was a topic many times | |
| 20:46:59 | mriedem | we'd only do that if we knew the old attached volume was created with the new flow i htink | |
| 20:47:01 | mriedem | *think | |
| 20:47:05 | mriedem | i'm not sure if that matters or not | |
| 20:47:08 | mriedem | but it probably does | |
| 20:47:33 | coffee_cat | so the volume is always created the same way | |
| 20:47:58 | mriedem | this is all new flow stuff | |
| 20:47:59 | mriedem | "In this section, we talk about what happens should the volume being swapped have the attachment_id present in the BDM, and as such we follow the new flow." | |
| 20:48:25 | mriedem | yeah, so if the bdm.attachment_id is set, we key off that for all of the swap volume goodies outlined in the spec | |
| 20:48:31 | mriedem | like creating a new attachment for the new volume, etc | |
| 20:48:36 | coffee_cat | and for the new volume we can create that with the new flow | |
| 20:49:01 | coffee_cat | if the API is available | |
| 20:49:11 | mriedem | well i think the point is, you can't swap from a volume attachd the old way, to a volume attached the new way | |
| 20:49:28 | mriedem | you have to swap from attached new way to attached new way | |
| 20:49:30 | mriedem | if that makes sense | |
| 20:49:42 | mriedem | i don't know if technically it makes a difference on the cinder side | |
| 20:49:46 | coffee_cat | I don't really see why not, but as we go by attachment_id in the BDM anyhow it pretty much doesn't matter | |
| 20:49:49 | sdague | mriedem: it would be less terrible if you used the fixture that actually captures the log stream | |
| 20:50:22 | smcginnis | mriedem: Should be OK on the Cinder side as long as each volume has its attachment info. | |
| 20:50:22 | mriedem | coffee_cat: the reason john and i said to do it this way in the spec is so nothing else in the code does anything using the new apis until we've plumbed in support for attaching volumes the new way | |