| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-04-13 | |||
| 17:14:19 | mriedem | roll the dice | |
| 17:14:40 | mriedem | plus on unshelve i believe we'll also create a new volume, but would have to check that | |
| 17:14:45 | mriedem | if it's bfv | |
| 17:17:28 | openstackgerrit | Steven Webster proposed openstack/nova master: Fix mitaka online migration for PCI devices https://review.openstack.org/456397 | |
| 17:18:21 | mriedem | yeah so on unshelve, we prep block devices but pass do_check_attach=False, because the volume is still in-use b/c it's still attached to the instance, by design, | |
| 17:18:29 | mriedem | we make a new connection on the new host, | |
| 17:18:37 | mriedem | and if volume['attach_status'] == "detached": is False, | |
| 17:18:42 | mriedem | so we don't try to attach again | |
| 17:19:43 | melwitt | how can it stay attached if the virt domain was destroyed? | |
| 17:20:49 | mriedem | yeah good point, let me see what driver.destroy() does | |
| 17:21:24 | mriedem | that calls cleanup() which will by default unplug vifs | |
| 17:21:49 | mriedem | we get the bdms and disconnect the volumes | |
| 17:22:07 | mriedem | we never call terminate_connection in cinder | |
| 17:22:11 | mriedem | but maybe that doesn't matter? | |
| 17:22:28 | mriedem | smcginnis: hemna: what does terminate_connection actually do on the cinder side? | |
| 17:26:17 | mriedem | melwitt: also, when we unplug the vifs during cleanup() that should trigger a network-changed event from the neutron side, but since the instance is now shelved offloaded, the server-external-events api can't rpc cast into the compute to refresh it's nw info cache | |
| 17:28:25 | melwitt | O.o | |
| 17:38:31 | hemna | mriedem, it's job is to tear down the external connection from the backend to the host | |
| 17:38:44 | hemna | effectively unexport the volume | |
| 17:40:32 | peter-hamilton | jaypipes: i'll be afk for a bit; if you have any questions on cert validation, i'll answer once i get back | |
| 17:57:21 | mriedem | hemna: so seems like that would be important when an instance with an attached volume disconnects from it's host and eventually shows up elsewhere (another compute host) yeah? | |
| 17:57:25 | mriedem | or does it depend on the backend? | |
| 17:57:48 | hemna | not sure I'm following | |
| 17:58:04 | mriedem | hemna: are you familiar with the shelve offload operation in nova? | |
| 17:58:20 | hemna | not imtimately | |
| 17:58:29 | mriedem | let me introduce you | |
| 17:58:33 | hemna | :P ok | |
| 17:58:39 | mriedem | basically keep your volume attached in the cinder db, | |
| 17:58:45 | mriedem | but we destroy the guest vm from the compute host | |
| 17:58:51 | mriedem | keep ports and volumes attached in dbs | |
| 17:59:02 | mriedem | when we destroy the guest, we disconnect any volumes | |
| 17:59:16 | mriedem | then the instance is in this shelve-offloaded state, where it doesn't have a host | |
| 17:59:24 | mriedem | (we also take a snapshot of it before offloading it) | |
| 17:59:27 | hemna | ok | |
| 17:59:33 | hemna | so it's like a dormant vm | |
| 17:59:36 | mriedem | when you unshelve the instance, we take the snapshot image and boot the server on a new host | |
| 17:59:43 | mriedem | sort of yeah | |
| 17:59:50 | mriedem | it might land on the same host as before, but it might not | |
| 18:00:25 | hemna | yah, so I believe terminate_connection should be called then in this case, to ensure that the backend disconnects from the host | |
| 18:00:35 | dane-fichter | jaypipes: also I can answer most cert validation questions while peter-hamilton is afk | |
| 18:00:35 | mriedem | when the instance is created on a new host, we call initialize_connection again | |
| 18:00:43 | hemna | makes sense | |
| 18:00:51 | openstackgerrit | Mathieu Gagné proposed openstack/nova-specs master: Nova support for attached volume extend https://review.openstack.org/453272 | |
| 18:01:58 | mriedem | hemna: the funny thing is on unshelve, | |
| 18:02:00 | jgriffith | mriedem hemna there is no terminate_connection, only “attachment_delete” or “update" | |
| 18:02:02 | jgriffith | :) | |
| 18:02:07 | hemna | :P | |
| 18:02:09 | mriedem | is i don't think we ever call connect_volume in brick on the new host | |
| 18:02:19 | hemna | ! | |
| 18:02:22 | mriedem | so even though we initialize the connection, we don't connect the volume | |
| 18:02:26 | mriedem | as far as i can tell | |
| 18:02:28 | hemna | oops | |
| 18:02:48 | mriedem | let me check the driver.spawn() method though | |
| 18:02:51 | mriedem | maybe it's done there | |
| 18:02:54 | mriedem | some of this garbage is hidden | |
| 18:03:09 | hemna | well if the volume is showing up there, then it has to get called at some point | |
| 18:03:21 | mriedem | i don't know if it does, this is all via code inspection, | |
| 18:03:26 | mriedem | i don't have a devstack handy to recreate | |
| 18:04:18 | mriedem | yeah i don't see brick called during spawn during unshelve | |
| 18:08:52 | openstack | Launchpad bug 1547142 in OpenStack Compute (nova) "A shelved_offload VM's volumes are still attached to a host" [Medium,Confirmed] | |
| 18:08:52 | mriedem | ha https://bugs.launchpad.net/nova/+bug/1547142 | |
| 18:09:30 | mriedem | the fix on the nova side for this was blocked by some changes on the cinder side | |
| 18:09:31 | mriedem | of course | |
| 18:10:35 | openstackgerrit | Dan Smith proposed openstack/nova master: Fix hypervisors api missing HostMappingNotFound handlers https://review.openstack.org/456227 | |
| 18:10:40 | hemna | oh missing the remove_export | |
| 18:12:21 | hemna | terminate_connection will do it for most of the cinder backends | |
| 18:12:21 | hemna | well It's backend dependent really | |
| 18:14:49 | hemna | lvm of course uses remove_export to do the real work, and does nothing in terminate_connection | |
| 18:14:56 | mriedem | hemna: that's what the guy on the bug originally asked, can't we at least call terminate connection | |
| 18:15:06 | mriedem | does terminate_connection call remove_export? | |
| 18:15:07 | hemna | you can | |
| 18:15:17 | hemna | but in the case of lvm tgt, terminate_connection does nothing | |
| 18:15:19 | hemna | nope | |
| 18:15:29 | hemna | the volume manager calls both of those | |
| 18:15:41 | mriedem | so what calls remove_export for lvm tgt? detach? | |
| 18:16:21 | hemna | volume manager's remove_export does | |
| 18:16:28 | hemna | and some other edge cases | |
| 18:16:42 | mriedem | yeah but what calls that from the api entry point? | |
| 18:16:48 | hemna | looking | |
| 18:16:50 | mriedem | detach? delete volume? | |
| 18:16:51 | mriedem | ok | |
| 18:17:33 | hemna | I don't see anything in the api that calls it..... | |
| 18:19:45 | mriedem | btw, i looked up shelved + volumes in nova bugs, | |
| 18:19:49 | mriedem | what a treasure trove of broken | |
| 18:19:57 | hemna | detach_volume calls it | |
| 18:20:06 | mriedem | hemna: ah ok, | |
| 18:20:13 | mriedem | and we don't call detach during shelve offload | |
| 18:20:20 | mriedem | because we want the volume to logically stay attached to the instance | |
| 18:20:25 | hemna | well detach_volume calls the driver's remove_export at least | |
| 18:21:19 | hemna | jgriffith, does the new API account for being able to unexport the volume w/o marking it as unattached ? | |
| 18:21:41 | hemna | or we'll have to add a new Cinder API entry point for remove export | |
| 18:22:13 | hemna | or...does it make sense that terminate_connection in the volume manager would automatically call remove_export too | |
| 18:22:17 | hemna | ... | |
| 18:23:13 | openstackgerrit | Daniel Pawlik proposed openstack/nova master: Changed network_info variable in test_create_configdrive https://review.openstack.org/436953 | |
| 18:23:19 | hemna | honestly I never understood why there was terminate_connection vs remove_export, and not just a single merged capability | |
| 18:23:21 | openstack | Launchpad bug 1554440 in OpenStack Compute (nova) "Show volume attachment exception does not looks good if server is in shelved_offloaded state" [Medium,Confirmed] | |
| 18:23:21 | mriedem | gmann: i just found https://bugs.launchpad.net/nova/+bug/1554440 and i don't agree that it requires a microversion to fix a regression introduced with 2.20 | |
| 18:23:27 | mriedem | hemna: yeah that would be nicer | |
| 18:23:47 | mriedem | oomichi: jaypipes: why do you think we need a microversion bump to fix this bug? https://review.openstack.org/#/c/290196/ | |
| 18:23:53 | mriedem | oomichi: jaypipes: it's clearly a regression in 2.20 | |