| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-04-13 | |||
| 16:46:59 | owalsh | sfinucan: cool, thanks | |
| 16:47:10 | dansmith | mriedem: must be missing something | |
| 16:50:06 | mriedem | i honestly hope i am missing something | |
| 16:52:03 | johnthetubaguy | melwitt: oh, interesting, once its offloaded I kinda assumed it wasn't going back there, in all likelylood | |
| 16:53:50 | openstackgerrit | Jianghua Wang proposed openstack/nova-specs master: Support VGPU https://review.openstack.org/450122 | |
| 16:57:11 | dansmith | melwitt: yeah, once offloaded it's not supposed to (have to) go back to the same host | |
| 16:57:15 | dansmith | that's the whole point | |
| 16:57:27 | melwitt | yeah | |
| 16:57:32 | melwitt | I do see it cleaning up network here https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L4378 | |
| 16:57:39 | melwitt | so maybe someone just forgot about volumes? | |
| 16:59:16 | openstackgerrit | Rajath Agasthya proposed openstack/nova master: Fix Bandit jinja2 warnings https://review.openstack.org/456709 | |
| 16:59:21 | mriedem | melwitt: that's a noop for neutron | |
| 16:59:36 | melwitt | I would not have expected that | |
| 16:59:49 | mriedem | https://github.com/openstack/nova/blob/master/nova/network/neutronv2/api.py#L2332 | |
| 17:00:03 | mriedem | melwitt: there are like 5 different methods in the base network api like that | |
| 17:00:19 | mriedem | melwitt: and of course some are only neutron and some are only nova-net | |
| 17:00:20 | melwitt | guh, what the hell | |
| 17:00:54 | mriedem | https://github.com/openstack/nova/blob/master/nova/network/api.py#L529 | |
| 17:01:52 | melwitt | sigh | |
| 17:10:33 | swebster | ^ Interesting that the neutron cleanup doesn't start a migration. This actually causes you to be completely screwed when unshelving with PCI devices. I fixed a symptom of that with https://review.openstack.org/#/c/453938 | |
| 17:14:09 | mriedem | hmm, yeah, so we don't unbind and we don't terminate volume connections on shelve offload, | |
| 17:14:16 | mriedem | so unshelve is just.... | |
| 17:14:19 | mriedem | roll the dice | |
| 17:14:40 | mriedem | plus on unshelve i believe we'll also create a new volume, but would have to check that | |
| 17:14:45 | mriedem | if it's bfv | |
| 17:17:28 | openstackgerrit | Steven Webster proposed openstack/nova master: Fix mitaka online migration for PCI devices https://review.openstack.org/456397 | |
| 17:18:21 | mriedem | yeah so on unshelve, we prep block devices but pass do_check_attach=False, because the volume is still in-use b/c it's still attached to the instance, by design, | |
| 17:18:29 | mriedem | we make a new connection on the new host, | |
| 17:18:37 | mriedem | and if volume['attach_status'] == "detached": is False, | |
| 17:18:42 | mriedem | so we don't try to attach again | |
| 17:19:43 | melwitt | how can it stay attached if the virt domain was destroyed? | |
| 17:20:49 | mriedem | yeah good point, let me see what driver.destroy() does | |
| 17:21:24 | mriedem | that calls cleanup() which will by default unplug vifs | |
| 17:21:49 | mriedem | we get the bdms and disconnect the volumes | |
| 17:22:07 | mriedem | we never call terminate_connection in cinder | |
| 17:22:11 | mriedem | but maybe that doesn't matter? | |
| 17:22:28 | mriedem | smcginnis: hemna: what does terminate_connection actually do on the cinder side? | |
| 17:26:17 | mriedem | melwitt: also, when we unplug the vifs during cleanup() that should trigger a network-changed event from the neutron side, but since the instance is now shelved offloaded, the server-external-events api can't rpc cast into the compute to refresh it's nw info cache | |
| 17:28:25 | melwitt | O.o | |
| 17:38:31 | hemna | mriedem, it's job is to tear down the external connection from the backend to the host | |
| 17:38:44 | hemna | effectively unexport the volume | |
| 17:40:32 | peter-hamilton | jaypipes: i'll be afk for a bit; if you have any questions on cert validation, i'll answer once i get back | |
| 17:57:21 | mriedem | hemna: so seems like that would be important when an instance with an attached volume disconnects from it's host and eventually shows up elsewhere (another compute host) yeah? | |
| 17:57:25 | mriedem | or does it depend on the backend? | |
| 17:57:48 | hemna | not sure I'm following | |
| 17:58:04 | mriedem | hemna: are you familiar with the shelve offload operation in nova? | |
| 17:58:20 | hemna | not imtimately | |
| 17:58:29 | mriedem | let me introduce you | |
| 17:58:33 | hemna | :P ok | |
| 17:58:39 | mriedem | basically keep your volume attached in the cinder db, | |
| 17:58:45 | mriedem | but we destroy the guest vm from the compute host | |
| 17:58:51 | mriedem | keep ports and volumes attached in dbs | |
| 17:59:02 | mriedem | when we destroy the guest, we disconnect any volumes | |
| 17:59:16 | mriedem | then the instance is in this shelve-offloaded state, where it doesn't have a host | |
| 17:59:24 | mriedem | (we also take a snapshot of it before offloading it) | |
| 17:59:27 | hemna | ok | |
| 17:59:33 | hemna | so it's like a dormant vm | |
| 17:59:36 | mriedem | when you unshelve the instance, we take the snapshot image and boot the server on a new host | |
| 17:59:43 | mriedem | sort of yeah | |
| 17:59:50 | mriedem | it might land on the same host as before, but it might not | |
| 18:00:25 | hemna | yah, so I believe terminate_connection should be called then in this case, to ensure that the backend disconnects from the host | |
| 18:00:35 | mriedem | when the instance is created on a new host, we call initialize_connection again | |
| 18:00:35 | dane-fichter | jaypipes: also I can answer most cert validation questions while peter-hamilton is afk | |
| 18:00:43 | hemna | makes sense | |
| 18:00:51 | openstackgerrit | Mathieu Gagné proposed openstack/nova-specs master: Nova support for attached volume extend https://review.openstack.org/453272 | |
| 18:01:58 | mriedem | hemna: the funny thing is on unshelve, | |
| 18:02:00 | jgriffith | mriedem hemna there is no terminate_connection, only “attachment_delete” or “update" | |
| 18:02:02 | jgriffith | :) | |
| 18:02:07 | hemna | :P | |
| 18:02:09 | mriedem | is i don't think we ever call connect_volume in brick on the new host | |
| 18:02:19 | hemna | ! | |
| 18:02:22 | mriedem | so even though we initialize the connection, we don't connect the volume | |
| 18:02:26 | mriedem | as far as i can tell | |
| 18:02:28 | hemna | oops | |
| 18:02:48 | mriedem | let me check the driver.spawn() method though | |
| 18:02:51 | mriedem | maybe it's done there | |
| 18:02:54 | mriedem | some of this garbage is hidden | |
| 18:03:09 | hemna | well if the volume is showing up there, then it has to get called at some point | |
| 18:03:21 | mriedem | i don't know if it does, this is all via code inspection, | |
| 18:03:26 | mriedem | i don't have a devstack handy to recreate | |
| 18:04:18 | mriedem | yeah i don't see brick called during spawn during unshelve | |
| 18:08:52 | mriedem | ha https://bugs.launchpad.net/nova/+bug/1547142 | |
| 18:08:52 | openstack | Launchpad bug 1547142 in OpenStack Compute (nova) "A shelved_offload VM's volumes are still attached to a host" [Medium,Confirmed] | |
| 18:09:30 | mriedem | the fix on the nova side for this was blocked by some changes on the cinder side | |
| 18:09:31 | mriedem | of course | |
| 18:10:35 | openstackgerrit | Dan Smith proposed openstack/nova master: Fix hypervisors api missing HostMappingNotFound handlers https://review.openstack.org/456227 | |
| 18:10:40 | hemna | oh missing the remove_export | |
| 18:12:21 | hemna | well It's backend dependent really | |
| 18:12:21 | hemna | terminate_connection will do it for most of the cinder backends | |
| 18:14:49 | hemna | lvm of course uses remove_export to do the real work, and does nothing in terminate_connection | |
| 18:14:56 | mriedem | hemna: that's what the guy on the bug originally asked, can't we at least call terminate connection | |
| 18:15:06 | mriedem | does terminate_connection call remove_export? | |
| 18:15:07 | hemna | you can | |
| 18:15:17 | hemna | but in the case of lvm tgt, terminate_connection does nothing | |
| 18:15:19 | hemna | nope | |
| 18:15:29 | hemna | the volume manager calls both of those | |
| 18:15:41 | mriedem | so what calls remove_export for lvm tgt? detach? | |
| 18:16:21 | hemna | volume manager's remove_export does | |
| 18:16:28 | hemna | and some other edge cases | |
| 18:16:42 | mriedem | yeah but what calls that from the api entry point? | |