Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-04-13
16:45:37 melwitt it assumes it will go back on the same host, but I don't know if it does or can do anything to guarantee that. with the network part, I could see it wanting to keep the IP allocated so that it can't be given out (and network possibly run out of IPs) when the person comes back to unshelve. same concept as keeping the quota occupied during offload
16:46:59 owalsh sfinucan: cool, thanks
16:47:10 dansmith mriedem: must be missing something
16:50:06 mriedem i honestly hope i am missing something
16:52:03 johnthetubaguy melwitt: oh, interesting, once its offloaded I kinda assumed it wasn't going back there, in all likelylood
16:53:50 openstackgerrit Jianghua Wang proposed openstack/nova-specs master: Support VGPU https://review.openstack.org/450122
16:57:11 dansmith melwitt: yeah, once offloaded it's not supposed to (have to) go back to the same host
16:57:15 dansmith that's the whole point
16:57:27 melwitt yeah
16:57:32 melwitt I do see it cleaning up network here https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L4378
16:57:39 melwitt so maybe someone just forgot about volumes?
16:59:16 openstackgerrit Rajath Agasthya proposed openstack/nova master: Fix Bandit jinja2 warnings https://review.openstack.org/456709
16:59:21 mriedem melwitt: that's a noop for neutron
16:59:36 melwitt I would not have expected that
16:59:49 mriedem https://github.com/openstack/nova/blob/master/nova/network/neutronv2/api.py#L2332
17:00:03 mriedem melwitt: there are like 5 different methods in the base network api like that
17:00:19 mriedem melwitt: and of course some are only neutron and some are only nova-net
17:00:20 melwitt guh, what the hell
17:00:54 mriedem https://github.com/openstack/nova/blob/master/nova/network/api.py#L529
17:01:52 melwitt sigh
17:10:33 swebster ^ Interesting that the neutron cleanup doesn't start a migration. This actually causes you to be completely screwed when unshelving with PCI devices. I fixed a symptom of that with https://review.openstack.org/#/c/453938
17:14:09 mriedem hmm, yeah, so we don't unbind and we don't terminate volume connections on shelve offload,
17:14:16 mriedem so unshelve is just....
17:14:19 mriedem roll the dice
17:14:40 mriedem plus on unshelve i believe we'll also create a new volume, but would have to check that
17:14:45 mriedem if it's bfv
17:17:28 openstackgerrit Steven Webster proposed openstack/nova master: Fix mitaka online migration for PCI devices https://review.openstack.org/456397
17:18:21 mriedem yeah so on unshelve, we prep block devices but pass do_check_attach=False, because the volume is still in-use b/c it's still attached to the instance, by design,
17:18:29 mriedem we make a new connection on the new host,
17:18:37 mriedem and if volume['attach_status'] == "detached": is False,
17:18:42 mriedem so we don't try to attach again
17:19:43 melwitt how can it stay attached if the virt domain was destroyed?
17:20:49 mriedem yeah good point, let me see what driver.destroy() does
17:21:24 mriedem that calls cleanup() which will by default unplug vifs
17:21:49 mriedem we get the bdms and disconnect the volumes
17:22:07 mriedem we never call terminate_connection in cinder
17:22:11 mriedem but maybe that doesn't matter?
17:22:28 mriedem smcginnis: hemna: what does terminate_connection actually do on the cinder side?
17:26:17 mriedem melwitt: also, when we unplug the vifs during cleanup() that should trigger a network-changed event from the neutron side, but since the instance is now shelved offloaded, the server-external-events api can't rpc cast into the compute to refresh it's nw info cache
17:28:25 melwitt O.o
17:38:31 hemna mriedem, it's job is to tear down the external connection from the backend to the host
17:38:44 hemna effectively unexport the volume
17:40:32 peter-hamilton jaypipes: i'll be afk for a bit; if you have any questions on cert validation, i'll answer once i get back
17:57:21 mriedem hemna: so seems like that would be important when an instance with an attached volume disconnects from it's host and eventually shows up elsewhere (another compute host) yeah?
17:57:25 mriedem or does it depend on the backend?
17:57:48 hemna not sure I'm following
17:58:04 mriedem hemna: are you familiar with the shelve offload operation in nova?
17:58:20 hemna not imtimately
17:58:29 mriedem let me introduce you
17:58:33 hemna :P ok
17:58:39 mriedem basically keep your volume attached in the cinder db,
17:58:45 mriedem but we destroy the guest vm from the compute host
17:58:51 mriedem keep ports and volumes attached in dbs
17:59:02 mriedem when we destroy the guest, we disconnect any volumes
17:59:16 mriedem then the instance is in this shelve-offloaded state, where it doesn't have a host
17:59:24 mriedem (we also take a snapshot of it before offloading it)
17:59:27 hemna ok
17:59:33 hemna so it's like a dormant vm
17:59:36 mriedem when you unshelve the instance, we take the snapshot image and boot the server on a new host
17:59:43 mriedem sort of yeah
17:59:50 mriedem it might land on the same host as before, but it might not
18:00:25 hemna yah, so I believe terminate_connection should be called then in this case, to ensure that the backend disconnects from the host
18:00:35 dane-fichter jaypipes: also I can answer most cert validation questions while peter-hamilton is afk
18:00:35 mriedem when the instance is created on a new host, we call initialize_connection again
18:00:43 hemna makes sense
18:00:51 openstackgerrit Mathieu Gagné proposed openstack/nova-specs master: Nova support for attached volume extend https://review.openstack.org/453272
18:01:58 mriedem hemna: the funny thing is on unshelve,
18:02:00 jgriffith mriedem hemna there is no terminate_connection, only “attachment_delete” or “update"
18:02:02 jgriffith :)
18:02:07 hemna :P
18:02:09 mriedem is i don't think we ever call connect_volume in brick on the new host
18:02:19 hemna !
18:02:22 mriedem so even though we initialize the connection, we don't connect the volume
18:02:26 mriedem as far as i can tell
18:02:28 hemna oops
18:02:48 mriedem let me check the driver.spawn() method though
18:02:51 mriedem maybe it's done there
18:02:54 mriedem some of this garbage is hidden
18:03:09 hemna well if the volume is showing up there, then it has to get called at some point
18:03:21 mriedem i don't know if it does, this is all via code inspection,
18:03:26 mriedem i don't have a devstack handy to recreate
18:04:18 mriedem yeah i don't see brick called during spawn during unshelve
18:08:52 openstack Launchpad bug 1547142 in OpenStack Compute (nova) "A shelved_offload VM's volumes are still attached to a host" [Medium,Confirmed]
18:08:52 mriedem ha https://bugs.launchpad.net/nova/+bug/1547142
18:09:30 mriedem the fix on the nova side for this was blocked by some changes on the cinder side
18:09:31 mriedem of course
18:10:35 openstackgerrit Dan Smith proposed openstack/nova master: Fix hypervisors api missing HostMappingNotFound handlers https://review.openstack.org/456227
18:10:40 hemna oh missing the remove_export
18:12:21 hemna terminate_connection will do it for most of the cinder backends
18:12:21 hemna well It's backend dependent really
18:14:49 hemna lvm of course uses remove_export to do the real work, and does nothing in terminate_connection
18:14:56 mriedem hemna: that's what the guy on the bug originally asked, can't we at least call terminate connection
18:15:06 mriedem does terminate_connection call remove_export?
18:15:07 hemna you can
18:15:17 hemna but in the case of lvm tgt, terminate_connection does nothing
18:15:19 hemna nope
18:15:29 hemna the volume manager calls both of those
18:15:41 mriedem so what calls remove_export for lvm tgt? detach?
18:16:21 hemna volume manager's remove_export does
18:16:28 hemna and some other edge cases

Earlier   Later