| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-04-13 | |||
| 16:36:49 | mriedem | like, the most latent bug ever... | |
| 16:37:00 | dansmith | I think there might've been some things that were left in place on the off chance it landed on the same host, but can't imagine that's right | |
| 16:37:00 | mriedem | i keep thinking i'm missing something obvious | |
| 16:37:11 | dansmith | unless we call to the old compute somehow, but we've lost that info I think | |
| 16:37:23 | dansmith | yeah, seems too obviously broken | |
| 16:37:52 | mriedem | conductor manager for unshelve calls out to scheduler to get new hosts | |
| 16:37:55 | openstackgerrit | Jianghua Wang proposed openstack/nova-specs master: Support VGPU https://review.openstack.org/450122 | |
| 16:38:03 | mriedem | plus, you could have unshelved from a host that is now down or gone | |
| 16:38:11 | mriedem | *shelved | |
| 16:38:17 | dansmith | yeah | |
| 16:41:40 | mriedem | oh btw, pike-1 milestone today | |
| 16:41:46 | mriedem | i had forgotten about the need for the tag | |
| 16:42:35 | jianghuaw_ | jaypipes, sbauza: could you help to look at this VGPU spec? | |
| 16:42:36 | jianghuaw_ | https://review.openstack.org/450122 | |
| 16:42:37 | melwitt_ | ngupta: I found this old spec about it https://review.openstack.org/#/c/239798/ and the consensus there mentions the way forward for rbd encrypted volumes should be with qemu native rbd encryption, but I don't think anyone worked on it yet | |
| 16:43:11 | mriedem | dansmith: also looks like we don't unbind ports from the host on shelve offload | |
| 16:43:20 | mriedem | this is all just very odd | |
| 16:43:59 | openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP! conf: Add three new '[libvirt] live_migration_*' options https://review.openstack.org/456571 | |
| 16:44:00 | openstackgerrit | Stephen Finucane proposed openstack/nova master: conf: Gather 'live_migration_scheme', 'live_migration_inbound_addr' https://review.openstack.org/456572 | |
| 16:44:00 | melwitt | mriedem: yeah, that's "intentional"? I noticed it just leaves everything there as-is, network connection, volume connection, etc | |
| 16:44:00 | openstackgerrit | Stephen Finucane proposed openstack/nova master: conf: Convert 'live_migration_inbound_addr' to HostAddressOpt https://review.openstack.org/456573 | |
| 16:44:45 | sfinucan | owalsh: I've to go to tag, so that's my WIP patches. Lemme know if you think that's the right direction. Will finish them tonight or Tuesday | |
| 16:45:37 | melwitt | it assumes it will go back on the same host, but I don't know if it does or can do anything to guarantee that. with the network part, I could see it wanting to keep the IP allocated so that it can't be given out (and network possibly run out of IPs) when the person comes back to unshelve. same concept as keeping the quota occupied during offload | |
| 16:46:59 | owalsh | sfinucan: cool, thanks | |
| 16:47:10 | dansmith | mriedem: must be missing something | |
| 16:50:06 | mriedem | i honestly hope i am missing something | |
| 16:52:03 | johnthetubaguy | melwitt: oh, interesting, once its offloaded I kinda assumed it wasn't going back there, in all likelylood | |
| 16:53:50 | openstackgerrit | Jianghua Wang proposed openstack/nova-specs master: Support VGPU https://review.openstack.org/450122 | |
| 16:57:11 | dansmith | melwitt: yeah, once offloaded it's not supposed to (have to) go back to the same host | |
| 16:57:15 | dansmith | that's the whole point | |
| 16:57:27 | melwitt | yeah | |
| 16:57:32 | melwitt | I do see it cleaning up network here https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L4378 | |
| 16:57:39 | melwitt | so maybe someone just forgot about volumes? | |
| 16:59:16 | openstackgerrit | Rajath Agasthya proposed openstack/nova master: Fix Bandit jinja2 warnings https://review.openstack.org/456709 | |
| 16:59:21 | mriedem | melwitt: that's a noop for neutron | |
| 16:59:36 | melwitt | I would not have expected that | |
| 16:59:49 | mriedem | https://github.com/openstack/nova/blob/master/nova/network/neutronv2/api.py#L2332 | |
| 17:00:03 | mriedem | melwitt: there are like 5 different methods in the base network api like that | |
| 17:00:19 | mriedem | melwitt: and of course some are only neutron and some are only nova-net | |
| 17:00:20 | melwitt | guh, what the hell | |
| 17:00:54 | mriedem | https://github.com/openstack/nova/blob/master/nova/network/api.py#L529 | |
| 17:01:52 | melwitt | sigh | |
| 17:10:33 | swebster | ^ Interesting that the neutron cleanup doesn't start a migration. This actually causes you to be completely screwed when unshelving with PCI devices. I fixed a symptom of that with https://review.openstack.org/#/c/453938 | |
| 17:14:09 | mriedem | hmm, yeah, so we don't unbind and we don't terminate volume connections on shelve offload, | |
| 17:14:16 | mriedem | so unshelve is just.... | |
| 17:14:19 | mriedem | roll the dice | |
| 17:14:40 | mriedem | plus on unshelve i believe we'll also create a new volume, but would have to check that | |
| 17:14:45 | mriedem | if it's bfv | |
| 17:17:28 | openstackgerrit | Steven Webster proposed openstack/nova master: Fix mitaka online migration for PCI devices https://review.openstack.org/456397 | |
| 17:18:21 | mriedem | yeah so on unshelve, we prep block devices but pass do_check_attach=False, because the volume is still in-use b/c it's still attached to the instance, by design, | |
| 17:18:29 | mriedem | we make a new connection on the new host, | |
| 17:18:37 | mriedem | and if volume['attach_status'] == "detached": is False, | |
| 17:18:42 | mriedem | so we don't try to attach again | |
| 17:19:43 | melwitt | how can it stay attached if the virt domain was destroyed? | |
| 17:20:49 | mriedem | yeah good point, let me see what driver.destroy() does | |
| 17:21:24 | mriedem | that calls cleanup() which will by default unplug vifs | |
| 17:21:49 | mriedem | we get the bdms and disconnect the volumes | |
| 17:22:07 | mriedem | we never call terminate_connection in cinder | |
| 17:22:11 | mriedem | but maybe that doesn't matter? | |
| 17:22:28 | mriedem | smcginnis: hemna: what does terminate_connection actually do on the cinder side? | |
| 17:26:17 | mriedem | melwitt: also, when we unplug the vifs during cleanup() that should trigger a network-changed event from the neutron side, but since the instance is now shelved offloaded, the server-external-events api can't rpc cast into the compute to refresh it's nw info cache | |
| 17:28:25 | melwitt | O.o | |
| 17:38:31 | hemna | mriedem, it's job is to tear down the external connection from the backend to the host | |
| 17:38:44 | hemna | effectively unexport the volume | |
| 17:40:32 | peter-hamilton | jaypipes: i'll be afk for a bit; if you have any questions on cert validation, i'll answer once i get back | |
| 17:57:21 | mriedem | hemna: so seems like that would be important when an instance with an attached volume disconnects from it's host and eventually shows up elsewhere (another compute host) yeah? | |
| 17:57:25 | mriedem | or does it depend on the backend? | |
| 17:57:48 | hemna | not sure I'm following | |
| 17:58:04 | mriedem | hemna: are you familiar with the shelve offload operation in nova? | |
| 17:58:20 | hemna | not imtimately | |
| 17:58:29 | mriedem | let me introduce you | |
| 17:58:33 | hemna | :P ok | |
| 17:58:39 | mriedem | basically keep your volume attached in the cinder db, | |
| 17:58:45 | mriedem | but we destroy the guest vm from the compute host | |
| 17:58:51 | mriedem | keep ports and volumes attached in dbs | |
| 17:59:02 | mriedem | when we destroy the guest, we disconnect any volumes | |
| 17:59:16 | mriedem | then the instance is in this shelve-offloaded state, where it doesn't have a host | |
| 17:59:24 | mriedem | (we also take a snapshot of it before offloading it) | |
| 17:59:27 | hemna | ok | |
| 17:59:33 | hemna | so it's like a dormant vm | |
| 17:59:36 | mriedem | when you unshelve the instance, we take the snapshot image and boot the server on a new host | |
| 17:59:43 | mriedem | sort of yeah | |
| 17:59:50 | mriedem | it might land on the same host as before, but it might not | |
| 18:00:25 | hemna | yah, so I believe terminate_connection should be called then in this case, to ensure that the backend disconnects from the host | |
| 18:00:35 | mriedem | when the instance is created on a new host, we call initialize_connection again | |
| 18:00:35 | dane-fichter | jaypipes: also I can answer most cert validation questions while peter-hamilton is afk | |
| 18:00:43 | hemna | makes sense | |
| 18:00:51 | openstackgerrit | Mathieu Gagné proposed openstack/nova-specs master: Nova support for attached volume extend https://review.openstack.org/453272 | |
| 18:01:58 | mriedem | hemna: the funny thing is on unshelve, | |
| 18:02:00 | jgriffith | mriedem hemna there is no terminate_connection, only “attachment_delete” or “update" | |
| 18:02:02 | jgriffith | :) | |
| 18:02:07 | hemna | :P | |
| 18:02:09 | mriedem | is i don't think we ever call connect_volume in brick on the new host | |
| 18:02:19 | hemna | ! | |
| 18:02:22 | mriedem | so even though we initialize the connection, we don't connect the volume | |
| 18:02:26 | mriedem | as far as i can tell | |
| 18:02:28 | hemna | oops | |
| 18:02:48 | mriedem | let me check the driver.spawn() method though | |
| 18:02:51 | mriedem | maybe it's done there | |
| 18:02:54 | mriedem | some of this garbage is hidden | |