| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-04-27 | |||
| 15:49:16 | zioproto | cant find this ide metadata doing select * from volume_glance_metadata where volume_id=uuid | |
| 15:51:06 | zioproto | I give up for today | |
| 15:51:10 | zioproto | thanks for the help | |
| 15:51:20 | zioproto | I will give feedback once this is fixed | |
| 15:51:45 | zioproto | it is probably a corner case because this VM with this cinder volume was created in a weird way | |
| 15:51:54 | zioproto | what is sure is that it booted once in the past | |
| 15:52:00 | zioproto | thank you | |
| 15:52:02 | mriedem | zioproto: i also wouldn't be surprised if swap volume or something was involved | |
| 15:52:17 | mriedem | maybe cinder has a record of volume retype/migrations you can find? | |
| 15:52:26 | zioproto | I have no idea | |
| 15:52:34 | zioproto | in the action log of nova I can find only the start and stop | |
| 15:53:02 | mriedem | right we don't record instance actions for swap volume apparently | |
| 15:53:05 | mriedem | which seems like a gap to me | |
| 15:57:16 | johnthetubaguy | mriedem: there was a spec up at one point to add instance actions into all the missing APIs, I think that had a list of them missing ones | |
| 15:57:24 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Plumbing for tagged nic attachment https://review.openstack.org/415622 | |
| 15:57:26 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Delete nic metadata when detaching interface https://review.openstack.org/435387 | |
| 15:57:27 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Libvirt support for tagged volume attachment https://review.openstack.org/418100 | |
| 15:57:31 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Delete disk metadata when detaching volume https://review.openstack.org/435082 | |
| 15:57:37 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: API support for tagged device attachment https://review.openstack.org/376021 | |
| 15:57:37 | mriedem | johnthetubaguy: ok so just an oversight and not intentional | |
| 15:57:37 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Plumbing for tagged volume attachment https://review.openstack.org/391941 | |
| 15:57:40 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Libvirt support for tagged nic attachment https://review.openstack.org/415520 | |
| 15:57:51 | johnthetubaguy | mriedem: yeah, its just a not finished thing, I believe | |
| 15:58:56 | johnthetubaguy | mriedem: as volume operations don't change the server state, generally, they are in a slight bit of limbo in our error handling, honestly | |
| 16:00:15 | mriedem | the task state changes | |
| 16:00:26 | imacdonn | mdbooth: still around ? | |
| 16:00:37 | prometheanfire | mriedem: you know about the novaclient issues (8.0.0 not in UC because it looks like it breaks things) https://review.openstack.org/454164 | |
| 16:00:42 | mriedem | nova/cinder meeting in #openstack-meeting-cp | |
| 16:00:56 | mriedem | prometheanfire: i know osc and horizon had issues | |
| 16:00:58 | mriedem | b/c of nova-network | |
| 16:01:11 | prometheanfire | k, just making sure you were aware | |
| 16:01:20 | mriedem | prometheanfire: it came up in the mailing list about a month ago i think | |
| 16:01:27 | mriedem | thanks | |
| 16:01:27 | mriedem | but yeah | |
| 16:01:35 | prometheanfire | ya, it's a older review now | |
| 16:02:10 | sdague | prometheanfire: horizon said they were dropping it | |
| 16:02:16 | mdbooth | zioproto: That metadata will only be set on the root volume, btw | |
| 16:02:28 | mdbooth | imacdonn: Yep | |
| 16:02:44 | openstack | Launchpad bug 1683972 in os-brick "Overlapping iSCSI volume detach/attach can leave behind broken SCSI devices and multipath maps." [Undecided,New] | |
| 16:02:44 | imacdonn | mdbooth: hi ... just checking if you're still following https://bugs.launchpad.net/nova/+bug/1683972 ? | |
| 16:02:58 | mriedem | johnthetubaguy: also related to cinder telling nova to do things https://specs.openstack.org/openstack/nova-specs/specs/pike/approved/nova-support-attached-volume-extend.html | |
| 16:03:07 | mriedem | ^ i brought up the error handling and instance actions stuff in that too | |
| 16:03:14 | mriedem | b/c if/when it fails we need a way to report it | |
| 16:03:42 | imacdonn | mdbooth: I added os-brick and geguileo to it, but haven't heard anything from that side either | |
| 16:03:54 | mdbooth | imacdonn: Just reading your comments | |
| 16:04:03 | imacdonn | mdbooth: k, tnx | |
| 16:04:54 | mdbooth | imacdonn: Did you get the chance to try with the very latest os-brick? | |
| 16:05:24 | mdbooth | Ignore me, I see you did | |
| 16:05:40 | imacdonn | mdbooth: I applied the patch for that specific fix... I suppose it's possible that something else changed in another commit, but I doubt it | |
| 16:10:36 | sfinucan | johnthetubaguy: This patch hit a merge conflict. Fancy reviewing it again to get it in? https://review.openstack.org/#/c/418489 | |
| 16:10:50 | sfinucan | (previously +w'd, but very prone to merge conflict) | |
| 16:11:15 | openstackgerrit | Stephen Finucane proposed openstack/nova master: trivial: Remove dead code https://review.openstack.org/460662 | |
| 16:11:15 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Replace reserved_host_disk_mb with reserved_host_disk_gb https://review.openstack.org/460661 | |
| 16:11:24 | mdbooth | imacdonn: So you're referring to _disconnect_volume_iscsi in os-brick, right? | |
| 16:11:34 | sfinucan | ^ edleafe, jaypipes: You might like the first of those | |
| 16:11:40 | mdbooth | imacdonn: The block starting: | |
| 16:11:46 | mdbooth | # target are in use. | |
| 16:11:46 | mdbooth | # NOTE(vish): Only disconnect from the target if no luns from the | |
| 16:11:52 | sfinucan | or you might not. It's kind of hacky, but oslo.config doesn't give me anything more powerful :( | |
| 16:12:02 | mdbooth | imacdonn: That's definitely looks racy | |
| 16:12:15 | edleafe | sfinucan: thx - will review | |
| 16:12:18 | jaypipes | sfinucan: cool, added to my queue. :) | |
| 16:12:54 | imacdonn | mdbooth: yes .. I understand why they want to clean up the session if it's not needed any more, but it reopens that window where another attachment cause zombie devices to appear | |
| 16:16:45 | mdbooth | imacdonn: So again the issue here is that we drop and then re-establish the connection to the target before terminate_connection has completed | |
| 16:17:17 | mdbooth | And on re-establishing the connection, we rescan and get the lun on the host | |
| 16:17:34 | mdbooth | And then terminate_connection does its thing and it goes stale | |
| 16:17:42 | mdbooth | Have I got the sequence right? | |
| 16:18:30 | imacdonn | mdbooth: there's no explicit rescan, but I suppose there is an effective one ... i.e. on new session establishment, *all* LUNs offered by the target are discovered | |
| 16:18:38 | mdbooth | Right | |
| 16:19:26 | mdbooth | So again we want a per-target lock which covers attach, detach, and terminate connection | |
| 16:19:54 | imacdonn | mdbooth: That seems to be the only way to really cover it | |
| 16:20:17 | imacdonn | mdbooth: I'm also a little bit wary of the os-brick patch .. it seems a bit sketchy, they way he's doing it now | |
| 16:20:33 | mdbooth | The problem being that at the scope where we could currently put that lock we have no knowledge of targets | |
| 16:23:21 | mdbooth | imacdonn mriedem johnthetubaguy: How many iscsi targets would typically exist in a deployment? | |
| 16:23:57 | mdbooth | I personally baulk at the idea of a Global ISCSI Lock | |
| 16:24:24 | mdbooth | But if a typical deployment would only have a single iscsi target anyway, then getting exercised about it is a bit pointless | |
| 16:24:48 | mriedem | mdbooth: 0 idea | |
| 16:24:54 | mriedem | mdbooth: you'd have to ask jgriffith or hemna probably | |
| 16:24:58 | imacdonn | in my deployments, I have two per AZ ... they are the two halves of a ZFSSA cluster | |
| 16:25:11 | mdbooth | Are there deployments out there doing lots of connect/disconnect which would be knobbled if you just serialised all iscsi operations regardless? | |
| 16:25:22 | dansmith | you have at least one target per cinder-volume worker I think, | |
| 16:25:23 | hemna | mdbooth, what do you mean? | |
| 16:25:44 | dansmith | so if you're running a bunch of cinder-volumes with local LVM type stuff, you could have lots of targets right? | |
| 16:25:51 | hemna | yes | |
| 16:26:11 | hemna | and you'd have lots of targets if you used a storage array behind cinder too | |
| 16:26:17 | dansmith | yeah | |
| 16:26:30 | hemna | a target per exported volume | |
| 16:26:32 | mdbooth | Ok, so a global iscsi lock would be a significant difference in practise to a per-target lock | |
| 16:26:57 | mdbooth | It would still be per-host though, I guess | |
| 16:27:00 | mdbooth | Perhaps it's not that bad | |
| 16:27:07 | imacdonn | depends on the type of storage array perhaps .. the ZFSSA has one target per control-head, and lots of volumes on that target with different LUN numbers | |
| 16:27:51 | hemna | but that's one target portal | |
| 16:28:02 | hemna | with multiple targets (one per lun) | |
| 16:28:28 | hemna | either way, yes, you can have multiple targets per storage backend going to the same initiator | |
| 16:28:54 | hemna | so there is a potential race between calling os-brick's disconnect_volume and calling cinder's terminate_connection. | |
| 16:29:06 | hemna | where a new attach comes in on the same backend and an scsi bus rescan can happen. | |
| 16:29:10 | mdbooth | Hmm, no. Even scoped to just iscsi is problematic because we need the lock to cover terminate_connection | |
| 16:29:33 | imacdonn | hemna: that's exactly what I've been experiencing (since the icehouse days) | |
| 16:29:42 | hemna | not surprising | |
| 16:29:56 | hemna | this can happen for FC based volumes too | |