Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-04-27
16:28:54 hemna so there is a potential race between calling os-brick's disconnect_volume and calling cinder's terminate_connection.
16:29:06 hemna where a new attach comes in on the same backend and an scsi bus rescan can happen.
16:29:10 mdbooth Hmm, no. Even scoped to just iscsi is problematic because we need the lock to cover terminate_connection
16:29:33 imacdonn hemna: that's exactly what I've been experiencing (since the icehouse days)
16:29:42 hemna not surprising
16:29:56 hemna this can happen for FC based volumes too
16:30:20 mdbooth terminate_connection really needs to live with disconnect. Soon...
16:31:13 hemna not sure what you mean by that
16:31:22 hemna os-brick calling cinder's terminate_connection ?
16:31:44 mdbooth hemna: I haven't thought that far down, but maybe
16:31:46 hemna or just do a lock around disconnect_volume() and terminate_connection()
16:32:01 hemna os-brick has no business calling cinder
16:32:08 mdbooth That's the issue. Those things are all over the place.
16:32:27 mdbooth I'm not sure I agree with that assertion.
16:32:30 imacdonn it's not that easy .. terminate_connection() is not called from disconnect_volume()
16:32:40 imacdonn maybe it could be, though
16:33:04 hemna os-brick is not going to call cinder.
16:33:10 hemna I'll -2 that immediately
16:33:34 smcginnis +1! os-brick is a utiltiy library.
16:33:38 mdbooth If terminate_connection is relevant to the correct operation of the driver (to be determined, but if...) then we would at the very least need to establish a way to make it happen
16:33:47 hemna besides, we are changing the entire attachment API process between nova and cinder
16:33:47 imacdonn is there some way that os-brick could do a call back into some part of nova that then calls cinder ?
16:34:06 hemna I don't want os-brick knowing anything about who's calling it
16:34:12 hemna it's a library used outside of openstack too.
16:34:44 imacdonn so it'd be a generic "do what you need to do to make this LUN to way on whatever your backend is" interface
16:34:44 mdbooth hemna: I was also thinking the other day that locking requirements are driver-specific
16:34:51 imacdonn go away*
16:35:02 mdbooth hemna: So if we don't want os-brick to do any locking, I get that
16:35:19 mdbooth However, we still need a way for os-brick to tell us what locks we need to hold
16:35:33 mdbooth Take this case, for eg, the lock we want is per-target
16:35:43 mdbooth The caller has no idea what a target is, only os-brick knows that
16:36:04 hemna just lock around the process of disconnecting a volume from the host
16:36:18 hemna which is the workflow of disconnect_volume(), then terminate_connection()
16:36:20 mdbooth lock *what* around the process of disconnecting a volume
16:36:31 imacdonn hemna: it's not, though
16:36:34 mdbooth And now all drivers are going to have the same serialisation requirements
16:36:47 mdbooth For some drivers, concurrent attach/detach is fine
16:36:50 imacdonn hemna: terminate_connection() is called from a bunch of different places in compute.manager
16:36:50 mdbooth Others not
16:37:13 mdbooth For some, async terminate_connection is fine, no locking required
16:37:15 mdbooth Others not
16:37:27 mdbooth This is a hard interface
16:39:55 mdbooth hemna: On the terminate_connection front, it could reasonably be a callback for eg
16:40:05 mdbooth Then os-brick wouldn't have to know anything about cinder
16:40:41 mdbooth But 'tell the target to kill this lun now' is a generic concept
16:41:15 imacdonn yeah, that's what I'm envisaging
16:59:06 hemna ok so why does this guy not call brick's disconnect_volume ?
16:59:07 hemna https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L3616
17:00:06 hemna This is what a proper disconnect workflow should look like in nova https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2264-L2268
17:00:59 hemna err
17:01:01 hemna wait...lol
17:01:09 hemna that doesn't call brick either.
17:01:34 imacdonn right ... brick gets called from within driver.destroy .. that's the problem
17:01:52 imacdonn as the comment right there says ;)
17:02:07 hemna boo!
17:04:08 imacdonn the hack that's gotten me by since icehouse is to put a lock on _shutdown_instance() (and a few similar places in manager) and the same lock on connect_volume() in virt.libvirt.volume.iscsi
17:05:28 openstackgerrit Abhishek Sharma M proposed openstack/nova master: Catching OverQuota Exception https://review.openstack.org/460305
17:05:58 johnthetubaguy mdbooth: that iSCSI question, isn't that backend specific? I thought it was one iSCSI target per volume for some?
17:11:42 openstackgerrit Samantha Blanco proposed openstack/nova master: Remove flavor-rxtx policy https://review.openstack.org/444036
17:37:59 openstackgerrit Steve Noyes proposed openstack/nova master: Add Cinder v3 detach to local_cleanup https://review.openstack.org/456851
17:43:52 efried mordred Have a chance to skim https://review.openstack.org/#/c/460337/ yet?
17:44:05 efried mordred I'm working on the nova changes to accomodate.
17:46:21 mordred oh good - they're fixed now
17:47:03 efried trivial pep8 stuff. Still haven't done live testing, just wanting to make sure I'm heading down the right path here.
18:08:52 mriedem johnthetubaguy: i thought we had a policy check for showing fault details?
18:09:01 mriedem but i don't see that here https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/views/servers.py#L280
18:09:08 mriedem except the is_admin check
18:28:12 efried sdague Got a sec to talk about [glance]api_server*s* ?
18:37:16 linuxman openstack-nova-scheduler service unable to start how to fix this issue
18:39:45 openstackgerrit Steve Noyes proposed openstack/nova master: Add Cinder v3 detach to shutdown_instance https://review.openstack.org/456877
18:40:47 efried linuxman I think you want to ask that in #openstack?
18:41:29 linuxman but i hava a problem with nova service
18:42:10 efried Counts as "support of Nova deployments", nah?
18:42:12 mriedem linuxman: see the channel topic please, this is generally not a support channel, especially without any sort of context on the specific issue
18:42:36 mriedem "does not start" is not very helpful for debugging
18:44:06 mriedem thorst: it's snowing here, just fyi
18:44:22 linuxman you have any idea about proper rdo openstack channel
18:45:25 mriedem linuxman: i don't see one on https://wiki.openstack.org/wiki/IRC
18:45:29 mriedem but i'd try #openstack-rdo?
18:45:47 mriedem doesn't look like that's it
18:46:09 mriedem linuxman: https://www.rdoproject.org/community/#discuss
18:46:18 imacdonn just #rdo I think
18:46:21 mriedem yes
18:46:46 linuxman just #rdo i tried but know one is there
18:47:09 mriedem i count 291 people
18:47:32 mriedem make sure you're on freenode
18:47:40 imacdonn I'm there .. I'm not just paying attention ;)
18:48:00 openstackgerrit Matt Riedemann proposed openstack/nova master: Expose StandardLogging fixture for use https://review.openstack.org/460642
18:48:00 openstackgerrit Matt Riedemann proposed openstack/nova master: Do not log live migration success when it actually failed https://review.openstack.org/458958
18:48:34 mriedem sdague: i made all of your log fixture dreams come true
18:51:30 mriedem dansmith: you want to tickle this little ocata fix in https://review.openstack.org/#/c/459840/ ?
18:52:11 mriedem dansmith: should i be reviewing https://review.openstack.org/#/c/460616/ ?
18:52:18 mriedem is ironic/tripleo testing that out?
18:52:48 thorst mriedem: lol. It's 80 in RTP
18:53:06 mriedem oh now that i look out the window again the snow stopped
18:53:09 mriedem that's like the 3rd time today
18:53:28 mriedem i could just be hallucinating
18:53:50 dansmith mriedem: working with them now and on functional tests
18:53:54 mriedem ok
18:59:16 mriedem EmilienM: "Postgresql support (or call it how you want) is also removed upstream in OpenStack." where was that decided?

Earlier   Later