Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-12-11
18:44:42 mriedem smatzek: hmm http://logs.openstack.org/39/524539/1/check/legacy-trove-functional-dsvm-mysql/3ef681d/logs/screen-n-cpu.txt.gz#_Dec_11_03_03_28_499795
18:45:02 mriedem {"attachment": {"status": "reserved", "detached_at": "", "connection_info": {}, "attached_at": "", "attach_mode": null, "instance": "85cdb482-63a5-487a-b103-95b9383ffcc7", "volume_id": "b3c82217-38f4-4c1f-b0f0-486a4fda6dda", "id": "848654a4-9647-46b8-9485-b3647fe87cfb"}}
18:45:03 _ix mriedem: Can you offer any conjecture about kernel versions, etc? We're on Centos 7.4, and upgraded to kernel 4.14. It looks like the rpcbind.socket is failing us.
18:45:13 mriedem _ix: nope
18:45:39 mriedem smatzek: that empty connection_info is presumably the problem
18:45:49 mriedem because nova pulls the driver_volume_type out of the connection_info on the attachment
18:46:00 mriedem smatzek: although,
18:46:11 mriedem i think that's normal for a 'reserved' attachment maybe? we'd have to ask jgriffith
18:46:26 mriedem until nova updates the attachment with the host connector, the attachment doesn't have a connection_info
18:46:39 _ix Will the openstack client one day be the only client?
18:46:48 smatzek yep, I came to the same conclusion, that the empty connection_info coming back from Cinder is part of the issue, but was drawing a blank on if it was suppose to be empty there or not.
18:47:06 ildikov mriedem: if it's only reserved connection_info is empty
18:47:18 _ix (That is, the openstack cli client)
18:47:26 mriedem _ix: at this rate i doubt it
18:47:36 _ix That's a sad story.
18:47:38 mriedem _ix: been talking about that for years now
18:47:43 smatzek To my knowledge Trove hasn't had changes here and it just started about when the Nova code landed so I thought I'd check. Also, it's not "mainline". Regular attach works and only resize and rebuild are having issues afaik.
18:48:04 mriedem _ix: would need to get feature parity in osc
18:48:08 mriedem _ix: and no one is working on that
18:48:15 dansmith mriedem: nice
18:48:18 _ix Forgive me, what is osc?
18:48:25 mriedem _ix: we could make that a priority, but that means de-prioritizing other work
18:48:28 mriedem openstack client
18:48:34 mriedem OpenStackClient
18:48:36 _ix Oh, I see.
18:49:48 ildikov smatzek: I need to jump on a call, but will look into the logs once I got off the phone
18:50:34 ildikov smatzek: the new flow shouldn't blow those calls up, but tempest might lack some coverage, I don't think we did a thorough manual testing :(
18:51:01 mriedem hmm
18:51:02 mriedem 'block_device_mapping': [{'guest_format': None, 'boot_index': -1, 'mount_device': u'/dev/vdb', 'connection_info': {'serial': u'b3c82217-38f4-4c1f-b0f0-486a4fda6dda'}, 'disk_bus': u'virtio', 'device_type': u'disk', 'attachment_id': '848654a4-9647-46b8-9485-b3647fe87cfb', 'delete_on_termination': True}]
18:53:48 mriedem smatzek: can you open a bug, the logstash query does show this starting on the 9th
18:53:58 mriedem http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22VolumeDriverNotFound%3A%20Could%20not%20find%20a%20handler%20for%20None%20volume.%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22&from=7d
18:54:09 mriedem *8th actually
18:56:16 smatzek mriedem: will do, and then I'm going to look into the Trove upgrade failure, which uses Nova rebuild a bit more. The stack/failure there is in Trove code but it has that bad /dev/na mountpoint which it fetched from an API (I don't recall which without re-digging)
18:57:10 ildikov I don't recall seeing this issue earlier
18:58:07 ildikov we should have connection_info by the time we get to the _connect_volume call, I need to look into what went wrong there
19:02:51 smatzek mriedem: https://bugs.launchpad.net/nova/+bug/1737599
19:02:52 openstack Launchpad bug 1737599 in OpenStack Compute (nova) "Instance resize with attach volume fails" [Undecided,New]
19:03:38 mriedem i can see what's different about this flow from the bfv one
19:04:19 mriedem we seem to lose the connection_info during some of the much complicated bdm object -> driver bdm -> dict block_device_info conversions...
19:04:28 mriedem or, it was just never set in the first place
19:04:35 mriedem my guess is the latter
19:05:08 mriedem i can definitely see where it blows up though yeah
19:10:51 openstackgerrit Ed Leafe proposed openstack/nova master: Change RPC for select_destinations() https://review.openstack.org/516707
19:10:52 openstackgerrit Ed Leafe proposed openstack/nova master: Make conductor pass and use host_lists https://review.openstack.org/511358
19:10:52 openstackgerrit Ed Leafe proposed openstack/nova master: Move the claim_resources method to scheduler utils https://review.openstack.org/511357
19:10:53 openstackgerrit Ed Leafe proposed openstack/nova master: Change compute RPC to use alternates for resize https://review.openstack.org/526436
19:11:25 edleafe mriedem: cleaned up ^^
19:18:49 openstackgerrit Chris Dent proposed openstack/nova master: [placement] Add cache headers to placement api requests https://review.openstack.org/521640
19:18:50 openstackgerrit Chris Dent proposed openstack/nova master: [placement] Add info about last-modified to contrib docs https://review.openstack.org/526084
19:20:15 cdent mriedem, edleafe ^ that fixes the several suggestions matt had
19:20:50 mriedem smatzek: ildikov: i think i have the resize thing figured out
19:23:25 smatzek mriedem: great, thanks
19:24:23 mriedem i'll post a patch and you can push a trove depends-on to see if it works
19:28:59 smatzek would the thing you found possibly cause the mountpoint to be lost in Cinder's attachments during rebuild?
19:29:19 smatzek mriedem: ^
19:29:52 ildikov mriedem: cool, that was quick!
19:30:54 mriedem smatzek: don't know, one problem at a time
19:31:02 mriedem smatzek: but maybe
19:31:52 openstackgerrit Murali Annamneni proposed openstack/nova master: Enables MySQL Cluster Support for Nova https://review.openstack.org/446643
19:31:54 mriedem smatzek: there is a good chance it is though
19:41:40 ildikov smatzek: that log indicates that there might be a step missing from rebuild
19:42:12 ildikov smatzek: will check what mriedem has as a fix and then we can compare whether we covered everything or not
19:42:55 ericyoung Any idea when https://review.openstack.org/#/c/407440/ will get some eyes on it?
19:43:08 ericyoung It is the patch for ScaleIO ephemeral/image volumes
19:48:30 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Save driver_volume_type when refreshing attachment connection_info https://review.openstack.org/527228
19:48:31 mriedem smatzek: ildikov: ^ see if that fixes the resize issue in trove; needs unit tests
19:51:30 mriedem actually, i think that will fix the VolumeDriverNotFound but we'll still fail to connect the volume on the destination host
19:51:39 mriedem because the new attachment connection_info doesn't have 'data' set
19:52:32 smatzek so should I hold off on pushing a depends-on test to Trove?
19:52:43 mriedem just start that depends-on to make sure it blows up where i think it is now
19:52:51 mriedem and we'll need it anyway to verify the eventual fix
19:54:10 smatzek mriedem: https://review.openstack.org/#/c/527230. It looks like the depends-on linked correctly. To be honest I've never done one cross-projects so hopefully this is right.
19:54:11 smatzek https://review.openstack.org/#/c/527230
19:54:29 mriedem yes that should work
20:06:34 mriedem ok this is a bit shittier than i thought, but i think i know of another relatively painless fix
20:06:51 mriedem to avoid doing crazy crap in the virt driver(s) or nova/virt/block_device.py code
20:07:16 ildikov mriedem: hmm, we are explicitly transforming 'driver_volume_type' so it is supposed to come from Cinder already
20:07:35 mriedem ildikov: it's pulled out of the connection_info from the attachment,
20:07:43 mriedem which is empty for the new attachment which is not yet connected on the dest host
20:07:58 mriedem i.e. we don't call attachment_update in the cold migrate flow
20:08:21 mriedem for cold migration, the drivers just do all the work on the hosts with connecting the volume
20:08:40 mriedem and the old init/terminate_connection calls
20:08:42 ildikov hmm, but shouldn't we call attachment_update with the new flow now?
20:08:42 mriedem on the volume
20:08:50 mriedem yes - but we aren't
20:08:57 mriedem so that's what i think i need to add
20:09:23 ildikov sorry, I just got confused with the changes in your patch in mind
20:09:43 ildikov as the log looks like we have a missing attahcment_update call somewhere
20:09:58 ildikov and then we need to check whether attachment_complete happens there or not
20:10:44 mriedem yes my patch does'nt fix it
20:10:51 mriedem i think i know what needs to happen
20:10:54 mriedem but making coffee first
20:13:00 ildikov fair enough
20:31:08 openstackgerrit Ed Leafe proposed openstack/nova master: Make conductor pass and use host_lists https://review.openstack.org/511358
20:31:09 openstackgerrit Ed Leafe proposed openstack/nova master: Change compute RPC to use alternates for resize https://review.openstack.org/526436
20:38:55 _ix More high praise if you're interested. I think that newish Pike feature in disabling nodes saved me about half a day of troubleshooting.
20:39:06 _ix Thank you!
20:41:04 mriedem thanks for the feedback
20:41:15 mriedem always nice to know when people actually "like" something
20:42:43 cdent 👍

Earlier   Later