Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-12-11
18:47:26 mriedem _ix: at this rate i doubt it
18:47:36 _ix That's a sad story.
18:47:38 mriedem _ix: been talking about that for years now
18:47:43 smatzek To my knowledge Trove hasn't had changes here and it just started about when the Nova code landed so I thought I'd check. Also, it's not "mainline". Regular attach works and only resize and rebuild are having issues afaik.
18:48:04 mriedem _ix: would need to get feature parity in osc
18:48:08 mriedem _ix: and no one is working on that
18:48:15 dansmith mriedem: nice
18:48:18 _ix Forgive me, what is osc?
18:48:25 mriedem _ix: we could make that a priority, but that means de-prioritizing other work
18:48:28 mriedem openstack client
18:48:34 mriedem OpenStackClient
18:48:36 _ix Oh, I see.
18:49:48 ildikov smatzek: I need to jump on a call, but will look into the logs once I got off the phone
18:50:34 ildikov smatzek: the new flow shouldn't blow those calls up, but tempest might lack some coverage, I don't think we did a thorough manual testing :(
18:51:01 mriedem hmm
18:51:02 mriedem 'block_device_mapping': [{'guest_format': None, 'boot_index': -1, 'mount_device': u'/dev/vdb', 'connection_info': {'serial': u'b3c82217-38f4-4c1f-b0f0-486a4fda6dda'}, 'disk_bus': u'virtio', 'device_type': u'disk', 'attachment_id': '848654a4-9647-46b8-9485-b3647fe87cfb', 'delete_on_termination': True}]
18:53:48 mriedem smatzek: can you open a bug, the logstash query does show this starting on the 9th
18:53:58 mriedem http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22VolumeDriverNotFound%3A%20Could%20not%20find%20a%20handler%20for%20None%20volume.%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22&from=7d
18:54:09 mriedem *8th actually
18:56:16 smatzek mriedem: will do, and then I'm going to look into the Trove upgrade failure, which uses Nova rebuild a bit more. The stack/failure there is in Trove code but it has that bad /dev/na mountpoint which it fetched from an API (I don't recall which without re-digging)
18:57:10 ildikov I don't recall seeing this issue earlier
18:58:07 ildikov we should have connection_info by the time we get to the _connect_volume call, I need to look into what went wrong there
19:02:51 smatzek mriedem: https://bugs.launchpad.net/nova/+bug/1737599
19:02:52 openstack Launchpad bug 1737599 in OpenStack Compute (nova) "Instance resize with attach volume fails" [Undecided,New]
19:03:38 mriedem i can see what's different about this flow from the bfv one
19:04:19 mriedem we seem to lose the connection_info during some of the much complicated bdm object -> driver bdm -> dict block_device_info conversions...
19:04:28 mriedem or, it was just never set in the first place
19:04:35 mriedem my guess is the latter
19:05:08 mriedem i can definitely see where it blows up though yeah
19:10:51 openstackgerrit Ed Leafe proposed openstack/nova master: Change RPC for select_destinations() https://review.openstack.org/516707
19:10:52 openstackgerrit Ed Leafe proposed openstack/nova master: Move the claim_resources method to scheduler utils https://review.openstack.org/511357
19:10:52 openstackgerrit Ed Leafe proposed openstack/nova master: Make conductor pass and use host_lists https://review.openstack.org/511358
19:10:53 openstackgerrit Ed Leafe proposed openstack/nova master: Change compute RPC to use alternates for resize https://review.openstack.org/526436
19:11:25 edleafe mriedem: cleaned up ^^
19:18:49 openstackgerrit Chris Dent proposed openstack/nova master: [placement] Add cache headers to placement api requests https://review.openstack.org/521640
19:18:50 openstackgerrit Chris Dent proposed openstack/nova master: [placement] Add info about last-modified to contrib docs https://review.openstack.org/526084
19:20:15 cdent mriedem, edleafe ^ that fixes the several suggestions matt had
19:20:50 mriedem smatzek: ildikov: i think i have the resize thing figured out
19:23:25 smatzek mriedem: great, thanks
19:24:23 mriedem i'll post a patch and you can push a trove depends-on to see if it works
19:28:59 smatzek would the thing you found possibly cause the mountpoint to be lost in Cinder's attachments during rebuild?
19:29:19 smatzek mriedem: ^
19:29:52 ildikov mriedem: cool, that was quick!
19:30:54 mriedem smatzek: don't know, one problem at a time
19:31:02 mriedem smatzek: but maybe
19:31:52 openstackgerrit Murali Annamneni proposed openstack/nova master: Enables MySQL Cluster Support for Nova https://review.openstack.org/446643
19:31:54 mriedem smatzek: there is a good chance it is though
19:41:40 ildikov smatzek: that log indicates that there might be a step missing from rebuild
19:42:12 ildikov smatzek: will check what mriedem has as a fix and then we can compare whether we covered everything or not
19:42:55 ericyoung Any idea when https://review.openstack.org/#/c/407440/ will get some eyes on it?
19:43:08 ericyoung It is the patch for ScaleIO ephemeral/image volumes
19:48:30 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Save driver_volume_type when refreshing attachment connection_info https://review.openstack.org/527228
19:48:31 mriedem smatzek: ildikov: ^ see if that fixes the resize issue in trove; needs unit tests
19:51:30 mriedem actually, i think that will fix the VolumeDriverNotFound but we'll still fail to connect the volume on the destination host
19:51:39 mriedem because the new attachment connection_info doesn't have 'data' set
19:52:32 smatzek so should I hold off on pushing a depends-on test to Trove?
19:52:43 mriedem just start that depends-on to make sure it blows up where i think it is now
19:52:51 mriedem and we'll need it anyway to verify the eventual fix
19:54:10 smatzek mriedem: https://review.openstack.org/#/c/527230. It looks like the depends-on linked correctly. To be honest I've never done one cross-projects so hopefully this is right.
19:54:11 smatzek https://review.openstack.org/#/c/527230
19:54:29 mriedem yes that should work
20:06:34 mriedem ok this is a bit shittier than i thought, but i think i know of another relatively painless fix
20:06:51 mriedem to avoid doing crazy crap in the virt driver(s) or nova/virt/block_device.py code
20:07:16 ildikov mriedem: hmm, we are explicitly transforming 'driver_volume_type' so it is supposed to come from Cinder already
20:07:35 mriedem ildikov: it's pulled out of the connection_info from the attachment,
20:07:43 mriedem which is empty for the new attachment which is not yet connected on the dest host
20:07:58 mriedem i.e. we don't call attachment_update in the cold migrate flow
20:08:21 mriedem for cold migration, the drivers just do all the work on the hosts with connecting the volume
20:08:40 mriedem and the old init/terminate_connection calls
20:08:42 mriedem on the volume
20:08:42 ildikov hmm, but shouldn't we call attachment_update with the new flow now?
20:08:50 mriedem yes - but we aren't
20:08:57 mriedem so that's what i think i need to add
20:09:23 ildikov sorry, I just got confused with the changes in your patch in mind
20:09:43 ildikov as the log looks like we have a missing attahcment_update call somewhere
20:09:58 ildikov and then we need to check whether attachment_complete happens there or not
20:10:44 mriedem yes my patch does'nt fix it
20:10:51 mriedem i think i know what needs to happen
20:10:54 mriedem but making coffee first
20:13:00 ildikov fair enough
20:31:08 openstackgerrit Ed Leafe proposed openstack/nova master: Make conductor pass and use host_lists https://review.openstack.org/511358
20:31:09 openstackgerrit Ed Leafe proposed openstack/nova master: Change compute RPC to use alternates for resize https://review.openstack.org/526436
20:38:55 _ix More high praise if you're interested. I think that newish Pike feature in disabling nodes saved me about half a day of troubleshooting.
20:39:06 _ix Thank you!
20:41:04 mriedem thanks for the feedback
20:41:15 mriedem always nice to know when people actually "like" something
20:42:43 cdent 👍
20:43:42 mriedem dtantsur|afk: do you have a run on https://review.openstack.org/#/c/508555/ yet?
20:44:06 mriedem TheJulia: do you know where ^ is being tested to see if it unblocks the ironic multinode job?
20:44:50 mriedem dtantsur|afk: TheJulia: nevermind, found it https://review.openstack.org/#/c/527075/
20:45:11 TheJulia :)
20:45:29 mriedem is ironic-tempest-dsvm-ipa-wholedisk-agent_ipmitool-tinyipa-multinode the job to care about?
20:51:48 TheJulia mriedem: yes
20:56:54 mriedem dansmith: comments in john's rebalance patch - https://review.openstack.org/#/c/508555/ - i think it looks better, but we could maybe drop the new object query method too - i think there is an existing one that could be re-used in there
21:00:15 dansmith mriedem: yep, looks right
21:06:41 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Update and complete volume attachments during resize https://review.openstack.org/527228
21:06:55 mriedem smatzek: ildikov: ^ should do it for resize
21:08:06 mriedem smatzek: the rebuild fix will be different, so do you have a separate bug for that?
21:08:09 mriedem with links to the logs
21:08:41 smatzek I don't, but I'll open one.

Earlier   Later