Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-12-11
18:41:25 _ix Cool. I see that the host is disabled in the `nova hypervisor-list` Thanks for the help. Odd.
18:41:44 mriedem _ix: starting in pike,
18:41:46 smatzek rebuild, which I haven't dug as deep into today, is failing a bit more nuanced way. The mount point it is getting from either cinder or nova is giving /dev/na, as in "not applicable" which is a default for some fields in Cinder.
18:41:55 mriedem nova-compute will auto-disable the host if it's had 10 failed build attempts
18:41:58 mriedem in a row
18:42:57 openstackgerrit Andrey Volkov proposed openstack/osc-placement master: CLI for allocations https://review.openstack.org/457534
18:42:57 _ix Oh, hey. That's a fine tip. Thanks again! I'm sure that's what's going on here.
18:42:58 openstackgerrit Andrey Volkov proposed openstack/osc-placement master: CLI for aggregates (v1.1) https://review.openstack.org/505643
18:42:58 openstackgerrit Andrey Volkov proposed openstack/osc-placement master: CLI for usages https://review.openstack.org/457535
18:42:59 openstackgerrit Andrey Volkov proposed openstack/osc-placement master: RP list: member_of and resources parameters (v1.3, v1.4) https://review.openstack.org/511183
18:42:59 openstackgerrit Andrey Volkov proposed openstack/osc-placement master: CLI for resource classes (v1.2) https://review.openstack.org/511182
18:43:00 openstackgerrit Andrey Volkov proposed openstack/osc-placement master: CLI for traits (v1.6) https://review.openstack.org/514643
18:43:00 openstackgerrit Andrey Volkov proposed openstack/osc-placement master: RP delete inventories (v1.5) https://review.openstack.org/514642
18:43:01 openstackgerrit Andrey Volkov proposed openstack/osc-placement master: Usages per project and user (v1.8, v1.9) https://review.openstack.org/514646
18:43:01 openstackgerrit Andrey Volkov proposed openstack/osc-placement master: Resource class set (v1.7) https://review.openstack.org/514644
18:43:02 openstackgerrit Andrey Volkov proposed openstack/osc-placement master: CLI allocation candidates (v1.10) https://review.openstack.org/514647
18:43:15 _ix Also, a side note. That's a super handy feature from my previous deployments.
18:43:20 mriedem _ix: there is a config option if you want to change that
18:43:29 mriedem but yeah, ^ is why it's on by default
18:43:30 _ix I don't. I think that's great.
18:43:35 mriedem dansmith: ^ high praise
18:44:11 smatzek mriedem: the tempest test test_resize_volume_backed_server_confirm is doing resize with a volume backed / BFV instance. The trove test is different, the instance has ephemeral disk and a volume attached.
18:44:42 mriedem smatzek: hmm http://logs.openstack.org/39/524539/1/check/legacy-trove-functional-dsvm-mysql/3ef681d/logs/screen-n-cpu.txt.gz#_Dec_11_03_03_28_499795
18:45:02 mriedem {"attachment": {"status": "reserved", "detached_at": "", "connection_info": {}, "attached_at": "", "attach_mode": null, "instance": "85cdb482-63a5-487a-b103-95b9383ffcc7", "volume_id": "b3c82217-38f4-4c1f-b0f0-486a4fda6dda", "id": "848654a4-9647-46b8-9485-b3647fe87cfb"}}
18:45:03 _ix mriedem: Can you offer any conjecture about kernel versions, etc? We're on Centos 7.4, and upgraded to kernel 4.14. It looks like the rpcbind.socket is failing us.
18:45:13 mriedem _ix: nope
18:45:39 mriedem smatzek: that empty connection_info is presumably the problem
18:45:49 mriedem because nova pulls the driver_volume_type out of the connection_info on the attachment
18:46:00 mriedem smatzek: although,
18:46:11 mriedem i think that's normal for a 'reserved' attachment maybe? we'd have to ask jgriffith
18:46:26 mriedem until nova updates the attachment with the host connector, the attachment doesn't have a connection_info
18:46:39 _ix Will the openstack client one day be the only client?
18:46:48 smatzek yep, I came to the same conclusion, that the empty connection_info coming back from Cinder is part of the issue, but was drawing a blank on if it was suppose to be empty there or not.
18:47:06 ildikov mriedem: if it's only reserved connection_info is empty
18:47:18 _ix (That is, the openstack cli client)
18:47:26 mriedem _ix: at this rate i doubt it
18:47:36 _ix That's a sad story.
18:47:38 mriedem _ix: been talking about that for years now
18:47:43 smatzek To my knowledge Trove hasn't had changes here and it just started about when the Nova code landed so I thought I'd check. Also, it's not "mainline". Regular attach works and only resize and rebuild are having issues afaik.
18:48:04 mriedem _ix: would need to get feature parity in osc
18:48:08 mriedem _ix: and no one is working on that
18:48:15 dansmith mriedem: nice
18:48:18 _ix Forgive me, what is osc?
18:48:25 mriedem _ix: we could make that a priority, but that means de-prioritizing other work
18:48:28 mriedem openstack client
18:48:34 mriedem OpenStackClient
18:48:36 _ix Oh, I see.
18:49:48 ildikov smatzek: I need to jump on a call, but will look into the logs once I got off the phone
18:50:34 ildikov smatzek: the new flow shouldn't blow those calls up, but tempest might lack some coverage, I don't think we did a thorough manual testing :(
18:51:01 mriedem hmm
18:51:02 mriedem 'block_device_mapping': [{'guest_format': None, 'boot_index': -1, 'mount_device': u'/dev/vdb', 'connection_info': {'serial': u'b3c82217-38f4-4c1f-b0f0-486a4fda6dda'}, 'disk_bus': u'virtio', 'device_type': u'disk', 'attachment_id': '848654a4-9647-46b8-9485-b3647fe87cfb', 'delete_on_termination': True}]
18:53:48 mriedem smatzek: can you open a bug, the logstash query does show this starting on the 9th
18:53:58 mriedem http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22VolumeDriverNotFound%3A%20Could%20not%20find%20a%20handler%20for%20None%20volume.%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22&from=7d
18:54:09 mriedem *8th actually
18:56:16 smatzek mriedem: will do, and then I'm going to look into the Trove upgrade failure, which uses Nova rebuild a bit more. The stack/failure there is in Trove code but it has that bad /dev/na mountpoint which it fetched from an API (I don't recall which without re-digging)
18:57:10 ildikov I don't recall seeing this issue earlier
18:58:07 ildikov we should have connection_info by the time we get to the _connect_volume call, I need to look into what went wrong there
19:02:51 smatzek mriedem: https://bugs.launchpad.net/nova/+bug/1737599
19:02:52 openstack Launchpad bug 1737599 in OpenStack Compute (nova) "Instance resize with attach volume fails" [Undecided,New]
19:03:38 mriedem i can see what's different about this flow from the bfv one
19:04:19 mriedem we seem to lose the connection_info during some of the much complicated bdm object -> driver bdm -> dict block_device_info conversions...
19:04:28 mriedem or, it was just never set in the first place
19:04:35 mriedem my guess is the latter
19:05:08 mriedem i can definitely see where it blows up though yeah
19:10:51 openstackgerrit Ed Leafe proposed openstack/nova master: Change RPC for select_destinations() https://review.openstack.org/516707
19:10:52 openstackgerrit Ed Leafe proposed openstack/nova master: Make conductor pass and use host_lists https://review.openstack.org/511358
19:10:52 openstackgerrit Ed Leafe proposed openstack/nova master: Move the claim_resources method to scheduler utils https://review.openstack.org/511357
19:10:53 openstackgerrit Ed Leafe proposed openstack/nova master: Change compute RPC to use alternates for resize https://review.openstack.org/526436
19:11:25 edleafe mriedem: cleaned up ^^
19:18:49 openstackgerrit Chris Dent proposed openstack/nova master: [placement] Add cache headers to placement api requests https://review.openstack.org/521640
19:18:50 openstackgerrit Chris Dent proposed openstack/nova master: [placement] Add info about last-modified to contrib docs https://review.openstack.org/526084
19:20:15 cdent mriedem, edleafe ^ that fixes the several suggestions matt had
19:20:50 mriedem smatzek: ildikov: i think i have the resize thing figured out
19:23:25 smatzek mriedem: great, thanks
19:24:23 mriedem i'll post a patch and you can push a trove depends-on to see if it works
19:28:59 smatzek would the thing you found possibly cause the mountpoint to be lost in Cinder's attachments during rebuild?
19:29:19 smatzek mriedem: ^
19:29:52 ildikov mriedem: cool, that was quick!
19:30:54 mriedem smatzek: don't know, one problem at a time
19:31:02 mriedem smatzek: but maybe
19:31:52 openstackgerrit Murali Annamneni proposed openstack/nova master: Enables MySQL Cluster Support for Nova https://review.openstack.org/446643
19:31:54 mriedem smatzek: there is a good chance it is though
19:41:40 ildikov smatzek: that log indicates that there might be a step missing from rebuild
19:42:12 ildikov smatzek: will check what mriedem has as a fix and then we can compare whether we covered everything or not
19:42:55 ericyoung Any idea when https://review.openstack.org/#/c/407440/ will get some eyes on it?
19:43:08 ericyoung It is the patch for ScaleIO ephemeral/image volumes
19:48:30 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Save driver_volume_type when refreshing attachment connection_info https://review.openstack.org/527228
19:48:31 mriedem smatzek: ildikov: ^ see if that fixes the resize issue in trove; needs unit tests
19:51:30 mriedem actually, i think that will fix the VolumeDriverNotFound but we'll still fail to connect the volume on the destination host
19:51:39 mriedem because the new attachment connection_info doesn't have 'data' set
19:52:32 smatzek so should I hold off on pushing a depends-on test to Trove?
19:52:43 mriedem just start that depends-on to make sure it blows up where i think it is now
19:52:51 mriedem and we'll need it anyway to verify the eventual fix
19:54:10 smatzek mriedem: https://review.openstack.org/#/c/527230. It looks like the depends-on linked correctly. To be honest I've never done one cross-projects so hopefully this is right.
19:54:11 smatzek https://review.openstack.org/#/c/527230
19:54:29 mriedem yes that should work
20:06:34 mriedem ok this is a bit shittier than i thought, but i think i know of another relatively painless fix
20:06:51 mriedem to avoid doing crazy crap in the virt driver(s) or nova/virt/block_device.py code
20:07:16 ildikov mriedem: hmm, we are explicitly transforming 'driver_volume_type' so it is supposed to come from Cinder already
20:07:35 mriedem ildikov: it's pulled out of the connection_info from the attachment,

Earlier   Later