Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-12-11
18:35:41 mriedem _ix: you have to run the discover_hosts CLI
18:35:47 mriedem nova-manage cell_v2 discover_hosts --verbose
18:35:52 _ix I have. It says there aren't any unmapped hosts.
18:36:05 mriedem what does nova service-list show?
18:36:11 mriedem does the entry show up there?
18:36:44 _ix For nova compute? Yes.
18:37:08 _ix This is just 1/5 of the hosts not showing up.
18:37:20 _ix But, I might be misleading a bit. I see there's an rpc problem on the host.
18:37:31 mriedem what does nova hypervisor-list show?
18:37:40 mriedem nova host-list is deprecated since pike btw
18:37:53 smatzek The Trove gates started failing on Friday or Saturday, right about the time the new attach flows landed. In particular they are failing when resizing or rebuilding instances with attached volumes. Does Nova have gate tests that do this?
18:38:11 openstackgerrit Dmitry Tantsur proposed openstack/nova master: Re-use existing ComputeNode on ironic rebalance https://review.openstack.org/508555
18:38:14 dtantsur mriedem: I hope I did not mess up anything ^^^
18:38:17 mriedem smatzek: yes for resize with volume attached - pretty sure anyway
18:38:24 mriedem smatzek: not sure about rebuild with a volume attached (in tempest anyway)
18:38:51 _ix mriedem: I generally use the openstack client. Is `openstack host list` also deprecated?
18:39:49 mriedem smatzek: test_resize_volume_backed_server_confirm
18:39:51 mriedem in tempest
18:40:07 smatzek mriedem: resize is failing like this, which in looking at the code means that connection_info is missing the driver type, and in looking at the previous log entries I can see that connection_info is indeed missing it: http://logs.openstack.org/39/524539/1/check/legacy-trove-functional-dsvm-mysql/3ef681d/logs/screen-n-cpu.txt.gz#_Dec_11_03_03_28_755007
18:40:43 mriedem _ix: i deprecated nova host-list and didn't update openstack client so it's probably not deprecated
18:41:25 _ix Cool. I see that the host is disabled in the `nova hypervisor-list` Thanks for the help. Odd.
18:41:44 mriedem _ix: starting in pike,
18:41:46 smatzek rebuild, which I haven't dug as deep into today, is failing a bit more nuanced way. The mount point it is getting from either cinder or nova is giving /dev/na, as in "not applicable" which is a default for some fields in Cinder.
18:41:55 mriedem nova-compute will auto-disable the host if it's had 10 failed build attempts
18:41:58 mriedem in a row
18:42:57 openstackgerrit Andrey Volkov proposed openstack/osc-placement master: CLI for allocations https://review.openstack.org/457534
18:42:57 _ix Oh, hey. That's a fine tip. Thanks again! I'm sure that's what's going on here.
18:42:58 openstackgerrit Andrey Volkov proposed openstack/osc-placement master: CLI for aggregates (v1.1) https://review.openstack.org/505643
18:42:58 openstackgerrit Andrey Volkov proposed openstack/osc-placement master: CLI for usages https://review.openstack.org/457535
18:42:59 openstackgerrit Andrey Volkov proposed openstack/osc-placement master: RP list: member_of and resources parameters (v1.3, v1.4) https://review.openstack.org/511183
18:42:59 openstackgerrit Andrey Volkov proposed openstack/osc-placement master: CLI for resource classes (v1.2) https://review.openstack.org/511182
18:43:00 openstackgerrit Andrey Volkov proposed openstack/osc-placement master: CLI for traits (v1.6) https://review.openstack.org/514643
18:43:00 openstackgerrit Andrey Volkov proposed openstack/osc-placement master: RP delete inventories (v1.5) https://review.openstack.org/514642
18:43:01 openstackgerrit Andrey Volkov proposed openstack/osc-placement master: Usages per project and user (v1.8, v1.9) https://review.openstack.org/514646
18:43:01 openstackgerrit Andrey Volkov proposed openstack/osc-placement master: Resource class set (v1.7) https://review.openstack.org/514644
18:43:02 openstackgerrit Andrey Volkov proposed openstack/osc-placement master: CLI allocation candidates (v1.10) https://review.openstack.org/514647
18:43:15 _ix Also, a side note. That's a super handy feature from my previous deployments.
18:43:20 mriedem _ix: there is a config option if you want to change that
18:43:29 mriedem but yeah, ^ is why it's on by default
18:43:30 _ix I don't. I think that's great.
18:43:35 mriedem dansmith: ^ high praise
18:44:11 smatzek mriedem: the tempest test test_resize_volume_backed_server_confirm is doing resize with a volume backed / BFV instance. The trove test is different, the instance has ephemeral disk and a volume attached.
18:44:42 mriedem smatzek: hmm http://logs.openstack.org/39/524539/1/check/legacy-trove-functional-dsvm-mysql/3ef681d/logs/screen-n-cpu.txt.gz#_Dec_11_03_03_28_499795
18:45:02 mriedem {"attachment": {"status": "reserved", "detached_at": "", "connection_info": {}, "attached_at": "", "attach_mode": null, "instance": "85cdb482-63a5-487a-b103-95b9383ffcc7", "volume_id": "b3c82217-38f4-4c1f-b0f0-486a4fda6dda", "id": "848654a4-9647-46b8-9485-b3647fe87cfb"}}
18:45:03 _ix mriedem: Can you offer any conjecture about kernel versions, etc? We're on Centos 7.4, and upgraded to kernel 4.14. It looks like the rpcbind.socket is failing us.
18:45:13 mriedem _ix: nope
18:45:39 mriedem smatzek: that empty connection_info is presumably the problem
18:45:49 mriedem because nova pulls the driver_volume_type out of the connection_info on the attachment
18:46:00 mriedem smatzek: although,
18:46:11 mriedem i think that's normal for a 'reserved' attachment maybe? we'd have to ask jgriffith
18:46:26 mriedem until nova updates the attachment with the host connector, the attachment doesn't have a connection_info
18:46:39 _ix Will the openstack client one day be the only client?
18:46:48 smatzek yep, I came to the same conclusion, that the empty connection_info coming back from Cinder is part of the issue, but was drawing a blank on if it was suppose to be empty there or not.
18:47:06 ildikov mriedem: if it's only reserved connection_info is empty
18:47:18 _ix (That is, the openstack cli client)
18:47:26 mriedem _ix: at this rate i doubt it
18:47:36 _ix That's a sad story.
18:47:38 mriedem _ix: been talking about that for years now
18:47:43 smatzek To my knowledge Trove hasn't had changes here and it just started about when the Nova code landed so I thought I'd check. Also, it's not "mainline". Regular attach works and only resize and rebuild are having issues afaik.
18:48:04 mriedem _ix: would need to get feature parity in osc
18:48:08 mriedem _ix: and no one is working on that
18:48:15 dansmith mriedem: nice
18:48:18 _ix Forgive me, what is osc?
18:48:25 mriedem _ix: we could make that a priority, but that means de-prioritizing other work
18:48:28 mriedem openstack client
18:48:34 mriedem OpenStackClient
18:48:36 _ix Oh, I see.
18:49:48 ildikov smatzek: I need to jump on a call, but will look into the logs once I got off the phone
18:50:34 ildikov smatzek: the new flow shouldn't blow those calls up, but tempest might lack some coverage, I don't think we did a thorough manual testing :(
18:51:01 mriedem hmm
18:51:02 mriedem 'block_device_mapping': [{'guest_format': None, 'boot_index': -1, 'mount_device': u'/dev/vdb', 'connection_info': {'serial': u'b3c82217-38f4-4c1f-b0f0-486a4fda6dda'}, 'disk_bus': u'virtio', 'device_type': u'disk', 'attachment_id': '848654a4-9647-46b8-9485-b3647fe87cfb', 'delete_on_termination': True}]
18:53:48 mriedem smatzek: can you open a bug, the logstash query does show this starting on the 9th
18:53:58 mriedem http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22VolumeDriverNotFound%3A%20Could%20not%20find%20a%20handler%20for%20None%20volume.%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22&from=7d
18:54:09 mriedem *8th actually
18:56:16 smatzek mriedem: will do, and then I'm going to look into the Trove upgrade failure, which uses Nova rebuild a bit more. The stack/failure there is in Trove code but it has that bad /dev/na mountpoint which it fetched from an API (I don't recall which without re-digging)
18:57:10 ildikov I don't recall seeing this issue earlier
18:58:07 ildikov we should have connection_info by the time we get to the _connect_volume call, I need to look into what went wrong there
19:02:51 smatzek mriedem: https://bugs.launchpad.net/nova/+bug/1737599
19:02:52 openstack Launchpad bug 1737599 in OpenStack Compute (nova) "Instance resize with attach volume fails" [Undecided,New]
19:03:38 mriedem i can see what's different about this flow from the bfv one
19:04:19 mriedem we seem to lose the connection_info during some of the much complicated bdm object -> driver bdm -> dict block_device_info conversions...
19:04:28 mriedem or, it was just never set in the first place
19:04:35 mriedem my guess is the latter
19:05:08 mriedem i can definitely see where it blows up though yeah
19:10:51 openstackgerrit Ed Leafe proposed openstack/nova master: Change RPC for select_destinations() https://review.openstack.org/516707
19:10:52 openstackgerrit Ed Leafe proposed openstack/nova master: Make conductor pass and use host_lists https://review.openstack.org/511358
19:10:52 openstackgerrit Ed Leafe proposed openstack/nova master: Move the claim_resources method to scheduler utils https://review.openstack.org/511357
19:10:53 openstackgerrit Ed Leafe proposed openstack/nova master: Change compute RPC to use alternates for resize https://review.openstack.org/526436
19:11:25 edleafe mriedem: cleaned up ^^
19:18:49 openstackgerrit Chris Dent proposed openstack/nova master: [placement] Add cache headers to placement api requests https://review.openstack.org/521640
19:18:50 openstackgerrit Chris Dent proposed openstack/nova master: [placement] Add info about last-modified to contrib docs https://review.openstack.org/526084
19:20:15 cdent mriedem, edleafe ^ that fixes the several suggestions matt had
19:20:50 mriedem smatzek: ildikov: i think i have the resize thing figured out
19:23:25 smatzek mriedem: great, thanks
19:24:23 mriedem i'll post a patch and you can push a trove depends-on to see if it works
19:28:59 smatzek would the thing you found possibly cause the mountpoint to be lost in Cinder's attachments during rebuild?
19:29:19 smatzek mriedem: ^
19:29:52 ildikov mriedem: cool, that was quick!
19:30:54 mriedem smatzek: don't know, one problem at a time
19:31:02 mriedem smatzek: but maybe

Earlier   Later