| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-12-11 | |||
| 18:31:42 | mriedem | *vcenter | |
| 18:32:06 | cdent | i was mostly thinking of my.petapbytediskfarm.rp-5 | |
| 18:32:32 | mriedem | but then the inventory is per resource class per provider isn't it? | |
| 18:32:46 | mriedem | so you'd only have a lot of inventory entries for a provider if you have a lot of resource classes on that provider | |
| 18:34:28 | _ix | Hello friends. I'm having some trouble getting a compute node to show up in the hosts list. | |
| 18:34:40 | cdent | mriedem: right, which isn’t predicted, so that one is left off | |
| 18:35:07 | cdent | but when it turns into a problem, is fixable | |
| 18:35:31 | mriedem | stephenfin: question in https://review.openstack.org/#/c/526329/ | |
| 18:35:41 | mriedem | _ix: you have to run the discover_hosts CLI | |
| 18:35:47 | mriedem | nova-manage cell_v2 discover_hosts --verbose | |
| 18:35:52 | _ix | I have. It says there aren't any unmapped hosts. | |
| 18:36:05 | mriedem | what does nova service-list show? | |
| 18:36:11 | mriedem | does the entry show up there? | |
| 18:36:44 | _ix | For nova compute? Yes. | |
| 18:37:08 | _ix | This is just 1/5 of the hosts not showing up. | |
| 18:37:20 | _ix | But, I might be misleading a bit. I see there's an rpc problem on the host. | |
| 18:37:31 | mriedem | what does nova hypervisor-list show? | |
| 18:37:40 | mriedem | nova host-list is deprecated since pike btw | |
| 18:37:53 | smatzek | The Trove gates started failing on Friday or Saturday, right about the time the new attach flows landed. In particular they are failing when resizing or rebuilding instances with attached volumes. Does Nova have gate tests that do this? | |
| 18:38:11 | openstackgerrit | Dmitry Tantsur proposed openstack/nova master: Re-use existing ComputeNode on ironic rebalance https://review.openstack.org/508555 | |
| 18:38:14 | dtantsur | mriedem: I hope I did not mess up anything ^^^ | |
| 18:38:17 | mriedem | smatzek: yes for resize with volume attached - pretty sure anyway | |
| 18:38:24 | mriedem | smatzek: not sure about rebuild with a volume attached (in tempest anyway) | |
| 18:38:51 | _ix | mriedem: I generally use the openstack client. Is `openstack host list` also deprecated? | |
| 18:39:49 | mriedem | smatzek: test_resize_volume_backed_server_confirm | |
| 18:39:51 | mriedem | in tempest | |
| 18:40:07 | smatzek | mriedem: resize is failing like this, which in looking at the code means that connection_info is missing the driver type, and in looking at the previous log entries I can see that connection_info is indeed missing it: http://logs.openstack.org/39/524539/1/check/legacy-trove-functional-dsvm-mysql/3ef681d/logs/screen-n-cpu.txt.gz#_Dec_11_03_03_28_755007 | |
| 18:40:43 | mriedem | _ix: i deprecated nova host-list and didn't update openstack client so it's probably not deprecated | |
| 18:41:25 | _ix | Cool. I see that the host is disabled in the `nova hypervisor-list` Thanks for the help. Odd. | |
| 18:41:44 | mriedem | _ix: starting in pike, | |
| 18:41:46 | smatzek | rebuild, which I haven't dug as deep into today, is failing a bit more nuanced way. The mount point it is getting from either cinder or nova is giving /dev/na, as in "not applicable" which is a default for some fields in Cinder. | |
| 18:41:55 | mriedem | nova-compute will auto-disable the host if it's had 10 failed build attempts | |
| 18:41:58 | mriedem | in a row | |
| 18:42:57 | openstackgerrit | Andrey Volkov proposed openstack/osc-placement master: CLI for allocations https://review.openstack.org/457534 | |
| 18:42:57 | _ix | Oh, hey. That's a fine tip. Thanks again! I'm sure that's what's going on here. | |
| 18:42:58 | openstackgerrit | Andrey Volkov proposed openstack/osc-placement master: CLI for aggregates (v1.1) https://review.openstack.org/505643 | |
| 18:42:58 | openstackgerrit | Andrey Volkov proposed openstack/osc-placement master: CLI for usages https://review.openstack.org/457535 | |
| 18:42:59 | openstackgerrit | Andrey Volkov proposed openstack/osc-placement master: RP list: member_of and resources parameters (v1.3, v1.4) https://review.openstack.org/511183 | |
| 18:42:59 | openstackgerrit | Andrey Volkov proposed openstack/osc-placement master: CLI for resource classes (v1.2) https://review.openstack.org/511182 | |
| 18:43:00 | openstackgerrit | Andrey Volkov proposed openstack/osc-placement master: CLI for traits (v1.6) https://review.openstack.org/514643 | |
| 18:43:00 | openstackgerrit | Andrey Volkov proposed openstack/osc-placement master: RP delete inventories (v1.5) https://review.openstack.org/514642 | |
| 18:43:01 | openstackgerrit | Andrey Volkov proposed openstack/osc-placement master: Usages per project and user (v1.8, v1.9) https://review.openstack.org/514646 | |
| 18:43:01 | openstackgerrit | Andrey Volkov proposed openstack/osc-placement master: Resource class set (v1.7) https://review.openstack.org/514644 | |
| 18:43:02 | openstackgerrit | Andrey Volkov proposed openstack/osc-placement master: CLI allocation candidates (v1.10) https://review.openstack.org/514647 | |
| 18:43:15 | _ix | Also, a side note. That's a super handy feature from my previous deployments. | |
| 18:43:20 | mriedem | _ix: there is a config option if you want to change that | |
| 18:43:29 | mriedem | but yeah, ^ is why it's on by default | |
| 18:43:30 | _ix | I don't. I think that's great. | |
| 18:43:35 | mriedem | dansmith: ^ high praise | |
| 18:44:11 | smatzek | mriedem: the tempest test test_resize_volume_backed_server_confirm is doing resize with a volume backed / BFV instance. The trove test is different, the instance has ephemeral disk and a volume attached. | |
| 18:44:42 | mriedem | smatzek: hmm http://logs.openstack.org/39/524539/1/check/legacy-trove-functional-dsvm-mysql/3ef681d/logs/screen-n-cpu.txt.gz#_Dec_11_03_03_28_499795 | |
| 18:45:02 | mriedem | {"attachment": {"status": "reserved", "detached_at": "", "connection_info": {}, "attached_at": "", "attach_mode": null, "instance": "85cdb482-63a5-487a-b103-95b9383ffcc7", "volume_id": "b3c82217-38f4-4c1f-b0f0-486a4fda6dda", "id": "848654a4-9647-46b8-9485-b3647fe87cfb"}} | |
| 18:45:03 | _ix | mriedem: Can you offer any conjecture about kernel versions, etc? We're on Centos 7.4, and upgraded to kernel 4.14. It looks like the rpcbind.socket is failing us. | |
| 18:45:13 | mriedem | _ix: nope | |
| 18:45:39 | mriedem | smatzek: that empty connection_info is presumably the problem | |
| 18:45:49 | mriedem | because nova pulls the driver_volume_type out of the connection_info on the attachment | |
| 18:46:00 | mriedem | smatzek: although, | |
| 18:46:11 | mriedem | i think that's normal for a 'reserved' attachment maybe? we'd have to ask jgriffith | |
| 18:46:26 | mriedem | until nova updates the attachment with the host connector, the attachment doesn't have a connection_info | |
| 18:46:39 | _ix | Will the openstack client one day be the only client? | |
| 18:46:48 | smatzek | yep, I came to the same conclusion, that the empty connection_info coming back from Cinder is part of the issue, but was drawing a blank on if it was suppose to be empty there or not. | |
| 18:47:06 | ildikov | mriedem: if it's only reserved connection_info is empty | |
| 18:47:18 | _ix | (That is, the openstack cli client) | |
| 18:47:26 | mriedem | _ix: at this rate i doubt it | |
| 18:47:36 | _ix | That's a sad story. | |
| 18:47:38 | mriedem | _ix: been talking about that for years now | |
| 18:47:43 | smatzek | To my knowledge Trove hasn't had changes here and it just started about when the Nova code landed so I thought I'd check. Also, it's not "mainline". Regular attach works and only resize and rebuild are having issues afaik. | |
| 18:48:04 | mriedem | _ix: would need to get feature parity in osc | |
| 18:48:08 | mriedem | _ix: and no one is working on that | |
| 18:48:15 | dansmith | mriedem: nice | |
| 18:48:18 | _ix | Forgive me, what is osc? | |
| 18:48:25 | mriedem | _ix: we could make that a priority, but that means de-prioritizing other work | |
| 18:48:28 | mriedem | openstack client | |
| 18:48:34 | mriedem | OpenStackClient | |
| 18:48:36 | _ix | Oh, I see. | |
| 18:49:48 | ildikov | smatzek: I need to jump on a call, but will look into the logs once I got off the phone | |
| 18:50:34 | ildikov | smatzek: the new flow shouldn't blow those calls up, but tempest might lack some coverage, I don't think we did a thorough manual testing :( | |
| 18:51:01 | mriedem | hmm | |
| 18:51:02 | mriedem | 'block_device_mapping': [{'guest_format': None, 'boot_index': -1, 'mount_device': u'/dev/vdb', 'connection_info': {'serial': u'b3c82217-38f4-4c1f-b0f0-486a4fda6dda'}, 'disk_bus': u'virtio', 'device_type': u'disk', 'attachment_id': '848654a4-9647-46b8-9485-b3647fe87cfb', 'delete_on_termination': True}] | |
| 18:53:48 | mriedem | smatzek: can you open a bug, the logstash query does show this starting on the 9th | |
| 18:53:58 | mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22VolumeDriverNotFound%3A%20Could%20not%20find%20a%20handler%20for%20None%20volume.%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22&from=7d | |
| 18:54:09 | mriedem | *8th actually | |
| 18:56:16 | smatzek | mriedem: will do, and then I'm going to look into the Trove upgrade failure, which uses Nova rebuild a bit more. The stack/failure there is in Trove code but it has that bad /dev/na mountpoint which it fetched from an API (I don't recall which without re-digging) | |
| 18:57:10 | ildikov | I don't recall seeing this issue earlier | |
| 18:58:07 | ildikov | we should have connection_info by the time we get to the _connect_volume call, I need to look into what went wrong there | |
| 19:02:51 | smatzek | mriedem: https://bugs.launchpad.net/nova/+bug/1737599 | |
| 19:02:52 | openstack | Launchpad bug 1737599 in OpenStack Compute (nova) "Instance resize with attach volume fails" [Undecided,New] | |
| 19:03:38 | mriedem | i can see what's different about this flow from the bfv one | |
| 19:04:19 | mriedem | we seem to lose the connection_info during some of the much complicated bdm object -> driver bdm -> dict block_device_info conversions... | |
| 19:04:28 | mriedem | or, it was just never set in the first place | |
| 19:04:35 | mriedem | my guess is the latter | |
| 19:05:08 | mriedem | i can definitely see where it blows up though yeah | |
| 19:10:51 | openstackgerrit | Ed Leafe proposed openstack/nova master: Change RPC for select_destinations() https://review.openstack.org/516707 | |
| 19:10:52 | openstackgerrit | Ed Leafe proposed openstack/nova master: Make conductor pass and use host_lists https://review.openstack.org/511358 | |
| 19:10:52 | openstackgerrit | Ed Leafe proposed openstack/nova master: Move the claim_resources method to scheduler utils https://review.openstack.org/511357 | |
| 19:10:53 | openstackgerrit | Ed Leafe proposed openstack/nova master: Change compute RPC to use alternates for resize https://review.openstack.org/526436 | |
| 19:11:25 | edleafe | mriedem: cleaned up ^^ | |
| 19:18:49 | openstackgerrit | Chris Dent proposed openstack/nova master: [placement] Add cache headers to placement api requests https://review.openstack.org/521640 | |
| 19:18:50 | openstackgerrit | Chris Dent proposed openstack/nova master: [placement] Add info about last-modified to contrib docs https://review.openstack.org/526084 | |
| 19:20:15 | cdent | mriedem, edleafe ^ that fixes the several suggestions matt had | |