Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-02-07
18:03:41 claudiub|2 as for the nova-compute logs, before I deleted the service from the nova cell and the service-list and restarted, nothing. afterwards, indeed: ResourceProviderCreationFailed: Failed to create resource provider NUCHV12R202
18:04:10 mriedem "openstack resource provider list" should list resource providers
18:04:13 mriedem and not require an id
18:04:16 mriedem it's a GET /resource_providers call
18:04:24 claudiub|2 mriedem: and i have the placement section in nova.conf
18:04:25 mriedem openstack resource provider show <id> would require an rp uuid
18:04:34 mriedem so figure out why creation of the RP falied
18:04:36 mriedem *failed
18:04:48 mriedem claudiub|2: welcome new contributor, you know you should be in #openstack yeah? :P
18:05:30 claudiub|2 hm, the compute node is in the openstack resource provider list
18:05:56 claudiub|2 well, this is #openstack, it only has a suffix as well. :D
18:07:09 openstackgerrit Matt Riedemann proposed openstack/nova master: Workaround glanceclient bug when CONF.glance.api_servers not set https://review.openstack.org/541008
18:07:16 mriedem claudiub|2: then "openstack resource provider inventory show <rp_uuid>"
18:07:27 mriedem does that show vcpu/memory_mb/disk_gb?
18:10:06 claudiub|2 it does. :/
18:10:55 claudiub|2 anyways gonna check why the exception occurs for now
18:11:18 openstackgerrit Matt Riedemann proposed openstack/nova master: Move the nova-next job in-tree and update it https://review.openstack.org/541474
18:11:19 openstackgerrit Matt Riedemann proposed openstack/nova master: Test websocketproxy with TLS in the nova-next job https://review.openstack.org/513160
18:11:53 mriedem claudiub|2: you can also run, i think, "openstack resource provider show --allocations <rp_uuid>" to see if placement thinks there are instances on the node still
18:12:05 mriedem might need to check the CLI help for the proper command option
18:12:25 dansmith mriedem: these compute rpc bumps are passing now and ready for you to sh*t on them: https://review.openstack.org/#/c/541005
18:12:44 mriedem dansmith: let me go eat some vietnamese quick and work up a healthy dump
18:13:13 melwitt o.O
18:13:14 dansmith um, thanks?
18:13:22 mriedem you wanted me to shit on them
18:13:24 mriedem i need ammo
18:13:31 dansmith no, I don't want you to, I just know you will
18:13:50 mriedem do'nt act like you're not into it
18:14:00 claudiub|2 hm, a lot of allocations it seems.
18:14:18 mriedem claudiub|2: ok, so something blew up on instance delete (or something) and allocations aren't getting cleaned up
18:14:20 claudiub|2 even though i only have 1 instance
18:40:38 stvnoyes mriedem: create multiple servers with a bdm bug is filed - https://bugs.launchpad.net/nova/+bug/1747985
18:40:39 openstack Launchpad bug 1747985 in OpenStack Compute (nova) "Create Multiple Servers fails when a volume is specified" [Undecided,New]
18:41:53 stvnoyes mriedem: I going on to see what else I hit. it doesn't just work once that check is removed. fyi
19:29:51 hrw can someone review https://review.openstack.org/#/c/541728/ and decide is it worth second +2?
19:51:21 cdent mriedem: did you see this bit of my FML: https://review.openstack.org/#/c/541832/
20:01:08 openstackgerrit Matt Riedemann proposed openstack/nova master: XenAPI: Provide support matrix and doc for VGPU https://review.openstack.org/540808
20:03:38 mriedem cdent: nope
20:03:41 mriedem just got back from lunch
20:03:48 cdent such shame
20:04:53 mriedem oh i see it's your fault
20:04:54 mriedem geez
20:05:27 cdent mriedem: but once I fixed that I started running with 20 hypervisors and launching 1000s of instances. ran into the rpc timeouts you did. rabbit is a pig, the conductors eats all the cpu, placement does not sweat
20:05:32 cdent yeah, my bad, three years ago
20:05:51 mriedem cdent: rpc timeouts for select_destinations?
20:06:05 cdent hold, lemme check
20:06:37 mriedem that's what i was hitting at sone point, b/c select_destinations is an rpc call, and when you're booting 1000 instances at once, we timeout on that call
20:06:40 dansmith cdent: I think devstack by default configures few workers per core which is unlike a real deployment, so with thousands of real builds you'll be pushing pretty hard on a small system
20:06:55 mriedem devstack is i think ncpu / 2
20:06:56 mriedem for each service
20:07:00 mriedem that has multiple workers
20:07:16 dansmith yeah
20:07:22 mriedem if you did ncpu workers per service, which devstack used to do, you get OOM
20:07:25 dansmith right
20:08:20 cdent yes in select_destinations
20:08:48 mriedem yeah for that issue i had put some thoughts in https://review.openstack.org/#/c/510235/
20:08:56 mriedem but very much quarter baked
20:09:48 cdent dansmith: I'm multinode, using the fake driver, and I'm in the process of now of moving all the nova-compute action to a separate host. the control plane host is 16GB and 16 cores
20:10:07 cdent rabbit is eating a _ton_ of ram, followed by mysql
20:10:08 dansmith ah fake driver, well, that might be different
20:10:28 dansmith cdent: do you have notifications disabled?
20:10:38 cdent no, but that's an excellent idea
20:10:39 dansmith cdent: otherwise I think you're just filling a queue with no consumer
20:10:50 cdent thanks
20:10:58 dansmith thank me if/when it matters :P
20:11:29 mriedem that's something i've been wondering about benchmarking lately,
20:11:34 mriedem with and without versioned notifications enabled
20:11:44 mriedem by default, we have both versioned and unversioned notifications enabled,
20:11:56 mriedem and versioned notifications do a lot of lazy-loading of fields to build the notification payload
20:11:58 cdent well it's a good idea nonetheless as I'm trying to remove as much stuff I don't care about™ as possible
20:12:06 cdent and at the moment, notifications are basically noise
20:12:20 mriedem yeah - take a note of what impact disabling that makes if you can
20:12:39 mriedem i.e. we might want to consider changing the default
20:14:49 mriedem looks like we might have a rogue unit test http://logs.openstack.org/74/541474/2/check/openstack-tox-py27/a6dfdf8/testr_results.html.gz
20:16:29 mriedem and another one http://logs.openstack.org/08/541008/5/check/openstack-tox-py27/c3a8321/testr_results.html.gz
20:16:29 mriedem gdi
20:24:41 mriedem mordred: apparently that glanceclient.common.utils.strip_version thing isn't working here https://review.openstack.org/#/c/541008/
20:26:03 mordred mriedem: looking
20:26:38 mordred 541008
20:26:40 mordred gah
20:29:43 mordred mriedem: well, at least if fails in unittests - that's nice
20:29:44 mriedem mordred: i see the problem
20:29:56 mriedem it doesn't expect the 'image' prefix on the path
20:30:04 mriedem >>> url = 'http://23.253.94.203/v2'
20:30:04 mriedem >>> url_parts = urlparse.urlparse(url)
20:30:04 mriedem >>> url_parts
20:30:04 mriedem ParseResult(scheme='http', netloc='23.253.94.203', path='/v2', params='', query='', fragment='')
20:30:04 mriedem >>> utils.strip_version(url)
20:30:05 mriedem ('http://23.253.94.203', 2.0)
20:30:05 mriedem >>>
20:30:57 mordred oh. bother. yeah
20:37:00 mriedem https://bugs.launchpad.net/python-glanceclient/+bug/1748009
20:37:01 openstack Launchpad bug 1748009 in Glance Client "glanceclient.common.utils.strip_version doesn't handle a service type in the path" [Undecided,New]
20:37:18 mordred mriedem: update coming for ya
20:38:16 mriedem here is my hacktastic workaround
20:38:17 mriedem >>> url.replace('image/', '')
20:38:17 mriedem 'http://23.253.94.203/v2'
20:38:17 mriedem >>> url = url.replace('image/', '')
20:38:17 mriedem >>> utils.strip_version(url)
20:38:17 mriedem ('http://23.253.94.203', 2.0)

Earlier   Later