| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-02-07 | |||
| 20:05:32 | cdent | yeah, my bad, three years ago | |
| 20:05:51 | mriedem | cdent: rpc timeouts for select_destinations? | |
| 20:06:05 | cdent | hold, lemme check | |
| 20:06:37 | mriedem | that's what i was hitting at sone point, b/c select_destinations is an rpc call, and when you're booting 1000 instances at once, we timeout on that call | |
| 20:06:40 | dansmith | cdent: I think devstack by default configures few workers per core which is unlike a real deployment, so with thousands of real builds you'll be pushing pretty hard on a small system | |
| 20:06:55 | mriedem | devstack is i think ncpu / 2 | |
| 20:06:56 | mriedem | for each service | |
| 20:07:00 | mriedem | that has multiple workers | |
| 20:07:16 | dansmith | yeah | |
| 20:07:22 | mriedem | if you did ncpu workers per service, which devstack used to do, you get OOM | |
| 20:07:25 | dansmith | right | |
| 20:08:20 | cdent | yes in select_destinations | |
| 20:08:48 | mriedem | yeah for that issue i had put some thoughts in https://review.openstack.org/#/c/510235/ | |
| 20:08:56 | mriedem | but very much quarter baked | |
| 20:09:48 | cdent | dansmith: I'm multinode, using the fake driver, and I'm in the process of now of moving all the nova-compute action to a separate host. the control plane host is 16GB and 16 cores | |
| 20:10:07 | cdent | rabbit is eating a _ton_ of ram, followed by mysql | |
| 20:10:08 | dansmith | ah fake driver, well, that might be different | |
| 20:10:28 | dansmith | cdent: do you have notifications disabled? | |
| 20:10:38 | cdent | no, but that's an excellent idea | |
| 20:10:39 | dansmith | cdent: otherwise I think you're just filling a queue with no consumer | |
| 20:10:50 | cdent | thanks | |
| 20:10:58 | dansmith | thank me if/when it matters :P | |
| 20:11:29 | mriedem | that's something i've been wondering about benchmarking lately, | |
| 20:11:34 | mriedem | with and without versioned notifications enabled | |
| 20:11:44 | mriedem | by default, we have both versioned and unversioned notifications enabled, | |
| 20:11:56 | mriedem | and versioned notifications do a lot of lazy-loading of fields to build the notification payload | |
| 20:11:58 | cdent | well it's a good idea nonetheless as I'm trying to remove as much stuff I don't care about™ as possible | |
| 20:12:06 | cdent | and at the moment, notifications are basically noise | |
| 20:12:20 | mriedem | yeah - take a note of what impact disabling that makes if you can | |
| 20:12:39 | mriedem | i.e. we might want to consider changing the default | |
| 20:14:49 | mriedem | looks like we might have a rogue unit test http://logs.openstack.org/74/541474/2/check/openstack-tox-py27/a6dfdf8/testr_results.html.gz | |
| 20:16:29 | mriedem | and another one http://logs.openstack.org/08/541008/5/check/openstack-tox-py27/c3a8321/testr_results.html.gz | |
| 20:16:29 | mriedem | gdi | |
| 20:24:41 | mriedem | mordred: apparently that glanceclient.common.utils.strip_version thing isn't working here https://review.openstack.org/#/c/541008/ | |
| 20:26:03 | mordred | mriedem: looking | |
| 20:26:38 | mordred | 541008 | |
| 20:26:40 | mordred | gah | |
| 20:29:43 | mordred | mriedem: well, at least if fails in unittests - that's nice | |
| 20:29:44 | mriedem | mordred: i see the problem | |
| 20:29:56 | mriedem | it doesn't expect the 'image' prefix on the path | |
| 20:30:04 | mriedem | >>> url = 'http://23.253.94.203/v2' | |
| 20:30:04 | mriedem | >>> url_parts = urlparse.urlparse(url) | |
| 20:30:04 | mriedem | >>> url_parts | |
| 20:30:04 | mriedem | ParseResult(scheme='http', netloc='23.253.94.203', path='/v2', params='', query='', fragment='') | |
| 20:30:04 | mriedem | >>> utils.strip_version(url) | |
| 20:30:05 | mriedem | ('http://23.253.94.203', 2.0) | |
| 20:30:05 | mriedem | >>> | |
| 20:30:57 | mordred | oh. bother. yeah | |
| 20:37:00 | mriedem | https://bugs.launchpad.net/python-glanceclient/+bug/1748009 | |
| 20:37:01 | openstack | Launchpad bug 1748009 in Glance Client "glanceclient.common.utils.strip_version doesn't handle a service type in the path" [Undecided,New] | |
| 20:37:18 | mordred | mriedem: update coming for ya | |
| 20:38:16 | mriedem | here is my hacktastic workaround | |
| 20:38:17 | mriedem | >>> url.replace('image/', '') | |
| 20:38:17 | mriedem | 'http://23.253.94.203/v2' | |
| 20:38:17 | mriedem | >>> url = url.replace('image/', '') | |
| 20:38:17 | mriedem | >>> utils.strip_version(url) | |
| 20:38:17 | mriedem | ('http://23.253.94.203', 2.0) | |
| 20:40:55 | mordred | mriedem: right - but that's not going to work if you need image in the url | |
| 20:42:41 | mordred | mriedem: I think it's better to just ditch strip_version, lop off a trailing /v2 and be done with it - then we can circle back around and talk about a better plan for all of this in dublin :) | |
| 20:44:01 | mriedem | yeah good point | |
| 20:46:07 | mriedem | >>> import re | |
| 20:46:07 | mriedem | >>> url = 'http://23.253.94.203/image/v2' | |
| 20:46:07 | mriedem | >>> re.sub('v\d+\.?\d*', '', url) | |
| 20:46:07 | mriedem | 'http://23.253.94.203/image/' | |
| 20:46:07 | mriedem | >>> | |
| 20:46:16 | mriedem | \O/ | |
| 20:47:35 | openstackgerrit | Merged openstack/python-novaclient master: Fix listing of instances above API max_limit https://review.openstack.org/534222 | |
| 20:48:10 | dansmith | mriedem: that would match "v2." right? | |
| 20:48:45 | dansmith | and it would also match "myapi-123.novav2.foo.com" | |
| 20:48:48 | mriedem | farq yes it will | |
| 20:49:05 | mriedem | i'm basically copying the existing glanceclient utility http://git.openstack.org/cgit/openstack/python-glanceclient/tree/glanceclient/common/utils.py#n383 | |
| 20:49:19 | mriedem | well, | |
| 20:49:30 | mriedem | you can always use CONF.glance.api_servers (like everyone has had to forever) if you don't like it | |
| 20:49:56 | dansmith | I guess we do this a lot, but re magic replacements on the url we don't control is playing with fire | |
| 20:50:26 | mriedem | i know | |
| 20:50:28 | mriedem | how about | |
| 20:50:28 | mriedem | >>> re.sub('/v\d+\.?\d*', '/', url) | |
| 20:50:28 | mriedem | 'http://23.253.94.203/image/' | |
| 20:50:42 | mriedem | since / is a special character, | |
| 20:50:49 | mriedem | i don't know if it's legit to have myapi-123.nova/v2.foo.com | |
| 20:50:59 | dansmith | it's not | |
| 20:51:11 | mriedem | so is that a smaller fire at least? | |
| 20:51:16 | mriedem | otherwise we have to release note this bug | |
| 20:51:19 | dansmith | so I would group the dot and the decimal and require them together or not at all | |
| 20:51:38 | dansmith | (\.\d+)? | |
| 20:51:51 | dansmith | and require / at the end immediately after the last digit | |
| 20:52:06 | dansmith | so "/v2.0oohhmy/" doesn't match | |
| 20:54:35 | dansmith | naw mean? | |
| 20:54:52 | mriedem | i don't regex well | |
| 20:55:17 | mriedem | >>> re.sub('/v\d+(\.\d+)?', '', 'http://23.253.94.203/image/v2.0') | |
| 20:55:18 | mriedem | 'http://23.253.94.203/image' | |
| 20:55:18 | mriedem | ? | |
| 20:56:01 | dansmith | re.sub('/\d+(\.\d+)?/', '/', url) | |
| 20:56:44 | mriedem | that doesn't work | |
| 20:56:58 | dansmith | hrm yeah | |
| 20:57:59 | openstackgerrit | Merged openstack/nova master: XenAPI: Provide support matrix and doc for VGPU https://review.openstack.org/540808 | |
| 20:58:48 | dansmith | because no trailing slash in your example | |
| 20:59:05 | dansmith | so I guess you can't do that | |
| 20:59:10 | mriedem | yar | |
| 20:59:16 | dansmith | does pinning to $ work because you know it'll be the end? | |