Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-12-06
15:30:40 openstack Launchpad bug 1806239 in OpenStack Compute (nova) "nova-api should handle type conversion while creating server snapshots " [Undecided,New]
15:30:40 mriedem then i don't know why tempest would not fail on https://bugs.launchpad.net/nova/+bug/1806239
15:31:07 mordred nova is still using glanceclient right?
15:31:12 mriedem yeah
15:31:33 mordred yeah - tempest uses direct rest calls - so it's possible glanceclient is doing something wrong. or tempest is doing something wrong
15:32:29 mriedem well, tempest would just create a server and tell nova to snapshot it
15:32:33 mriedem and then nova will use glanceclient
15:32:52 mriedem if that's all it takes to tickle this with rocky glance, i'm not sure why tempest wouldn't blow up
15:33:37 mriedem anyway, i'll wait for diliprenkila to provide recreate steps
15:33:59 mordred oh. gotcha
15:34:05 mordred yeah. that's super weird
15:35:10 ShilpaSD Hi All, facing issue while stack, E: Sub-process /usr/bin/dpkg returned an error code (1), any suggestions to resolve this?
15:37:28 dansmith mriedem: so on that rpc logging thing at startup, do we think the actual query is slowing things down, or the logging of the pointless message?
15:37:47 mriedem idk
15:38:26 mriedem looking at logs
15:39:20 mriedem well we take 2 seconds dumping our gd config options :)
15:39:35 dansmith which should be mostly just log traffic, right?
15:39:40 mriedem yeah
15:39:41 dansmith so maybe the logging is the thing
15:39:42 dansmith also
15:39:44 mriedem we start loading extensions at Dec 05 20:14:00.919520
15:40:03 dansmith you know that if we were to actually start compute first, we would cache the service version the way we want and avoid the multiple lookups
15:40:24 mriedem looks like we're done loading extensions at Dec 05 20:14:27.718587
15:40:43 mriedem so there is another thing here,
15:40:49 mriedem the rpcapi client does the version query thing,
15:41:04 mriedem Dec 05 20:14:27.687766 ubuntu-xenial-ovh-bhs1-0000959981 devstack@n-api.service[23459]: DEBUG oslo_concurrency.lockutils [None req-dfdfad07-2ff4-43ed-9f67-2acd59687e0c None None] Lock "placement_client" acquired by "nova.scheduler.client.report._create_client" :: waited 0.000s {{(pid=23462) inner /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:327}}
15:41:04 mriedem but API also constructs a SchedulerReportClient per instance, which apparently uses a lock
15:41:49 mriedem so...
15:41:55 dansmith I also just noticed/remembered that the multi-cell version of this does not cache at all
15:41:57 mriedem and that's an in-memory lock
15:44:28 mriedem heh remember how efried_cya_jan removed that lazy-load scheduler report client stuff? https://github.com/openstack/nova/blob/master/nova/compute/api.py#L256
15:44:45 dansmith yeah
15:44:51 efried_cya_jan uh oh
15:45:17 dansmith mriedem: is there a warn-once pattern I should be using for logs?
15:46:23 mriedem dansmith: i think just a global
15:46:31 dansmith ack
15:46:48 mriedem i'll strip out a separate bug for this report client init thing
15:51:01 openstack Launchpad bug 1807219 in OpenStack Compute (nova) "SchedulerReporClient init slows down nova-api startup" [Medium,Triaged]
15:51:01 mriedem https://bugs.launchpad.net/nova/+bug/1807219
15:52:22 cdent mdbooth: is that ^ related at all to the slow down you were seeing in your explorations yesterday(?)?
15:53:20 efried_cya_jan mriedem: We ought to singleton that guy. If we're not having caching conflicts, it can only be out of luck because the API obj isn't doing anything that touches the cache.
15:53:47 edleafe efried_cya_jan: more like efried_big_fat_liar
15:54:45 efried_cya_jan I'm here for like another 20 minutes today. This does two things: 1) keeps my inbox down to triple digits; 2) makes you scared to talk about me behind my back.
15:55:03 mriedem i intentionally talked about you to summon you
15:55:06 mdbooth cdent: Only if we also do this during server create. Possible. The number of API objects created was insane.
15:55:08 kaisers_ stephenfin: mdbooth: Updated https://review.openstack.org/#/c/554195/ as discussed yesterday
15:55:15 edleafe Oh, it's more fun to talk about you to your face!
15:56:23 efried_cya_jan mriedem: Is the removal of lazy-load causing (or even related to) that one-reportclient-per-API?
15:57:59 mriedem efried_cya_jan: not sure, we might have always been loading it during API init
15:58:41 mriedem but i think it would have done the lazy-load
15:58:52 mriedem since LazyLoader only creates the client thing until something is accessed on it
15:59:10 efried_cya_jan And it's only expensive because it's serialized, not because the report client is doing anything heavy, right?
15:59:29 mriedem yeah i think so
16:00:00 mriedem https://github.com/openstack/nova/blob/c9dca64fa64005e5bea327f06a7a3f4821ab72b1/nova/scheduler/client/report.py#L260
16:00:21 mriedem we create 2 provider trees in there for some reason...
16:00:26 mriedem https://github.com/openstack/nova/blob/c9dca64fa64005e5bea327f06a7a3f4821ab72b1/nova/scheduler/client/report.py#L270
16:00:31 mriedem https://github.com/openstack/nova/blob/c9dca64fa64005e5bea327f06a7a3f4821ab72b1/nova/scheduler/client/report.py#L279
16:00:35 mriedem oops
16:00:40 mriedem https://github.com/openstack/nova/blob/c9dca64fa64005e5bea327f06a7a3f4821ab72b1/nova/scheduler/client/report.py#L286
16:01:15 mriedem is there any good reason we need to do that in both places?
16:02:33 efried_cya_jan mriedem: There should never be a need for one process to create two separate provider trees <=> report client instances, period.
16:02:48 efried_cya_jan That could only ever lead to bugs.
16:02:57 efried_cya_jan It will only ever *not* lead to bugs by luck.
16:03:12 efried_cya_jan hold on, looking at your links...
16:03:24 mriedem so, i'm going to remove this https://github.com/openstack/nova/blob/c9dca64fa64005e5bea327f06a7a3f4821ab72b1/nova/scheduler/client/report.py#L286-L287
16:03:35 mriedem and change https://github.com/openstack/nova/blob/c9dca64fa64005e5bea327f06a7a3f4821ab72b1/nova/scheduler/client/report.py#L270-L272
16:03:39 mriedem to just call clear_provider_cache
16:03:45 mriedem for great success
16:03:47 mriedem ok?!
16:03:56 mriedem butthole surfers got me all amped up
16:05:18 efried_cya_jan mriedem: That will be fine, it's removing redundant calls, but it's not going to change anything functionally. We were calling ProviderTree init twice before, but not creating two separate provider trees, just overwriting self._provider_tree.
16:06:06 efried_cya_jan mriedem: However, I don't agree with the details. You can't get rid of _create_client without affecting @safe_connect. For now (until @safe_connect has died a fiery death), I would just remove the provider tree and assoc timer inits from __init__.
16:06:39 efried_cya_jan And if you would like to take a step on the road to that fiery demise of @safe_connect, you could review https://review.openstack.org/#/c/613613/ :P
16:07:52 mriedem efried_cya_jan: safe_connect doesn't care about provider tree and association refresh
16:08:01 mriedem only the ksa client thing
16:08:10 melwitt o/
16:08:49 efried_cya_jan mriedem: reason we want to clear the cache in safe_connect is because it happened due to an error, so we can't count on our cache being correct.
16:09:29 efried_cya_jan I sense you're trying to find a way to remove the use of that semaphore. I don't think we can do that.
16:09:36 mriedem i'm not
16:09:48 mriedem i'm just trying to get rid of this code creating the provider tree data structure 20 times in different places
16:09:51 mriedem because i'm OCD
16:10:00 mriedem but i hear you
16:10:20 efried_cya_jan What you can do is make _create_client call clear_provider_cache, and remove those inits from __init__. That ought to consolidate the actual LOCs that do the init to one place.
16:10:26 efried_cya_jan I think.
16:10:51 efried_cya_jan but hm, this makes me wonder whether clear_provider_cache needs to be under that same lock.
16:10:59 efried_cya_jan (for the other cases where it's called)
16:11:16 efried_cya_jan I'll leave that steaming pile in front of you and walk away.
16:15:36 efried_cya_jan kashyap: Yeah, right about... now.
16:15:37 efried_cya_jan o/
16:15:45 kashyap (That's good; happy to see screen-starers getting breaks...)
16:15:54 sean-k-mooney efried_cya_jan: enjoy the break
16:15:58 kashyap efried_cya_jan: Glad; don't make the mistake of staying connected to the VPN :D
16:29:23 mriedem heh putting the same lock on clear_provider_cache makes the tests lock up
16:32:53 openstackgerrit Matt Riedemann proposed openstack/nova master: DRY up SchedulerReportClient init https://review.openstack.org/623247
16:32:53 openstackgerrit Matt Riedemann proposed openstack/nova master: Only construct SchedulerReportClient on first access from API https://review.openstack.org/623246
16:33:58 mriedem someone should probably tell the intel nfv ci to stop reporting b/c it's busted
16:34:01 mriedem and has been for awhile
16:40:58 mnaser https://wiki.openstack.org/wiki/ThirdPartySystems/Intel_NFV_CI
16:41:08 mnaser looks like it's in wip or something
16:43:52 gibi mriedem: my organized thoughts about the legacy notification deprecation http://lists.openstack.org/pipermail/openstack-discuss/2018-December/000685.html

Earlier   Later