Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-05-17
16:56:34 melwitt okay. well, with cells v1, wouldn't compute be reporting updates to the scheduler and then it wouldn't build that dict?
16:56:42 mriedem belmorei_: was it disabled when you were doing cells v1?
16:56:50 dansmith melwitt: that's what I just said above
16:56:59 bauzas melwitt: dansmith: catching up the convo, what's the problem with computes updating the scheduler ?
16:57:05 dansmith but I thought we still had to hit the db once per host even with that enbabled
16:57:09 bauzas there is a fanout upcall
16:57:10 melwitt ? so doesn't that make it obvious why it was fast with cells v1 and slow with cells v2?
16:57:31 melwitt because with cells v2 it's building the dict every time in the absence of updates? I'm confused
16:57:32 dansmith melwitt: I had asked him if they had it enabled
16:57:46 dansmith melwitt: affinity doesn't work with cellsv1 so I would have expected they didn't have that enabled
16:57:50 dansmith (afaik)
16:58:03 melwitt okay, I see
16:58:09 dansmith and, I thought we still hit the db once per host even with it enabled for the compute info, but not for the instance info part
16:58:15 dansmith so I wouldn't expect a lage difference
16:58:16 melwitt it probably works for them by way of patches, maybe
16:58:17 dansmith *large
16:58:23 dansmith well, then they're cheating :)
16:59:05 melwitt what's the compute info part?
17:00:01 melwitt the normal host states stuff?
17:00:13 dansmith yeah
17:01:07 belmorei_ ok, we will continue to dig on this
17:01:13 belmorei_ thanks
17:01:29 belmorei_ I'm leaving now
17:01:59 mriedem random thinking out loud, i think the instance info list only uses the uuid from the instances for the affinity filters,
17:02:07 melwitt belmorei_: seeya next week
17:02:15 mriedem we could probably use placement to just pull uuids of consumers that have allocations on a given node and replace that
17:02:29 mriedem would be a lot fast
17:02:31 mriedem *faster
17:02:39 dansmith well, it depends on which piece of his linked function is taking a while
17:02:46 dansmith we do hit the db for the hosts every time,
17:02:55 dansmith but the instance list is the thing we would skip with track_changes
17:02:59 dansmith so yeah, maybe it's just that
17:03:09 mriedem https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L753 could also avoid some default joins...
17:03:16 dansmith still once call to placement per instance would be not great
17:03:24 dansmith er per compute
17:03:27 mriedem i meant per compute
17:03:43 mriedem right now it looks like this does a full instance list per host including default joined tables
17:03:50 mriedem which is i think info_cache and security_groups
17:03:55 mriedem and all the affinity filters care about is the uuid
17:03:59 dansmith yeah probably
17:04:04 melwitt good find
17:04:13 mriedem https://blueprints.launchpad.net/nova/+spec/put-host-manager-instance-info-on-a-diet
17:04:19 mriedem maybe i should have worked on this blueprint some more
17:05:16 dansmith it would be even better to just have a single db call that would take the compute uuids, and return a dict of {host: [uuid, ]}
17:05:48 dansmith although it does have a full instance object in the result there.. are we sure that's not used?
17:05:59 mriedem pretty sure, read that blueprint
17:06:20 mriedem it used to also pull the instance.instance_type_id for the TypeAffinityFilter, but we removed that filter
17:06:26 mriedem s/pull/use/
17:06:39 dansmith yeah, but any filter out there that expects a full list of instances in the host state will care
17:06:46 mriedem sure, but.
17:06:50 mriedem f'em
17:07:11 dansmith we could put fake things in there that would lazy-load if they touch them :)
17:08:02 dansmith but okay yeah, that bp text makes sense
17:08:07 mriedem so if you have a filter that cares about the network info cache, and we stop auto-joining on that by default, you'd start lazy-loading and see a perf hit during scheduling
17:08:11 dansmith (he says noting he's the approver)
17:08:20 dansmith yeah
17:08:36 dansmith makes it look like their fault though! :)
17:08:40 mriedem but at that point, you either (1) live with it (2) fork the host manager code (3) upstream your filter
17:09:16 dansmith it'd be nice to get some clarification from them that this is indeed where their hotspot is (grabbing that large instance list)
17:10:38 mriedem yeah and they are both gone now
17:10:41 mriedem tired of our antics
17:11:00 openstackgerrit Matt Riedemann proposed openstack/nova master: Update instance.availability_zone during live migration https://review.openstack.org/567701
17:14:46 melwitt did y'all look at the bug I linked earlier? https://bugs.launchpad.net/nova/+bug/1737465
17:14:47 openstack Launchpad bug 1737465 in OpenStack Compute (nova) "[cellv2] the performance issue of cellv2 when creating 500 instances concurrently" [Undecided,Confirmed] - Assigned to Jiang (jiangpf)
17:14:58 dansmith yeah
17:15:52 dansmith I'm sure it's the same thing, but I can't really make sense of most of the text in there
17:16:00 melwitt okay, I was thinking they're hitting the same thing in _get_instance_info and that's doing the instance list, it doesn't do anything else
17:16:41 melwitt yeah, it's not easy to read but they did point out it was the call to _get_instance_info where they get the slowdown
17:17:12 dansmith yeah, that's why I say I'm sure it's the same
17:17:29 mriedem "Because cellv2 have disable instance info cache." - i assume that means they disabled track_instance_changes
17:17:55 melwitt yeah, me too. so you're thinking there could possibly be something else in _get_instance_info that could cause a slowdown other than the instance list, when you said it would be good if they could clarify?
17:18:24 dansmith no, I'm just saying it would be good to clarify, since he linked at a rather high-up spot
17:18:29 melwitt mriedem: I think it means because the upcall is disallowed by default, yeah?
17:18:39 melwitt dansmith: oh, okay
17:18:45 mriedem melwitt: no
17:18:58 mriedem track_instance_changes is on by default,
17:19:07 mriedem if he's using devstack, then yes that upcall is disabled
17:19:21 mriedem https://docs.openstack.org/nova/latest/configuration/config.html#workarounds.disable_group_policy_check_upcall is not disabled by default though
17:19:30 melwitt but with cells v2 it won't cache it, because the upcall is disabled by default, right? oh nevermind
17:19:34 mriedem so out of the box brand new install that's not devstack is going to be doing the rpc blast upcall
17:19:37 mriedem no
17:19:48 mriedem has nothing to do with cells v2
17:21:00 melwitt yeah, I was thinking maybe that upcall was disabled by default because of cells v2. but it's not
17:21:16 dansmith it just won't ever make it to the scheduler
17:21:25 mriedem if you're doing split MQ
17:21:41 mriedem i'd be kind of surprised if someone was following that setup if they didn't know what they were doing
17:21:44 melwitt ohh...
17:24:23 openstackgerrit Matt Riedemann proposed openstack/nova master: Metadata-API fails to retrieve avz for instances created before Pike https://review.openstack.org/567878
17:32:17 dansmith edleafe: hmm, so I'm a bit confused actually
17:45:16 edleafe dansmith: do tell
17:46:12 dansmith edleafe: something is off by one, one way or the other and I'm not sure I get it
17:46:24 dansmith either the test is wrong or needs some other tweak, or
17:46:26 dansmith something
17:46:39 edleafe I have a few minutes - let me look
17:46:41 dansmith edleafe: anyway, I've been working on it in parallel to other things and I think I've confused myself out of the way
17:46:50 dansmith so yeah, maybe you should look without me telling you what I'm confused about :)
17:47:15 edleafe I'd much rather confuse myself :)
17:56:05 edleafe dansmith: Looks like this needs to be adjusted, too: https://github.com/openstack/nova/blob/master/nova/scheduler/filter_scheduler.py#L367
17:56:22 dansmith edleafe: yeah, I did that
17:56:37 dansmith and I can make the tests pass at that point,

Earlier   Later