| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-12-06 | |||
| 13:17:24 | mriedem | or are you? | |
| 13:17:34 | tssurya | we model cells as aggregates | |
| 13:17:48 | mriedem | because of https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html#tenant-isolation-with-placement right> | |
| 13:17:48 | tssurya | as in every cell is an aggregate | |
| 13:17:49 | mriedem | ? | |
| 13:17:52 | mriedem | ok | |
| 13:18:04 | tssurya | yea because of the pre-filter | |
| 13:18:07 | mriedem | so, really, | |
| 13:18:22 | jaypipes | mriedem: wait, there's a bug in shelve code?! | |
| 13:18:28 | sean-k-mooney | tssurya: so you then use the teanay affinity filter to map tenants to thos aggreates and therefor to cells? | |
| 13:18:30 | mriedem | jaypipes: working as designed | |
| 13:18:47 | tssurya | sean-k-mooney: yea | |
| 13:18:48 | mriedem | tssurya: if the pre-filter is working, then the max_placement_results probably doesn't need to be so low... | |
| 13:19:01 | jaypipes | mriedem: I only have time to argue with cfriesen today, so I'll wait til he's online. :P | |
| 13:19:12 | sean-k-mooney | mriedem: that is what i was wondering too | |
| 13:19:16 | tssurya | mriedem: surely it need not be as low as 10, but even with it being 10 we have 5 to 8 seconds scheduling time | |
| 13:19:19 | mriedem | because if my project is mapped to a cell with ~200 hosts, then filtering on 200 hosts shouldn't be so bad | |
| 13:19:54 | tssurya | the time it would tkae to gather all those host states , well we are still trying to check iut why its 5 to 8 seconds | |
| 13:20:06 | tssurya | out* | |
| 13:20:17 | sean-k-mooney | tssurya: when you say schduiling are you refering to just the time it take for the filter schduler or the time the vm is in that sate | |
| 13:20:34 | mriedem | tssurya: do you know if this is set on the computes and the scheduler nodes? https://docs.openstack.org/nova/latest/configuration/config.html#filter_scheduler.track_instance_changes | |
| 13:20:44 | tssurya | time for the whole boot until select destinations is done fully | |
| 13:20:48 | jaypipes | mriedem: I think it's pretty obvious I wouldn't support adding disabled flags to provider records, yes? | |
| 13:21:01 | mriedem | jaypipes: yes, i could find you my reply to that | |
| 13:21:08 | mriedem | which was probably not very nice | |
| 13:21:20 | jaypipes | mriedem: yes, I saw it. | |
| 13:21:23 | mriedem | and i hope cfriesen will forgive me | |
| 13:21:33 | jaypipes | mriedem: BTW, it's possible to do something like this: | |
| 13:21:35 | tssurya | mriedem: not sure let me check | |
| 13:21:41 | sean-k-mooney | jaypipes: would you support usign a placement aggrage for disabled compute nodes | |
| 13:22:29 | sean-k-mooney | jaypipes: and then use the new member_of:!<aggreage> feature | |
| 13:22:36 | jaypipes | a) request max_placement_records. get those records. b) do a filter() against those provider UUIDs and the set of disabled compute services (would need to grab the compute node UUID, not the hostname, though), and c) if len(a) < len(b), request more from placement | |
| 13:23:02 | mriedem | tssurya: track_instance_changes enables the computes to rpc broadcast to the schedules the information about the instances running on them (the hosts) and then the scheduler workers cache that information so that during a scheduling request the scheduler doesn't need to iterate all of the hosts (via db) to get their current instance list | |
| 13:23:06 | jaypipes | sean-k-mooney: sure, though forbidden aggregates are not approved spec yet... | |
| 13:23:21 | mriedem | tssurya: it's not recommended for split MQ because the computes would be disconnected from the schedulers (so the cast goes in the trash) | |
| 13:23:49 | mriedem | jaypipes: i've been +2 on that forbidden aggregates spec for a couple of weeks now | |
| 13:23:52 | jaypipes | sean-k-mooney: the solution should remain entirely on the nova side, though, IMHO, which is why I recommend the approach above as a method to take if NoValidHosts is encountered. | |
| 13:23:54 | sean-k-mooney | jaypipes: true but its less of an abuse of the placment api and is just as simple to manage as adding or removing a trait | |
| 13:24:07 | mriedem | jaypipes: what we talked about in channel yesterday was all nova side solutions | |
| 13:24:30 | jaypipes | mriedem: gotta find another +2? do you want me to +2 that since I was the one who came up with the idea? | |
| 13:24:34 | sean-k-mooney | jaypipes: e.g. page in more results if the filter elimidate them | |
| 13:24:50 | mriedem | jaypipes: what you suggested about sounds like paging, but also not great performance wise since we'd have to pull the data from the nova cell dbs to check the disabled status | |
| 13:24:55 | jaypipes | sean-k-mooney: is elimidate a combination of eliminate and intimidate? | |
| 13:24:57 | mriedem | jaypipes: you were +2 on the spec before | |
| 13:24:58 | tssurya | mriedem: its False | |
| 13:25:06 | mriedem | tssurya: ok. you aren't running split MQ right? | |
| 13:25:20 | jaypipes | mriedem: are we talking about the same spec? | |
| 13:25:30 | jaypipes | mriedem: the nova side one or the placement-side one? | |
| 13:25:31 | mriedem | https://review.openstack.org/#/c/603352/ | |
| 13:25:46 | mriedem | ^ is the placement spec | |
| 13:25:56 | mriedem | tpatil has a spec leveraging that, which i couldn't understand | |
| 13:26:14 | mriedem | and sounded like well if all the planets were aligned config wise things *might* not shit the bed | |
| 13:26:20 | jaypipes | mriedem: ok, sorry I thought you were referring to the latter. | |
| 13:26:26 | sean-k-mooney | jaypipes: yes it likely was. | |
| 13:26:31 | mriedem | the latter is https://review.openstack.org/#/c/609960/ | |
| 13:26:45 | jaypipes | yes, that's what I thought you were referring to :) | |
| 13:26:48 | mriedem | you can see my confusion in https://review.openstack.org/#/c/609960/2/specs/stein/approved/placement-req-filter-forbidden-aggregates.rst@14 | |
| 13:26:58 | jaypipes | you being +2 on that one was news to me ;) | |
| 13:27:06 | mriedem | heh, not even close | |
| 13:27:10 | jaypipes | reight... | |
| 13:27:28 | mriedem | i don't feel great about that, because this is at least the 3rd spec that tushar has had to chase for his issue | |
| 13:27:40 | jaypipes | mriedem: ok, well at least you know why I was a bit confused above then :) | |
| 13:27:58 | sean-k-mooney | why do we have two specs for the same thing? | |
| 13:27:58 | jaypipes | mriedem: ack, agreed. but the specs should be clear. and this isn't (yet) | |
| 13:28:12 | mriedem | sean-k-mooney: it's not | |
| 13:28:15 | jaypipes | sean-k-mooney: mriedem had (correctly) asked to separate the nova-config stuff from the placement needs | |
| 13:28:19 | mriedem | once is the placement change, one is the nova change ot use it | |
| 13:28:24 | jaypipes | jinx | |
| 13:28:27 | sean-k-mooney | oh ok | |
| 13:28:33 | mriedem | because they are separate services... | |
| 13:28:38 | sean-k-mooney | i was just aware of https://review.openstack.org/#/c/603352/ | |
| 13:28:54 | jaypipes | sean-k-mooney: it's kind of like if you combine the two words eliminate and intimidate... better to have two words that mean different things. | |
| 13:29:04 | jaypipes | sean-k-mooney: :P | |
| 13:29:14 | mriedem | tssurya: so if your computes can talk to the scheduler nodes over rpc, enabling that option might help you some with the scheduling performance issues | |
| 13:29:42 | mriedem | but with 14K computes, you might also melt your mq... | |
| 13:29:44 | sean-k-mooney | :) | |
| 13:30:13 | mriedem | jaypipes: they severely limit max_placement_results to 10 because scheduling takes too long | |
| 13:30:21 | mriedem | mostly boils down to https://bugs.launchpad.net/nova/+bug/1737465 i think | |
| 13:30:21 | openstack | Launchpad bug 1737465 in OpenStack Compute (nova) "[cellv2] the performance issue of cellv2 when creating 500 instances concurrently" [Medium,Confirmed] | |
| 13:30:38 | mriedem | from the forum session on cells v2: | |
| 13:30:39 | mriedem | "TODO: need someone to dig into optimized DB queries in HostManager._get_host_states()." | |
| 13:30:43 | jaypipes | mriedem: by "scheduling" you are referring to the entire process on the nova side or are you referring to just the placement allocation candidates query? | |
| 13:30:50 | mriedem | the entire process | |
| 13:30:52 | jaypipes | k | |
| 13:31:05 | mriedem | the biggest issue, as far as i know right now, is likely HostManager._get_host_states() | |
| 13:31:09 | jaypipes | but they only have one filter enabled... so that shouldn't be a big perf hit, right? | |
| 13:31:33 | jaypipes | mriedem: and all ten of the hosts returned are disabled? | |
| 13:31:45 | sean-k-mooney | jaypipes: yep | |
| 13:31:45 | mriedem | different issue... | |
| 13:31:52 | mriedem | we're talking past each other | |
| 13:31:52 | sean-k-mooney | well its related | |
| 13:32:09 | sean-k-mooney | either they hit all are diable | |
| 13:32:14 | mriedem | the disabled computes issue is a result of setting max_placement_results to 10 | |
| 13:32:15 | jaypipes | lemme read all the bug contents... one sec. | |
| 13:32:33 | sean-k-mooney | or they are not but then when the weight the host its not a even spread or packing behavior anymore | |
| 13:32:33 | mriedem | the root issue is that they have max_placement_results set so low because of shitty performance in nova-scheduler | |
| 13:34:08 | mriedem | https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L704 is a problem, because it iterates all the resulting compute nodes (via alloc candidates), and for each it joins the host aggregates and instances per host https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L734 | |
| 13:34:11 | jaypipes | mriedem: does CERN have placement request filtering enabled? | |
| 13:34:20 | mriedem | so that the HostState has info about aggregates and instances before the HostState goes ot the filters | |
| 13:34:23 | mriedem | jaypipes: yes | |