| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-12-06 | |||
| 13:24:55 | jaypipes | sean-k-mooney: is elimidate a combination of eliminate and intimidate? | |
| 13:24:57 | mriedem | jaypipes: you were +2 on the spec before | |
| 13:24:58 | tssurya | mriedem: its False | |
| 13:25:06 | mriedem | tssurya: ok. you aren't running split MQ right? | |
| 13:25:20 | jaypipes | mriedem: are we talking about the same spec? | |
| 13:25:30 | jaypipes | mriedem: the nova side one or the placement-side one? | |
| 13:25:31 | mriedem | https://review.openstack.org/#/c/603352/ | |
| 13:25:46 | mriedem | ^ is the placement spec | |
| 13:25:56 | mriedem | tpatil has a spec leveraging that, which i couldn't understand | |
| 13:26:14 | mriedem | and sounded like well if all the planets were aligned config wise things *might* not shit the bed | |
| 13:26:20 | jaypipes | mriedem: ok, sorry I thought you were referring to the latter. | |
| 13:26:26 | sean-k-mooney | jaypipes: yes it likely was. | |
| 13:26:31 | mriedem | the latter is https://review.openstack.org/#/c/609960/ | |
| 13:26:45 | jaypipes | yes, that's what I thought you were referring to :) | |
| 13:26:48 | mriedem | you can see my confusion in https://review.openstack.org/#/c/609960/2/specs/stein/approved/placement-req-filter-forbidden-aggregates.rst@14 | |
| 13:26:58 | jaypipes | you being +2 on that one was news to me ;) | |
| 13:27:06 | mriedem | heh, not even close | |
| 13:27:10 | jaypipes | reight... | |
| 13:27:28 | mriedem | i don't feel great about that, because this is at least the 3rd spec that tushar has had to chase for his issue | |
| 13:27:40 | jaypipes | mriedem: ok, well at least you know why I was a bit confused above then :) | |
| 13:27:58 | sean-k-mooney | why do we have two specs for the same thing? | |
| 13:27:58 | jaypipes | mriedem: ack, agreed. but the specs should be clear. and this isn't (yet) | |
| 13:28:12 | mriedem | sean-k-mooney: it's not | |
| 13:28:15 | jaypipes | sean-k-mooney: mriedem had (correctly) asked to separate the nova-config stuff from the placement needs | |
| 13:28:19 | mriedem | once is the placement change, one is the nova change ot use it | |
| 13:28:24 | jaypipes | jinx | |
| 13:28:27 | sean-k-mooney | oh ok | |
| 13:28:33 | mriedem | because they are separate services... | |
| 13:28:38 | sean-k-mooney | i was just aware of https://review.openstack.org/#/c/603352/ | |
| 13:28:54 | jaypipes | sean-k-mooney: it's kind of like if you combine the two words eliminate and intimidate... better to have two words that mean different things. | |
| 13:29:04 | jaypipes | sean-k-mooney: :P | |
| 13:29:14 | mriedem | tssurya: so if your computes can talk to the scheduler nodes over rpc, enabling that option might help you some with the scheduling performance issues | |
| 13:29:42 | mriedem | but with 14K computes, you might also melt your mq... | |
| 13:29:44 | sean-k-mooney | :) | |
| 13:30:13 | mriedem | jaypipes: they severely limit max_placement_results to 10 because scheduling takes too long | |
| 13:30:21 | mriedem | mostly boils down to https://bugs.launchpad.net/nova/+bug/1737465 i think | |
| 13:30:21 | openstack | Launchpad bug 1737465 in OpenStack Compute (nova) "[cellv2] the performance issue of cellv2 when creating 500 instances concurrently" [Medium,Confirmed] | |
| 13:30:38 | mriedem | from the forum session on cells v2: | |
| 13:30:39 | mriedem | "TODO: need someone to dig into optimized DB queries in HostManager._get_host_states()." | |
| 13:30:43 | jaypipes | mriedem: by "scheduling" you are referring to the entire process on the nova side or are you referring to just the placement allocation candidates query? | |
| 13:30:50 | mriedem | the entire process | |
| 13:30:52 | jaypipes | k | |
| 13:31:05 | mriedem | the biggest issue, as far as i know right now, is likely HostManager._get_host_states() | |
| 13:31:09 | jaypipes | but they only have one filter enabled... so that shouldn't be a big perf hit, right? | |
| 13:31:33 | jaypipes | mriedem: and all ten of the hosts returned are disabled? | |
| 13:31:45 | sean-k-mooney | jaypipes: yep | |
| 13:31:45 | mriedem | different issue... | |
| 13:31:52 | mriedem | we're talking past each other | |
| 13:31:52 | sean-k-mooney | well its related | |
| 13:32:09 | sean-k-mooney | either they hit all are diable | |
| 13:32:14 | mriedem | the disabled computes issue is a result of setting max_placement_results to 10 | |
| 13:32:15 | jaypipes | lemme read all the bug contents... one sec. | |
| 13:32:33 | sean-k-mooney | or they are not but then when the weight the host its not a even spread or packing behavior anymore | |
| 13:32:33 | mriedem | the root issue is that they have max_placement_results set so low because of shitty performance in nova-scheduler | |
| 13:34:08 | mriedem | https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L704 is a problem, because it iterates all the resulting compute nodes (via alloc candidates), and for each it joins the host aggregates and instances per host https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L734 | |
| 13:34:11 | jaypipes | mriedem: does CERN have placement request filtering enabled? | |
| 13:34:20 | mriedem | so that the HostState has info about aggregates and instances before the HostState goes ot the filters | |
| 13:34:23 | mriedem | jaypipes: yes | |
| 13:34:26 | mriedem | it was added for them :) | |
| 13:34:32 | sean-k-mooney | mriedem: ya by seting it to 10 the schuler only considerd 5% of posibel host in a cell or less then 1% of host that tenatn could be schduled too | |
| 13:34:46 | mriedem | sean-k-mooney: you know i understand the problem right? | |
| 13:35:03 | sean-k-mooney | mriedem: ya i know | |
| 13:35:23 | sean-k-mooney | sorry ill leave you too it i need to finish fixing my own spec | |
| 13:35:52 | mriedem | jaypipes: so if you have some wild sql fu thoughts on how we could optimize https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L704 that would be super | |
| 13:37:01 | mriedem | like, in a single query per cell, give me all the computes in a uuid list along with the list of instance uuids on those hosts | |
| 13:37:39 | jaypipes | mriedem: the two links to host_manager.py above are not doing SQL operations... they are doing lookups on the host manager's local hashmaps of aggregate and instance information... | |
| 13:38:08 | jaypipes | mriedem: the nova-compute services are reporting aggregate and instance usage via the queryclient interface, and the host manager caches that information. | |
| 13:38:32 | mriedem | https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L747 | |
| 13:38:36 | mriedem | is definitely doing db stuff | |
| 13:38:46 | mriedem | there is no cache here | |
| 13:38:51 | mriedem | maybe for aggregates | |
| 13:38:56 | mriedem | but not for host->instance mappings | |
| 13:38:58 | jaypipes | mriedem: that's only called if there's no cache entry for the host. | |
| 13:39:13 | mriedem | which there won't be per request | |
| 13:39:24 | jaypipes | mriedem: those cache entries are populated on startup of the host manager and when the compute services report to the scheduler via the queryclient | |
| 13:39:42 | mriedem | jaypipes: that's only if you have enabled track_instance_changes | |
| 13:39:45 | mriedem | which cern does not do, | |
| 13:39:55 | mriedem | and which won't work in a split MQ cells deployemtn where the computes can't rpc cast to the scheduler | |
| 13:40:14 | jaypipes | guh... | |
| 13:40:22 | jaypipes | so THAT'S the source of this issue, really. | |
| 13:40:25 | jaypipes | ok. | |
| 13:40:39 | mriedem | so i've suggested that cern try track_instance_changes=true to see if that caching helps | |
| 13:40:49 | mriedem | maybe at least for a handful of cells to start | |
| 13:40:49 | jaypipes | mriedem: alright, I can do some SQL-fu on the _get_instances_by_host() method. | |
| 13:41:30 | jaypipes | mriedem: OK, I understand the issue better now. Gimme a little to experiment with something, ok? | |
| 13:41:46 | mriedem | sure | |
| 13:41:55 | mriedem | let this be your white whale for the day | |
| 13:42:08 | jaypipes | just call me Ahab. | |
| 13:42:19 | jaypipes | oh, hi tssurya :) | |
| 13:43:27 | jaypipes | mriedem: ok, so now I understand why CERN is setting max_placement_records so low... it's because a single HostMapping.get_by_host() is being a issued for each found host. | |
| 13:43:43 | jaypipes | instead of a batched approach. | |
| 13:45:09 | mriedem | sure, it's more than just the host mapping query | |
| 13:45:35 | jaypipes | https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L758 | |
| 13:45:40 | mriedem | bingo | |
| 13:45:40 | jaypipes | InstalceList too... | |
| 13:45:43 | jaypipes | ack. | |
| 13:45:47 | mriedem | yes, and i have already neutered that | |
| 13:45:56 | mriedem | that used to pull full Instance objects back | |
| 13:45:59 | mriedem | now it's just uuids | |
| 13:46:03 | mriedem | https://github.com/openstack/nova/commit/91f5af7ee7f7140eafb7237875f6cd6ea1abcd38 | |