| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-12-06 | |||
| 13:31:52 | mriedem | we're talking past each other | |
| 13:31:52 | sean-k-mooney | well its related | |
| 13:32:09 | sean-k-mooney | either they hit all are diable | |
| 13:32:14 | mriedem | the disabled computes issue is a result of setting max_placement_results to 10 | |
| 13:32:15 | jaypipes | lemme read all the bug contents... one sec. | |
| 13:32:33 | sean-k-mooney | or they are not but then when the weight the host its not a even spread or packing behavior anymore | |
| 13:32:33 | mriedem | the root issue is that they have max_placement_results set so low because of shitty performance in nova-scheduler | |
| 13:34:08 | mriedem | https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L704 is a problem, because it iterates all the resulting compute nodes (via alloc candidates), and for each it joins the host aggregates and instances per host https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L734 | |
| 13:34:11 | jaypipes | mriedem: does CERN have placement request filtering enabled? | |
| 13:34:20 | mriedem | so that the HostState has info about aggregates and instances before the HostState goes ot the filters | |
| 13:34:23 | mriedem | jaypipes: yes | |
| 13:34:26 | mriedem | it was added for them :) | |
| 13:34:32 | sean-k-mooney | mriedem: ya by seting it to 10 the schuler only considerd 5% of posibel host in a cell or less then 1% of host that tenatn could be schduled too | |
| 13:34:46 | mriedem | sean-k-mooney: you know i understand the problem right? | |
| 13:35:03 | sean-k-mooney | mriedem: ya i know | |
| 13:35:23 | sean-k-mooney | sorry ill leave you too it i need to finish fixing my own spec | |
| 13:35:52 | mriedem | jaypipes: so if you have some wild sql fu thoughts on how we could optimize https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L704 that would be super | |
| 13:37:01 | mriedem | like, in a single query per cell, give me all the computes in a uuid list along with the list of instance uuids on those hosts | |
| 13:37:39 | jaypipes | mriedem: the two links to host_manager.py above are not doing SQL operations... they are doing lookups on the host manager's local hashmaps of aggregate and instance information... | |
| 13:38:08 | jaypipes | mriedem: the nova-compute services are reporting aggregate and instance usage via the queryclient interface, and the host manager caches that information. | |
| 13:38:32 | mriedem | https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L747 | |
| 13:38:36 | mriedem | is definitely doing db stuff | |
| 13:38:46 | mriedem | there is no cache here | |
| 13:38:51 | mriedem | maybe for aggregates | |
| 13:38:56 | mriedem | but not for host->instance mappings | |
| 13:38:58 | jaypipes | mriedem: that's only called if there's no cache entry for the host. | |
| 13:39:13 | mriedem | which there won't be per request | |
| 13:39:24 | jaypipes | mriedem: those cache entries are populated on startup of the host manager and when the compute services report to the scheduler via the queryclient | |
| 13:39:42 | mriedem | jaypipes: that's only if you have enabled track_instance_changes | |
| 13:39:45 | mriedem | which cern does not do, | |
| 13:39:55 | mriedem | and which won't work in a split MQ cells deployemtn where the computes can't rpc cast to the scheduler | |
| 13:40:14 | jaypipes | guh... | |
| 13:40:22 | jaypipes | so THAT'S the source of this issue, really. | |
| 13:40:25 | jaypipes | ok. | |
| 13:40:39 | mriedem | so i've suggested that cern try track_instance_changes=true to see if that caching helps | |
| 13:40:49 | mriedem | maybe at least for a handful of cells to start | |
| 13:40:49 | jaypipes | mriedem: alright, I can do some SQL-fu on the _get_instances_by_host() method. | |
| 13:41:30 | jaypipes | mriedem: OK, I understand the issue better now. Gimme a little to experiment with something, ok? | |
| 13:41:46 | mriedem | sure | |
| 13:41:55 | mriedem | let this be your white whale for the day | |
| 13:42:08 | jaypipes | just call me Ahab. | |
| 13:42:19 | jaypipes | oh, hi tssurya :) | |
| 13:43:27 | jaypipes | mriedem: ok, so now I understand why CERN is setting max_placement_records so low... it's because a single HostMapping.get_by_host() is being a issued for each found host. | |
| 13:43:43 | jaypipes | instead of a batched approach. | |
| 13:45:09 | mriedem | sure, it's more than just the host mapping query | |
| 13:45:35 | jaypipes | https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L758 | |
| 13:45:40 | mriedem | bingo | |
| 13:45:40 | jaypipes | InstalceList too... | |
| 13:45:43 | jaypipes | ack. | |
| 13:45:47 | mriedem | yes, and i have already neutered that | |
| 13:45:56 | mriedem | that used to pull full Instance objects back | |
| 13:45:59 | mriedem | now it's just uuids | |
| 13:46:03 | mriedem | https://github.com/openstack/nova/commit/91f5af7ee7f7140eafb7237875f6cd6ea1abcd38 | |
| 13:46:07 | jaypipes | k | |
| 13:46:26 | frickler | mriedem: any chance to give these stable reviews progress? https://review.openstack.org/619220 https://review.openstack.org/619254 | |
| 13:46:45 | mriedem | frickler: there is a >0% chance yes | |
| 13:47:16 | openstackgerrit | Lee Yarwood proposed openstack/nova master: fixtures: Return a mocked class instead of method within fake_imagebackend https://review.openstack.org/619804 | |
| 13:48:13 | mriedem | frickler: why didn't you cherry pick the pike change from the queens change? looks like you had to redo some of the conflict resolution? | |
| 13:48:27 | mriedem | normally backports should just all go in a row, stein -> rocky -> queens -> pike | |
| 13:48:29 | mriedem | like a snowball | |
| 13:49:14 | frickler | mriedem: it was yet another conflict, yes, that function likes to do file hopping it seems | |
| 13:49:38 | mriedem | lyarwood: want to get this? https://review.openstack.org/#/c/619220/1 | |
| 13:49:43 | mriedem | ok | |
| 13:50:09 | lyarwood | mriedem: looking | |
| 13:50:57 | mriedem | ok scored both | |
| 13:51:52 | lyarwood | mriedem: okay done, would you mind taking a look at https://review.openstack.org/#/q/status:open+project:openstack/nova+topic:bug/1764883 when you get a chance? | |
| 13:56:00 | mriedem | i see how it is, i give you something easy and in return you give me this | |
| 13:56:22 | mriedem | oh these are backports.. | |
| 13:56:22 | mriedem | nvm | |
| 13:57:46 | lyarwood | mriedem: something just as easy, yeah, what a guy ;) | |
| 14:06:57 | openstackgerrit | sean mooney proposed openstack/nova-specs master: Add spec for sriov live migration https://review.openstack.org/605116 | |
| 14:07:25 | sean-k-mooney | bauzas: adrianc sorry for the delay on updating ^ | |
| 14:19:21 | adrianc | ack sean-k-mooney, will look into it | |
| 14:27:24 | openstackgerrit | Merged openstack/nova-specs master: Per aggregate scheduling weight (spec) https://review.openstack.org/599308 | |
| 14:28:38 | tssurya | hi back jaypipes :) | |
| 14:29:20 | tssurya | mriedem: thanks for the info on the config option, will see if enabling it makes it better | |
| 14:29:50 | tssurya | I guess the documentation says something like "f the configured filters and weighers do not need this information, disabling this option will improve performance" | |
| 14:30:14 | tssurya | but I didn't know what this exactly did until after looking into the code | |
| 14:31:23 | sean-k-mooney | adrianc: there are very minor changes as there seam to be little questions raised | |
| 14:31:37 | sean-k-mooney | adrianc: it should not affect you poc | |
| 14:35:16 | mriedem | tssurya: yeah it's mostly for the affinity type filters | |
| 14:36:04 | tssurya | mriedem: ah ok | |
| 14:36:22 | tssurya | yea | |
| 14:37:40 | mriedem | the performance thing there is probably about not needing to needlessly send instance information from all computes to the scheduler | |
| 14:37:53 | mriedem | but misses the part about how the scheduler then blindly pulls the information from the db anyway | |
| 14:38:06 | mriedem | hmm... | |
| 14:38:10 | mriedem | i wonder... | |
| 14:38:26 | mriedem | if we only got the HostState.instances in the scheduler if track_instance_changes was true | |
| 14:38:37 | mriedem | since they are kind of connected | |
| 14:38:52 | mriedem | we wouldn't want to make anything conditional on enabled filters b/c people can have out of tree filters | |
| 14:39:16 | mriedem | or we could just add a new option, CONF.filter_scheduler.do_your_filters_care_about_instances_on_hosts | |
| 14:39:24 | mriedem | with a better name | |
| 14:39:57 | mriedem | more config options is gross, but if you have split mq then track_instance_changes won't work and you don't need to enable it | |
| 14:40:32 | tssurya | right, I am pretty sure this option was enabled by default and three months ago we disabled it and saw a very slight performance improvement actually | |
| 14:40:54 | mriedem | because of less mq traffic/ | |
| 14:40:55 | mriedem | ? | |
| 14:41:04 | tssurya | but yea I am getting this from a tracking ticket on the issue, probably we will look into this next week | |
| 14:41:11 | mriedem | ok | |
| 14:41:11 | tssurya | yea I think so | |
| 14:41:41 | mriedem | as i said, i'd expect your mq traffic to go up by enabling it, but your overall scheduling time, at least for hosts in the cells reporting instance info, might go down | |