| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-12-06 | |||
| 13:07:55 | sean-k-mooney | it maters if you have lost of disabled host in the limited set of result you got form plamcnet | |
| 13:08:40 | sean-k-mooney | it will skew the weighing by reducing the set of results | |
| 13:08:45 | tssurya | mriedem: exactly for us when not using "max_placement_results" filters would give us almost all the hosts from a cell | |
| 13:09:15 | openstack | bug 1805989 in OpenStack Compute (nova) "Weight policy to stack/spread instances and "max_placement_results"" [Undecided,New] https://launchpad.net/bugs/1805989 | |
| 13:09:15 | mriedem | tssurya: yeah, so IMO bug 1805989 is not really a bug, it's working as designed | |
| 13:09:20 | tssurya | but yea probably this iss not a bug its design thing | |
| 13:09:29 | openstack | bug 1805984 in OpenStack Compute (nova) "Placement is not aware of disable compute nodes" [Wishlist,Triaged] https://launchpad.net/bugs/1805984 | |
| 13:09:29 | mriedem | bug 1805984 is definitely a problem for which we have solutions | |
| 13:09:38 | sean-k-mooney | that said it is working as desinged as mriedem said. that desing my not be desireable and we might want to consider how to change it | |
| 13:09:58 | gibi | mriedem: I saw your comment on the notification deprecation patch. I have to organize my thoughts formulate an opinion | |
| 13:10:15 | mriedem | what we need to do is get CERN to the point that they don't have to workaround perf issues by setting max_placement_results to 10 | |
| 13:10:18 | mriedem | when they have 14K hosts | |
| 13:10:27 | mriedem | gibi: heh ok :) | |
| 13:10:29 | mriedem | no rush | |
| 13:10:35 | gibi | mriedem: yeah, I will take my time | |
| 13:10:38 | gibi | :) | |
| 13:10:46 | mriedem | gibi: it's a bit depressing huh? | |
| 13:10:48 | tssurya | mriedem: sure its a design thing, we just laid it out in case there were ideas | |
| 13:10:57 | tssurya | mriedem: yea perf stuff is still going on | |
| 13:11:20 | tssurya | hopefully we won't need to set "max_placement_results" to a low number then | |
| 13:11:22 | gibi | mriedem: I understand that the original goals of that work might not be applicable in the today situaion on OpenStack | |
| 13:11:37 | sean-k-mooney | tssurya: the placment randomisation was how we wanted peole to enable spreading behavior | |
| 13:12:55 | sean-k-mooney | so if you had a 1000 hosts that could fit the request and you requested 50 then with the randomisation enbled you would get a random 50 out of that 1000 for the filters and weigher to select form | |
| 13:12:55 | mriedem | tssurya: ok i marked the bug as invalid (working as designed) | |
| 13:14:00 | sean-k-mooney | tssurya: do you rember what you set currently and what is that value as a propotion of your average cell size | |
| 13:14:26 | mriedem | average cell size is 200 at cern | |
| 13:14:32 | mriedem | i think they set max_placement_results to 10 | |
| 13:14:32 | tssurya | sean-k-mooney: 10 versus 800 in normal scenario | |
| 13:14:53 | mriedem | 800? i thought it was 200 on average | |
| 13:15:10 | mriedem | with about 72-74 cells? | |
| 13:15:27 | tssurya | mriedem: yea 200 for special project to cell mappings, on an average for normal users we set aside 5 to 7 default cells | |
| 13:15:43 | tssurya | each cell having 200 | |
| 13:15:59 | mriedem | ok | |
| 13:16:29 | openstack | Launchpad bug 1737465 in OpenStack Compute (nova) "[cellv2] the performance issue of cellv2 when creating 500 instances concurrently" [Medium,Confirmed] | |
| 13:16:29 | mriedem | as far as i know, https://bugs.launchpad.net/nova/+bug/1737465 is still the biggest perf issue | |
| 13:16:35 | sean-k-mooney | ok so could you increase the limit to say 10% of the host that tenat can be expect to selct form | |
| 13:16:57 | sean-k-mooney | e.g. 80 in this case? | |
| 13:17:15 | mriedem | also, as far as i know, cern is not yet grouping cells via host aggregate | |
| 13:17:24 | mriedem | or are you? | |
| 13:17:34 | tssurya | we model cells as aggregates | |
| 13:17:48 | tssurya | as in every cell is an aggregate | |
| 13:17:48 | mriedem | because of https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html#tenant-isolation-with-placement right> | |
| 13:17:49 | mriedem | ? | |
| 13:17:52 | mriedem | ok | |
| 13:18:04 | tssurya | yea because of the pre-filter | |
| 13:18:07 | mriedem | so, really, | |
| 13:18:22 | jaypipes | mriedem: wait, there's a bug in shelve code?! | |
| 13:18:28 | sean-k-mooney | tssurya: so you then use the teanay affinity filter to map tenants to thos aggreates and therefor to cells? | |
| 13:18:30 | mriedem | jaypipes: working as designed | |
| 13:18:47 | tssurya | sean-k-mooney: yea | |
| 13:18:48 | mriedem | tssurya: if the pre-filter is working, then the max_placement_results probably doesn't need to be so low... | |
| 13:19:01 | jaypipes | mriedem: I only have time to argue with cfriesen today, so I'll wait til he's online. :P | |
| 13:19:12 | sean-k-mooney | mriedem: that is what i was wondering too | |
| 13:19:16 | tssurya | mriedem: surely it need not be as low as 10, but even with it being 10 we have 5 to 8 seconds scheduling time | |
| 13:19:19 | mriedem | because if my project is mapped to a cell with ~200 hosts, then filtering on 200 hosts shouldn't be so bad | |
| 13:19:54 | tssurya | the time it would tkae to gather all those host states , well we are still trying to check iut why its 5 to 8 seconds | |
| 13:20:06 | tssurya | out* | |
| 13:20:17 | sean-k-mooney | tssurya: when you say schduiling are you refering to just the time it take for the filter schduler or the time the vm is in that sate | |
| 13:20:34 | mriedem | tssurya: do you know if this is set on the computes and the scheduler nodes? https://docs.openstack.org/nova/latest/configuration/config.html#filter_scheduler.track_instance_changes | |
| 13:20:44 | tssurya | time for the whole boot until select destinations is done fully | |
| 13:20:48 | jaypipes | mriedem: I think it's pretty obvious I wouldn't support adding disabled flags to provider records, yes? | |
| 13:21:01 | mriedem | jaypipes: yes, i could find you my reply to that | |
| 13:21:08 | mriedem | which was probably not very nice | |
| 13:21:20 | jaypipes | mriedem: yes, I saw it. | |
| 13:21:23 | mriedem | and i hope cfriesen will forgive me | |
| 13:21:33 | jaypipes | mriedem: BTW, it's possible to do something like this: | |
| 13:21:35 | tssurya | mriedem: not sure let me check | |
| 13:21:41 | sean-k-mooney | jaypipes: would you support usign a placement aggrage for disabled compute nodes | |
| 13:22:29 | sean-k-mooney | jaypipes: and then use the new member_of:!<aggreage> feature | |
| 13:22:36 | jaypipes | a) request max_placement_records. get those records. b) do a filter() against those provider UUIDs and the set of disabled compute services (would need to grab the compute node UUID, not the hostname, though), and c) if len(a) < len(b), request more from placement | |
| 13:23:02 | mriedem | tssurya: track_instance_changes enables the computes to rpc broadcast to the schedules the information about the instances running on them (the hosts) and then the scheduler workers cache that information so that during a scheduling request the scheduler doesn't need to iterate all of the hosts (via db) to get their current instance list | |
| 13:23:06 | jaypipes | sean-k-mooney: sure, though forbidden aggregates are not approved spec yet... | |
| 13:23:21 | mriedem | tssurya: it's not recommended for split MQ because the computes would be disconnected from the schedulers (so the cast goes in the trash) | |
| 13:23:49 | mriedem | jaypipes: i've been +2 on that forbidden aggregates spec for a couple of weeks now | |
| 13:23:52 | jaypipes | sean-k-mooney: the solution should remain entirely on the nova side, though, IMHO, which is why I recommend the approach above as a method to take if NoValidHosts is encountered. | |
| 13:23:54 | sean-k-mooney | jaypipes: true but its less of an abuse of the placment api and is just as simple to manage as adding or removing a trait | |
| 13:24:07 | mriedem | jaypipes: what we talked about in channel yesterday was all nova side solutions | |
| 13:24:30 | jaypipes | mriedem: gotta find another +2? do you want me to +2 that since I was the one who came up with the idea? | |
| 13:24:34 | sean-k-mooney | jaypipes: e.g. page in more results if the filter elimidate them | |
| 13:24:50 | mriedem | jaypipes: what you suggested about sounds like paging, but also not great performance wise since we'd have to pull the data from the nova cell dbs to check the disabled status | |
| 13:24:55 | jaypipes | sean-k-mooney: is elimidate a combination of eliminate and intimidate? | |
| 13:24:57 | mriedem | jaypipes: you were +2 on the spec before | |
| 13:24:58 | tssurya | mriedem: its False | |
| 13:25:06 | mriedem | tssurya: ok. you aren't running split MQ right? | |
| 13:25:20 | jaypipes | mriedem: are we talking about the same spec? | |
| 13:25:30 | jaypipes | mriedem: the nova side one or the placement-side one? | |
| 13:25:31 | mriedem | https://review.openstack.org/#/c/603352/ | |
| 13:25:46 | mriedem | ^ is the placement spec | |
| 13:25:56 | mriedem | tpatil has a spec leveraging that, which i couldn't understand | |
| 13:26:14 | mriedem | and sounded like well if all the planets were aligned config wise things *might* not shit the bed | |
| 13:26:20 | jaypipes | mriedem: ok, sorry I thought you were referring to the latter. | |
| 13:26:26 | sean-k-mooney | jaypipes: yes it likely was. | |
| 13:26:31 | mriedem | the latter is https://review.openstack.org/#/c/609960/ | |
| 13:26:45 | jaypipes | yes, that's what I thought you were referring to :) | |
| 13:26:48 | mriedem | you can see my confusion in https://review.openstack.org/#/c/609960/2/specs/stein/approved/placement-req-filter-forbidden-aggregates.rst@14 | |
| 13:26:58 | jaypipes | you being +2 on that one was news to me ;) | |
| 13:27:06 | mriedem | heh, not even close | |
| 13:27:10 | jaypipes | reight... | |
| 13:27:28 | mriedem | i don't feel great about that, because this is at least the 3rd spec that tushar has had to chase for his issue | |
| 13:27:40 | jaypipes | mriedem: ok, well at least you know why I was a bit confused above then :) | |