| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-02-15 | |||
| 15:35:39 | dansmith | bauzas: because cern would have to process 9000 hosts that couldn't possibly work before they get to the first one that would | |
| 15:36:14 | dansmith | bauzas: if you have a 10k node deployment with lots of space, and you boot a small instance, you get back a ridiculous number of hosts that you're not allowed to use and we have to run filters on all of them | |
| 15:36:23 | bauzas | dansmith: it ties to a Placement performance question, right? | |
| 15:36:28 | dansmith | no | |
| 15:36:36 | dansmith | placement is fast at doing that, | |
| 15:36:47 | dansmith | scheduler is slow at processing the result | |
| 15:36:48 | bauzas | dansmith: because for that kind of purpose, we always say to use the more important filters first | |
| 15:36:53 | mriedem | melwitt: i've started a technical debt / cleanup section in https://etherpad.openstack.org/p/nova-ptg-rocky L157 | |
| 15:37:09 | dansmith | bauzas: at CERN-level scale, that is still very wasteful | |
| 15:37:10 | bauzas | dansmith: since each filter is processed one after the other, depending on the ordering of the list | |
| 15:37:36 | bauzas | dansmith: did they found some bottleneck ? | |
| 15:37:49 | dansmith | bauzas: did you read the irc conversation? | |
| 15:37:55 | bauzas | because the last benches I had from the scheduler, the filtering process itself was way fast | |
| 15:38:18 | bauzas | dansmith: from the cells meeting ? unfortunately no | |
| 15:38:25 | dansmith | bauzas: we also only request 1000 hosts from placement, so if your hosts are after the first thousand, we wouldn't get back any that work for us | |
| 15:38:31 | dansmith | bauzas: no, linked in the spec | |
| 15:38:47 | mriedem | bauzas: cern ties tenants to cells, | |
| 15:38:56 | mriedem | so top level cells v1 scheduler will pick the cell for the tenant, | |
| 15:39:01 | mriedem | but with cells v2, we have a flat scheduler, | |
| 15:39:12 | dansmith | bauzas: so if you have more than 1000 hosts that could fit your size flavor, but the first one you're allowed to use is after that, you could never schedule, or have to bump that limit higher | |
| 15:39:20 | mriedem | so rather than a request from that tenant being just the computes in the chosen child cell, now it's all hosts from all cells, | |
| 15:39:21 | mriedem | as dan pointed out | |
| 15:39:48 | dansmith | and, any hard requirement we (nova) knows about that it can communicate to placement ahead of time will only improve performance, and ensure we get back a richer set of hosts to choose from | |
| 15:40:03 | dansmith | and "can never be on hosts in aggregate X" is a pretty concrete requirement :) | |
| 15:40:08 | bauzas | dansmith: mriedem: I understand that concern | |
| 15:40:20 | bauzas | I just wonder how big it can become | |
| 15:40:25 | mriedem | i like to also think of this as a way to eventually get rid of these other post-placement filters | |
| 15:40:34 | dansmith | yeah | |
| 15:40:37 | bauzas | the fact we said placement was efficient is that because it doesn't care of anything specific | |
| 15:40:37 | dansmith | that is also nice | |
| 15:40:38 | mriedem | we have already said don't use ram/disk/core filter b/c placement | |
| 15:40:49 | mriedem | this would also remove the aggregate tenant isolation filter | |
| 15:41:06 | dansmith | bauzas: placement already does aggregates, this is well within its scope | |
| 15:41:16 | mriedem | bauzas: i'm not aware of anyone that has done benchmarking before and after placement at scale | |
| 15:41:19 | mriedem | to say that we're way better now | |
| 15:41:39 | dansmith | bauzas: performance aside, you get the point about the request limit right? | |
| 15:41:43 | bauzas | mriedem: no, the benches were pre-placement, but saw the filtering process very fast | |
| 15:41:49 | bauzas | dansmith: sure | |
| 15:41:51 | dansmith | if we'reonly getting back 1000 hosts, then we could get back none | |
| 15:42:26 | openstackgerrit | Claudiu Belu proposed openstack/nova master: compute: Cleans up allocations after failed resize https://review.openstack.org/543971 | |
| 15:42:36 | bauzas | well, I don't know what to say | |
| 15:42:42 | dansmith | bauzas: performance aside, we want to get back a list of hosts that are candidates, and hosts that are in aggregates we're not allowed to be on are not candidates, so they're just busy work and they defeat the point of getting a solid fast set of canidates from placement based on things it knows about | |
| 15:43:47 | bauzas | dansmith: sure | |
| 15:43:52 | bauzas | anyway, here is my take | |
| 15:43:58 | bauzas | I'm not opposed at that | |
| 15:44:05 | bauzas | I just need to consider all the implications | |
| 15:44:20 | bauzas | because filters are like the nova things the most tweaked | |
| 15:44:25 | bauzas | by ops | |
| 15:44:41 | mriedem | dansmith: i'm assuming the pre-placement filters would also be pluggable? | |
| 15:45:01 | dansmith | mriedem: modular, like our scheduler filters, not necessarily pluggable | |
| 15:45:06 | bauzas | if we're about to introduce a new hook for cheating placement to only return the subset we want, chances are that in the future ops will come with solid use-cases | |
| 15:45:24 | dansmith | bauzas: cheating placement? I can't understand how this is that :) | |
| 15:45:30 | bauzas | and at the end of the day, we'd just introduce more and more complexity to placement | |
| 15:45:39 | dansmith | this doesn't change placement at all | |
| 15:45:41 | dansmith | not a single bit | |
| 15:45:48 | mriedem | these pre-placement filters are restricted to what placement api supports | |
| 15:45:55 | bauzas | dansmith: for your proposal, I agree | |
| 15:45:56 | mriedem | which is resource classes, traits and aggregates | |
| 15:46:00 | dansmith | they operate on request spec | |
| 15:46:20 | bauzas | dansmith: again, I'm not opposed to *your* use case which is tenant filtering | |
| 15:46:41 | mriedem | i also have a use case for this | |
| 15:46:46 | mriedem | if you care | |
| 15:46:49 | bauzas | that said, we could do that without using aggregateds | |
| 15:46:54 | bauzas | I guess | |
| 15:46:56 | bauzas | anyway | |
| 15:47:07 | bauzas | finding use-cases is not the thing I'm worried | |
| 15:47:15 | bauzas | I'm more worried by how big it can become | |
| 15:47:46 | mriedem | we knew we wanted to move filters in sql via placement as much as possible since the beginning right? | |
| 15:47:48 | dansmith | I think I need a more concrete argument than that to understand the concern :) | |
| 15:48:07 | bauzas | dansmith: hence why I'm discussing over IRC | |
| 15:48:19 | bauzas | FUD isn't a valid argument and I respect that | |
| 15:48:27 | dansmith | IMHO, this is the natural evolution of us asking placement for hosts based on resources (which it deals with) and then traits (which it deals with) and now aggregates (which it deals with) | |
| 15:48:31 | bauzas | I just need to understand the context | |
| 15:49:16 | dansmith | bauzas: why don't you want to hear mriedem's use case? | |
| 15:49:29 | bauzas | dansmith: I'm open to his use-case | |
| 15:49:52 | dansmith | bauzas: I also had a BFV use case in the first rev of the spec, | |
| 15:50:15 | dansmith | but removed it because it was distracting to some people | |
| 15:50:16 | bauzas | the limitation thing is certainly a valid argument | |
| 15:50:29 | bauzas | because we need to get all the corresponding hosts from placement | |
| 15:50:40 | bauzas | if we say we won't allow pluggability, I think I'd be okay | |
| 15:50:53 | dansmith | personally I don't want this to be pluggable, just modular | |
| 15:51:01 | dansmith | (brb) | |
| 15:51:02 | bauzas | then okay | |
| 15:51:14 | bauzas | if we're all clear that we will never open that to ops | |
| 15:51:16 | openstackgerrit | Eric Fried proposed openstack/nova-specs master: Update Provider Tree https://review.openstack.org/540111 | |
| 15:51:18 | mriedem | bauzas: what i want to use this for is making sure we can pick a host that supports multiattach volumes for bfv, and tagged attach during bfv | |
| 15:51:26 | efried | edleafe, jaypipes, cdent: There's that reword ^ | |
| 15:51:36 | mriedem | bauzas: because today that's just luck of the host and we don't reschedule on failed volume attaching during bfv | |
| 15:51:51 | bauzas | mriedem: you could achieve that with a filter, right? | |
| 15:52:12 | bauzas | atm, we have two placing mechanisms | |
| 15:52:16 | bauzas | #1 is placement | |
| 15:52:20 | bauzas | #2 is filters | |
| 15:52:29 | mriedem | do we have a filter that actually has the driver.capabilities information exposed to it? | |
| 15:52:33 | bauzas | I agree with both of you to try to make #1 prioritary | |
| 15:52:38 | cdent | efried: good, thanks | |
| 15:52:57 | efried | bauzas: IMO we should do any filtering that's possible/practical to do in placement, in placement. Cause that'll be more efficient. | |
| 15:52:58 | bauzas | mriedem: we have one caring about the compute capabilities themselves | |
| 15:53:08 | mriedem | bauzas: but i'm not sure if that filter actually gets that data | |
| 15:53:24 | bauzas | efried: yeah, hence my | |
| 15:53:25 | bauzas | (16:52:33) bauzas: I agree with both of you to try to make #1 prioritary | |