| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-10-13 | |||
| 20:04:07 | superdan | well, anyone that uses cachingscheduler and something like ramfilter will need that to be whole | |
| 20:04:30 | superdan | otherwise we guess in the caching scheduler with stale data, send to a compute node, and we don't even check to see if the thing can fit, right? | |
| 20:04:51 | figleaf | I had no idea that it was being used like that until mriedem brought it up | |
| 20:05:07 | superdan | that's how all the claiming on the compute node works, right? | |
| 20:06:43 | mriedem | i also fairly recently realized that myself, | |
| 20:07:04 | mriedem | that if the RamFilter isn't enabled, you don't get the memory_mb limit in the limits dict which means the claim in the RT doesn't check ram at all | |
| 20:07:22 | mriedem | same for vcpu and disk | |
| 20:07:55 | mriedem | for filter scheduler and placement it's cool beans since we filter and claim up front via placement | |
| 20:08:00 | figleaf | mriedem: so if deployers with caching scheduler add those filters, they will be ok? Or do they still need the limits sent down to compute? | |
| 20:09:01 | superdan | figleaf: no | |
| 20:09:19 | superdan | figleaf: the filter adds the limit, which makes it down to compute, which is what causes compute to actually check/claim for that thing | |
| 20:09:29 | superdan | figleaf: don't run ramfilter, you don't check ram on the compute | |
| 20:09:57 | superdan | figleaf: I mean, just adding the filter works _today_ | |
| 20:10:04 | figleaf | won't ramfilter check ram on the host state? If there isn't enough, it's discarded, no? | |
| 20:10:12 | superdan | figleaf: if you stop passing the limit down to compute, then compute never checks | |
| 20:10:21 | superdan | figleaf: yes, but it checks based on the state of the data it has | |
| 20:10:32 | superdan | figleaf: which is extra old in the case of caching scheduler | |
| 20:10:40 | superdan | figleaf: that's why we have reschedules now, | |
| 20:10:53 | superdan | for when the scheduler made a decision that is no longer correct by the time build starts on thecompute node | |
| 20:11:01 | superdan | without the limit, compute doesn't even check to see if it still fits | |
| 20:13:07 | figleaf | superdan: so the problem is here: https://github.com/openstack/nova/blob/master/nova/compute/claims.py#L240-L241 | |
| 20:13:21 | superdan | figleaf: yep | |
| 20:13:34 | figleaf | If there is no limit, can't we just use total - used? | |
| 20:13:40 | figleaf | Instead of unlimited? | |
| 20:14:29 | figleaf | Just add the line: limit = limit if limit else total | |
| 20:14:38 | superdan | figleaf: we don't want that for placement | |
| 20:15:12 | superdan | figleaf: you'd also need this level of logic: https://github.com/openstack/nova/blob/master/nova/scheduler/filters/ram_filter.py#L51-L51 | |
| 20:15:14 | superdan | not just total-used | |
| 20:15:18 | figleaf | But with placement, isn't compute claiming going away? | |
| 20:15:33 | superdan | figleaf: yes, but not with cachingscheduler, that's the point here | |
| 20:15:43 | superdan | and, | |
| 20:15:57 | superdan | we will have old computes that wouldn't have your updated claim logic, even if you baked it all in down there | |
| 20:17:18 | figleaf | They will get Selection objects with empty allocation_request fields, right? | |
| 20:17:28 | superdan | old computes can't | |
| 20:17:36 | superdan | we can't send selection objects to old computes | |
| 20:17:44 | figleaf | so conductor won't try to claim. Ugh | |
| 20:18:07 | figleaf | Selection objects will be converted to dicts for old computes | |
| 20:18:12 | figleaf | with empty limits | |
| 20:18:16 | superdan | but for the limits, the old computes will be getting empty limits dicts or something, which will mean you roll your scheduler/conductor to queens and all your pike computes stop checking | |
| 20:18:23 | superdan | right | |
| 20:18:25 | superdan | exactly that | |
| 20:20:04 | figleaf | So if we add a limits field back in there, we can at least populate the limits dict for old computes, right? | |
| 20:20:18 | figleaf | add to Selection object | |
| 20:20:39 | superdan | yep, that's what we're saying | |
| 20:21:08 | superdan | I hate to have to add such a field to our object, knowing we don't want that stuff in a cycle or two when that scheduler is gone, but I don't think we have an option | |
| 20:21:38 | superdan | I'm quite sure jay knows about this stuff, so that's why I'm highly interested in his thinking on this, | |
| 20:21:41 | figleaf | We could tell the caching scheduler users to take a hike :) | |
| 20:21:47 | superdan | but I expect he's just missing the full picture | |
| 20:22:09 | superdan | figleaf: I think mriedem would get spanked :) | |
| 20:22:17 | figleaf | "You want limits? I got your limits right here!" | |
| 20:22:21 | superdan | also, the cachingscheduler users tell us "we'll drop it when you make the other stuff usable" | |
| 20:23:15 | figleaf | superdan: yeah, some comparisons would be good to have | |
| 20:23:43 | superdan | I think it's not as straightforward.. | |
| 20:23:56 | superdan | with placement, total time and energy expended to boot should be faster, | |
| 20:24:10 | superdan | but caching scheduler means fewer db calls for each schedule operation, | |
| 20:24:12 | mriedem | yeah the huawei public cloud is using the caching scheduler for the same reason rax public cloud was, it outperforms the filter scheduler at scale b/c it's not hitting the database to pull 1000 compute node records on every select_destination request | |
| 20:24:19 | superdan | even though it's more likely to send you into a reschedule tailspin | |
| 20:24:52 | mriedem | huawei has set host_subset_size pretty high too so they are spreading the requests across computes, to avoid the reschedule collisions i think, | |
| 20:25:00 | mriedem | i still have to find out more about how much of a problem reschedules are for them | |
| 20:25:38 | mriedem | i noted that with placement, we're doing the ram/disk/cpu filtering in sql on the placement side, but they said that even with that it doesn't filter out enough nodes to make the db query to get compute nodes substantially smaller | |
| 20:25:42 | figleaf | mriedem: too bad scheduling queues never caught on. They would be perfect for that use case | |
| 20:25:50 | mriedem | so i suspect their limiting factor is not filtering on those resource classes, but other things like affinity | |
| 20:26:04 | mriedem | affinity, numa, etc | |
| 20:26:11 | mriedem | stuff we don't have working with placement yet | |
| 20:26:27 | mriedem | i'm still trying to figure out what their most expensive filters are too | |
| 20:26:56 | superdan | mriedem: ugh, that's disappointing | |
| 20:27:14 | mriedem | roughly speaking the scale they are trying to hit, from my limited knowledge so far, is 1024 computes per "cell", 30K nodes total | |
| 20:27:45 | superdan | well, it's probably worth noting that that's pretty high compared to what most people do in a "cell" | |
| 20:27:58 | mriedem | yeah, i just sent some lengthy emails about that last night, | |
| 20:28:07 | mriedem | suggesting they take a page from cern and shard their cells at 200 nodes each | |
| 20:28:11 | superdan | yeah | |
| 20:28:12 | mriedem | so 5 cells per "huawei cell" | |
| 20:28:22 | mriedem | huawei cell = cascaded openstack | |
| 20:28:29 | superdan | right | |
| 20:28:31 | mriedem | == cells v1 | |
| 20:28:36 | mriedem | :) | |
| 20:28:39 | superdan | although it's worth noting that cellsv2 cells won't help them here | |
| 20:28:40 | mriedem | but not cells v1 | |
| 20:28:49 | superdan | because global scheduler | |
| 20:28:52 | mriedem | yup | |
| 20:29:12 | mriedem | i have no idea yet about how their top-layer scheduler picks a cell to send requests to | |
| 20:29:43 | mriedem | nor do i know how large cells v1 deployments like cern are going to deal with moving to the global flat scheduler like in cells v2 | |
| 20:29:53 | mriedem | hopefully we'll get some feedback there someday | |
| 20:29:55 | superdan | well, placement was supposed to be the answer | |
| 20:30:24 | superdan | are we limiting the result size we're getting back from placement yet or are we still trying to pull the whole thing? | |
| 20:30:40 | superdan | because if we're returning everything, that means our query to list compute nodes is huge | |
| 20:30:40 | mriedem | cdent still has a spec up i believe | |
| 20:30:44 | superdan | okay | |
| 20:31:06 | mriedem | https://review.openstack.org/#/c/504540/ | |
| 20:35:22 | fried_rice | Quick, let's merge this pile^Wstack before the zuulv3 cutover https://review.openstack.org/#/c/509029/ | |
| 20:42:26 | tasker | mriedem: sorry to bug you, but is there any movement on https://review.openstack.org/#/c/505771/ ? | |
| 20:43:03 | mriedem | tasker: sorry not yet, it's on my list, but trying to get some stuff done with specs before the 10/19 deadline | |
| 20:43:10 | tasker | roger | |
| 20:43:11 | mriedem | i hope to have that fix polished up next week | |
| 20:43:49 | tasker | thanks! | |
| 20:44:28 | mriedem | give me a kidney punch early next week if you don't see movement | |
| 20:49:27 | openstackgerrit | Eric Young proposed openstack/nova master: 1:Implement ScaleIO image backend https://review.openstack.org/407440 | |
| 20:51:25 | openstackgerrit | Eric Young proposed openstack/nova master: Implement ScaleIO image backend https://review.openstack.org/407440 | |
| 21:03:13 | superdan | fried_rice: one more comment on your patch in case you were respinning right there | |
| 21:03:47 | superdan | mriedem would have shamed me post +2 if I had missed that | |