| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-10-13 | |||
| 20:20:39 | superdan | yep, that's what we're saying | |
| 20:21:08 | superdan | I hate to have to add such a field to our object, knowing we don't want that stuff in a cycle or two when that scheduler is gone, but I don't think we have an option | |
| 20:21:38 | superdan | I'm quite sure jay knows about this stuff, so that's why I'm highly interested in his thinking on this, | |
| 20:21:41 | figleaf | We could tell the caching scheduler users to take a hike :) | |
| 20:21:47 | superdan | but I expect he's just missing the full picture | |
| 20:22:09 | superdan | figleaf: I think mriedem would get spanked :) | |
| 20:22:17 | figleaf | "You want limits? I got your limits right here!" | |
| 20:22:21 | superdan | also, the cachingscheduler users tell us "we'll drop it when you make the other stuff usable" | |
| 20:23:15 | figleaf | superdan: yeah, some comparisons would be good to have | |
| 20:23:43 | superdan | I think it's not as straightforward.. | |
| 20:23:56 | superdan | with placement, total time and energy expended to boot should be faster, | |
| 20:24:10 | superdan | but caching scheduler means fewer db calls for each schedule operation, | |
| 20:24:12 | mriedem | yeah the huawei public cloud is using the caching scheduler for the same reason rax public cloud was, it outperforms the filter scheduler at scale b/c it's not hitting the database to pull 1000 compute node records on every select_destination request | |
| 20:24:19 | superdan | even though it's more likely to send you into a reschedule tailspin | |
| 20:24:52 | mriedem | huawei has set host_subset_size pretty high too so they are spreading the requests across computes, to avoid the reschedule collisions i think, | |
| 20:25:00 | mriedem | i still have to find out more about how much of a problem reschedules are for them | |
| 20:25:38 | mriedem | i noted that with placement, we're doing the ram/disk/cpu filtering in sql on the placement side, but they said that even with that it doesn't filter out enough nodes to make the db query to get compute nodes substantially smaller | |
| 20:25:42 | figleaf | mriedem: too bad scheduling queues never caught on. They would be perfect for that use case | |
| 20:25:50 | mriedem | so i suspect their limiting factor is not filtering on those resource classes, but other things like affinity | |
| 20:26:04 | mriedem | affinity, numa, etc | |
| 20:26:11 | mriedem | stuff we don't have working with placement yet | |
| 20:26:27 | mriedem | i'm still trying to figure out what their most expensive filters are too | |
| 20:26:56 | superdan | mriedem: ugh, that's disappointing | |
| 20:27:14 | mriedem | roughly speaking the scale they are trying to hit, from my limited knowledge so far, is 1024 computes per "cell", 30K nodes total | |
| 20:27:45 | superdan | well, it's probably worth noting that that's pretty high compared to what most people do in a "cell" | |
| 20:27:58 | mriedem | yeah, i just sent some lengthy emails about that last night, | |
| 20:28:07 | mriedem | suggesting they take a page from cern and shard their cells at 200 nodes each | |
| 20:28:11 | superdan | yeah | |
| 20:28:12 | mriedem | so 5 cells per "huawei cell" | |
| 20:28:22 | mriedem | huawei cell = cascaded openstack | |
| 20:28:29 | superdan | right | |
| 20:28:31 | mriedem | == cells v1 | |
| 20:28:36 | mriedem | :) | |
| 20:28:39 | superdan | although it's worth noting that cellsv2 cells won't help them here | |
| 20:28:40 | mriedem | but not cells v1 | |
| 20:28:49 | superdan | because global scheduler | |
| 20:28:52 | mriedem | yup | |
| 20:29:12 | mriedem | i have no idea yet about how their top-layer scheduler picks a cell to send requests to | |
| 20:29:43 | mriedem | nor do i know how large cells v1 deployments like cern are going to deal with moving to the global flat scheduler like in cells v2 | |
| 20:29:53 | mriedem | hopefully we'll get some feedback there someday | |
| 20:29:55 | superdan | well, placement was supposed to be the answer | |
| 20:30:24 | superdan | are we limiting the result size we're getting back from placement yet or are we still trying to pull the whole thing? | |
| 20:30:40 | mriedem | cdent still has a spec up i believe | |
| 20:30:40 | superdan | because if we're returning everything, that means our query to list compute nodes is huge | |
| 20:30:44 | superdan | okay | |
| 20:31:06 | mriedem | https://review.openstack.org/#/c/504540/ | |
| 20:35:22 | fried_rice | Quick, let's merge this pile^Wstack before the zuulv3 cutover https://review.openstack.org/#/c/509029/ | |
| 20:42:26 | tasker | mriedem: sorry to bug you, but is there any movement on https://review.openstack.org/#/c/505771/ ? | |
| 20:43:03 | mriedem | tasker: sorry not yet, it's on my list, but trying to get some stuff done with specs before the 10/19 deadline | |
| 20:43:10 | tasker | roger | |
| 20:43:11 | mriedem | i hope to have that fix polished up next week | |
| 20:43:49 | tasker | thanks! | |
| 20:44:28 | mriedem | give me a kidney punch early next week if you don't see movement | |
| 20:49:27 | openstackgerrit | Eric Young proposed openstack/nova master: 1:Implement ScaleIO image backend https://review.openstack.org/407440 | |
| 20:51:25 | openstackgerrit | Eric Young proposed openstack/nova master: Implement ScaleIO image backend https://review.openstack.org/407440 | |
| 21:03:13 | superdan | fried_rice: one more comment on your patch in case you were respinning right there | |
| 21:03:47 | superdan | mriedem would have shamed me post +2 if I had missed that | |
| 21:10:09 | mriedem | what patch? | |
| 21:11:04 | superdan | his allocation to spawn one | |
| 21:11:16 | superdan | https://review.openstack.org/#/c/511879/ | |
| 21:12:10 | mriedem | not on my radar | |
| 21:12:33 | superdan | it would be just as soon as I put a +2 on it with potential test nits to complain about | |
| 21:12:56 | superdan | I know you have the world's most sophisticated AI developers working on tuning your get-dan-a-disagreement goals | |
| 21:12:56 | mriedem | maybe, i do have that notification system setup | |
| 21:13:01 | mriedem | ha | |
| 21:25:07 | fried_rice | superdan Thanks, I was indeed respinning, though it's a longer road than I expected (gotta handle rebuild too). | |
| 21:25:19 | superdan | ah yeah | |
| 21:30:11 | openstackgerrit | Matt Riedemann proposed openstack/nova-specs master: Spec for volume-backed flavors https://review.openstack.org/511965 | |
| 21:35:40 | superdan | fried_rice: I think the ones above are wrong, but you can usually win on consistency, so fine to leave it I think | |
| 21:36:18 | fried_rice | superdan Roger that. (By "wrong" you mean poor style, not actually programmatically incorrect?) | |
| 21:36:31 | superdan | wrong in the moral sense of the word | |
| 21:36:36 | superdan | as in, "going to hell" | |
| 21:36:37 | fried_rice | Heh, got it. | |
| 22:00:35 | mriedem | on https://review.openstack.org/#/c/504540/ i thought we were also going to limit RPs in SQL, | |
| 22:00:43 | mriedem | but i can understand that would get hairy quickly | |
| 22:01:06 | mriedem | for testing i saw something in devstack about creating a bunch of fake computes, but i doubt it still works | |
| 22:01:15 | mriedem | would be sweet if it did though | |
| 22:01:31 | mriedem | https://github.com/openstack-dev/devstack/blob/master/lib/nova#L868 | |
| 22:01:52 | mriedem | fried_rice: ^ is something i bet you could get working in short order :) | |
| 22:03:11 | fried_rice | mriedem Okay, I'm looking at sh code. What's the context? | |
| 22:03:31 | mriedem | there is a flag in devstack to be able to say run like N number of fake driver nova compute services | |
| 22:03:37 | mriedem | which would give you N compute nodes | |
| 22:03:51 | mriedem | which would be cool for faking out a large deployment for scale testing the scheduler | |
| 22:04:01 | mriedem | i know the fake driver stuff in devstack works, but haven't tested NUMBER_FAKE_NOVA_COMPUTE | |
| 22:04:22 | mriedem | if that would be set to like 100 and actually work, it'd be sweet for testing scheduler performance changes, like cdents limit spec | |
| 22:05:19 | fried_rice | mriedem Gotcha. I'll see if I can try it real quick. | |
| 22:05:32 | mriedem | if not i'll fart around with it next week | |
| 22:05:44 | mriedem | i will do all the things i said i'd ever do, next week | |
| 22:05:45 | fried_rice | mriedem That said, I was thinking about that | |
| 22:06:05 | fried_rice | (getting a biggish test platform to look at performancey things e.g. limits) | |
| 22:06:23 | fried_rice | And really all a guy needs to do is have a placement service running and stuff it full of RPs. | |
| 22:06:23 | mriedem | fried_rice: http://forumtopics.openstack.org/cfp/details/21 | |
| 22:06:28 | mriedem | if you're going to sydney | |
| 22:06:47 | mriedem | well, you also need compute nodes in the db | |
| 22:06:49 | fried_rice | mriedem I'm not. | |
| 22:06:52 | mriedem | doh | |
| 22:07:09 | fried_rice | Got two or three I'll be watching videos after. | |
| 22:07:44 | fried_rice | mriedem I don't think I need compute nodes to do what I'm considering. | |
| 22:07:58 | fried_rice | Cause we're talking about having the limiting done on the placement side | |
| 22:08:08 | fried_rice | initially in the py code, eventually in the sql | |