Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-10-13
20:25:42 figleaf mriedem: too bad scheduling queues never caught on. They would be perfect for that use case
20:25:50 mriedem so i suspect their limiting factor is not filtering on those resource classes, but other things like affinity
20:26:04 mriedem affinity, numa, etc
20:26:11 mriedem stuff we don't have working with placement yet
20:26:27 mriedem i'm still trying to figure out what their most expensive filters are too
20:26:56 superdan mriedem: ugh, that's disappointing
20:27:14 mriedem roughly speaking the scale they are trying to hit, from my limited knowledge so far, is 1024 computes per "cell", 30K nodes total
20:27:45 superdan well, it's probably worth noting that that's pretty high compared to what most people do in a "cell"
20:27:58 mriedem yeah, i just sent some lengthy emails about that last night,
20:28:07 mriedem suggesting they take a page from cern and shard their cells at 200 nodes each
20:28:11 superdan yeah
20:28:12 mriedem so 5 cells per "huawei cell"
20:28:22 mriedem huawei cell = cascaded openstack
20:28:29 superdan right
20:28:31 mriedem == cells v1
20:28:36 mriedem :)
20:28:39 superdan although it's worth noting that cellsv2 cells won't help them here
20:28:40 mriedem but not cells v1
20:28:49 superdan because global scheduler
20:28:52 mriedem yup
20:29:12 mriedem i have no idea yet about how their top-layer scheduler picks a cell to send requests to
20:29:43 mriedem nor do i know how large cells v1 deployments like cern are going to deal with moving to the global flat scheduler like in cells v2
20:29:53 mriedem hopefully we'll get some feedback there someday
20:29:55 superdan well, placement was supposed to be the answer
20:30:24 superdan are we limiting the result size we're getting back from placement yet or are we still trying to pull the whole thing?
20:30:40 mriedem cdent still has a spec up i believe
20:30:40 superdan because if we're returning everything, that means our query to list compute nodes is huge
20:30:44 superdan okay
20:31:06 mriedem https://review.openstack.org/#/c/504540/
20:35:22 fried_rice Quick, let's merge this pile^Wstack before the zuulv3 cutover https://review.openstack.org/#/c/509029/
20:42:26 tasker mriedem: sorry to bug you, but is there any movement on https://review.openstack.org/#/c/505771/ ?
20:43:03 mriedem tasker: sorry not yet, it's on my list, but trying to get some stuff done with specs before the 10/19 deadline
20:43:10 tasker roger
20:43:11 mriedem i hope to have that fix polished up next week
20:43:49 tasker thanks!
20:44:28 mriedem give me a kidney punch early next week if you don't see movement
20:49:27 openstackgerrit Eric Young proposed openstack/nova master: 1:Implement ScaleIO image backend https://review.openstack.org/407440
20:51:25 openstackgerrit Eric Young proposed openstack/nova master: Implement ScaleIO image backend https://review.openstack.org/407440
21:03:13 superdan fried_rice: one more comment on your patch in case you were respinning right there
21:03:47 superdan mriedem would have shamed me post +2 if I had missed that
21:10:09 mriedem what patch?
21:11:04 superdan his allocation to spawn one
21:11:16 superdan https://review.openstack.org/#/c/511879/
21:12:10 mriedem not on my radar
21:12:33 superdan it would be just as soon as I put a +2 on it with potential test nits to complain about
21:12:56 superdan I know you have the world's most sophisticated AI developers working on tuning your get-dan-a-disagreement goals
21:12:56 mriedem maybe, i do have that notification system setup
21:13:01 mriedem ha
21:25:07 fried_rice superdan Thanks, I was indeed respinning, though it's a longer road than I expected (gotta handle rebuild too).
21:25:19 superdan ah yeah
21:30:11 openstackgerrit Matt Riedemann proposed openstack/nova-specs master: Spec for volume-backed flavors https://review.openstack.org/511965
21:35:40 superdan fried_rice: I think the ones above are wrong, but you can usually win on consistency, so fine to leave it I think
21:36:18 fried_rice superdan Roger that. (By "wrong" you mean poor style, not actually programmatically incorrect?)
21:36:31 superdan wrong in the moral sense of the word
21:36:36 superdan as in, "going to hell"
21:36:37 fried_rice Heh, got it.
22:00:35 mriedem on https://review.openstack.org/#/c/504540/ i thought we were also going to limit RPs in SQL,
22:00:43 mriedem but i can understand that would get hairy quickly
22:01:06 mriedem for testing i saw something in devstack about creating a bunch of fake computes, but i doubt it still works
22:01:15 mriedem would be sweet if it did though
22:01:31 mriedem https://github.com/openstack-dev/devstack/blob/master/lib/nova#L868
22:01:52 mriedem fried_rice: ^ is something i bet you could get working in short order :)
22:03:11 fried_rice mriedem Okay, I'm looking at sh code. What's the context?
22:03:31 mriedem there is a flag in devstack to be able to say run like N number of fake driver nova compute services
22:03:37 mriedem which would give you N compute nodes
22:03:51 mriedem which would be cool for faking out a large deployment for scale testing the scheduler
22:04:01 mriedem i know the fake driver stuff in devstack works, but haven't tested NUMBER_FAKE_NOVA_COMPUTE
22:04:22 mriedem if that would be set to like 100 and actually work, it'd be sweet for testing scheduler performance changes, like cdents limit spec
22:05:19 fried_rice mriedem Gotcha. I'll see if I can try it real quick.
22:05:32 mriedem if not i'll fart around with it next week
22:05:44 mriedem i will do all the things i said i'd ever do, next week
22:05:45 fried_rice mriedem That said, I was thinking about that
22:06:05 fried_rice (getting a biggish test platform to look at performancey things e.g. limits)
22:06:23 fried_rice And really all a guy needs to do is have a placement service running and stuff it full of RPs.
22:06:23 mriedem fried_rice: http://forumtopics.openstack.org/cfp/details/21
22:06:28 mriedem if you're going to sydney
22:06:47 mriedem well, you also need compute nodes in the db
22:06:49 fried_rice mriedem I'm not.
22:06:52 mriedem doh
22:07:09 fried_rice Got two or three I'll be watching videos after.
22:07:44 fried_rice mriedem I don't think I need compute nodes to do what I'm considering.
22:07:58 fried_rice Cause we're talking about having the limiting done on the placement side
22:08:08 fried_rice initially in the py code, eventually in the sql
22:08:13 fried_rice but on the placement side for sure
22:08:39 fried_rice So it seems like I could just stuff the db full of RPs and inventories, and then do plain ol' GET /allocation_candidates calls with a timer.
22:09:24 mriedem oh yeah, for this, sure
22:09:28 mriedem yup
22:09:39 mriedem oh and btw, we also have some osc support for RPs too now
22:09:55 mriedem so you can create them and such - there are some changes up for stuffing the inventory and allocations which you could pull down
22:09:59 fried_rice Yeah, was seeing that pucking around, wondering how far it had gotten.
22:10:01 mriedem if you didn't want to do a bunch of curling
22:10:06 mriedem it's moving
22:10:15 fried_rice Actually I gots me a little script to talk to placement.
22:10:31 mriedem reminds me, https://review.openstack.org/#/c/457533/ needs some small tweaks and then i'd +2 that
22:10:58 fried_rice Though in this case I'd rather get in an interactive so I don't have to reconnect with every call.
22:11:52 fried_rice Anyway, that's been a back-of-head thing for a couple weeks now. I'm a bit deep at the moment, and it's late on a Friday, so "next week".
22:12:02 mriedem i'll see you next week then
22:12:04 mriedem when all things will get done
22:16:01 openstackgerrit Merged openstack/nova master: rp: remove CRUD operations on Inventory class https://review.openstack.org/509029
22:16:47 openstackgerrit Merged openstack/nova master: rp: streamline InventoryList.get_all_by_rp_uuid() https://review.openstack.org/509030

Earlier   Later