Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-07-10
19:56:11 jaypipes mriedem: was trying to keep the LOCC low.
19:56:16 mriedem a single change makes it harder to review for sure
19:56:17 mriedem yeah
19:56:42 mriedem i guess we assume no one is doing CD
19:57:12 mriedem jaypipes: how about line up the retries change on top,
19:57:13 jaypipes mriedem: answered you.
19:57:16 mriedem and then we'll get them in order
19:57:22 mriedem so there isn't like a week between changes
19:57:22 jaypipes mriedem: sure thing.
19:57:30 jaypipes mriedem: gimme about an hour.
20:00:31 cdent is this canon: “i guess we assume no one is doing CD” ? If so, life just got a lot easier.
20:00:49 mriedem no, it's not
20:01:19 mriedem dims is supposed to make huawei public cloud doing CD in the next couple of weeks :)
20:01:25 cdent do you take bribes?
20:02:54 mriedem https://docs.openstack.org/nova/latest/project_scope.html#upgrade-expectations
20:03:01 mriedem "In addition, we also support users upgrading from the master branch, technically, between any two commits within the same release cycle."
20:03:39 cdent oh yeah, I know, I was just thinking maybe the world had collapsed
20:30:03 jaypipes mriedem: so... problem.
20:30:17 jaypipes mriedem: I need to do the alternatives stuff before the retries.
20:30:29 jaypipes mriedem: because the retries need to loop over the alternatives.
20:30:34 mriedem right, b/c we need the alternatives to do retries
20:30:35 mriedem yup
20:30:49 jaypipes mriedem: I don't necessarily need to *return* (over RPC) the alternative yet though.
20:34:20 mriedem correctamundo
20:34:35 mriedem so it's like 3 changes on top of this, maybe 2
20:34:40 mriedem 1. determine alternatives,
20:34:45 mriedem 2. retry on alternatives in scheduler
20:34:52 mriedem 3. pass alternatives to conductor
20:35:00 mriedem and then 4,5,6,7,8 do things with those in the cell
20:35:05 jaypipes yeah
20:38:50 jaypipes mriedem, dansmith: did we ever decide how many alternatives we wanted to return?
20:39:02 jaypipes mriedem, dansmith: limit to 10 per instance UUID maybe?
20:39:08 dansmith jaypipes: CONF.num_retries
20:39:13 jaypipes ah
20:39:13 dansmith times num_instances
20:39:14 jaypipes gotcha.
20:41:27 edleafe jaypipes: this was the consensus a couple of months ago: https://review.openstack.org/#/c/465767/
20:41:51 edleafe default to CONF.num_retries
20:42:17 edleafe also, won't the order need to be different?
20:43:06 edleafe IOW, claim fails on "best" host, so you try claiming next one. Once you succeed, you select N alternates from the next N hosts in the weighed list that are in the same cell as the successful claimed host
20:43:18 jaypipes edleafe: the consensus was to create a new CONF option build_max_attempts?
20:44:42 dansmith that sounds like a terrible idea
20:44:45 dansmith num_retries and build_max_attempts is highly confusing
20:45:08 dansmith we have the former, I see no reason for a billionth conf var
20:45:54 edleafe jaypipes: sorry, it was supposed to be 'max_attempts'
20:47:47 mriedem plus host_subset_size
20:48:05 openstackgerrit Merged openstack/nova master: placement: scheduler uses allocation candidates https://review.openstack.org/476631
20:48:13 mriedem we're also going to have to build in the filtering for the alternatives by cell
20:48:35 mriedem i'm guessing we're going to want to cache that as well
20:48:39 edleafe mriedem: host_subset_size only comes into play in the scheduler. The conductor wouldn't use it
20:48:42 mriedem so we don't lookup hostmappings every time we schedule an instance
20:48:55 mriedem edleafe: i was just commenting on the number of existing options
20:49:01 dansmith mriedem: we discussed that yeah, bauzas had a plan for just filtering the alternatives with matching cells
20:49:17 dansmith it was done in his set I think
20:49:36 edleafe it was also done here: https://review.openstack.org/#/c/465768/8/nova/scheduler/filter_scheduler.py
20:49:37 mriedem something else nagging me a bit here,
20:49:47 edleafe (that was before moving to scheduler manager)
20:50:03 jaypipes fuck the instance group shit.
20:50:10 mriedem say it's a create request with 100 servers
20:50:21 mriedem we get our allocation candidates and make claims on them
20:50:35 smcginnis jaypipes: How do you really feel?
20:50:47 mriedem then we're going to send some alternatives down, but couldn't those alternatives be full by the time we're doing with the claims?
20:50:49 jaypipes smcginnis: sorry, that wasn't meant for public consumption.
20:50:53 mriedem as the alternatives aren't really a fresh set?
20:50:53 smcginnis :)
20:51:18 mriedem s/doing/done/
20:51:19 edleafe mriedem: there could be dupes in the alternatives for each of the 100, too
20:51:20 jaypipes mriedem: yes, you're absolutely correct. this is the problem with num_instances > 1 :(
20:51:34 edleafe mriedem: just like the selected host could be the same for some of the 100
20:51:55 dansmith jaypipes: fwiw, I was just (minutes ago) helping a real nova user debug a problem with --num_instances=100 :/
20:52:01 mriedem edleafe: well filtering dupes from the list of alternatives seems trivial, just do a set based on rp uuid
20:52:14 edleafe mriedem: why?
20:52:23 mriedem i get the feeling that huawei's users are also lovers of multiple create
20:52:30 edleafe with the scheduler claiming, the only reason a build will fail will be a host issue
20:52:32 jaypipes it's an NFV thing...
20:52:32 mriedem b/c i've seen bug reports from them about it
20:52:51 edleafe so most will succeed, and we throw away the alternatives
20:52:55 mriedem edleafe: i assume you and i are talking about different things then
20:52:57 dansmith while googling for something, I also found a bug filed against another platform with "nova has --num_instances and I use it a lot, can't use your platform without it...."
20:53:09 mriedem ha, that's a first
20:53:16 mriedem 'every other cloud has spot instances but nova *doesn't* have it'
20:53:31 edleafe mriedem: what are you referring to as "alternatives"?
20:54:23 jaypipes so here's the thing... I could change the scheduler manager to get rid of all the "consume this stuff within the _schedule() filter scheduler loop" and call the placement API for allocation candidates once for each instance in num_instances.
20:54:44 jaypipes the problem with that is the "allocate 99 instances and the last one fails" problem that large Heat stacks have.
20:54:53 edleafe mriedem: I mean the set of N potential hosts we return with the selected host to the conductor. That's passed to the cell, and if the build fails, the cell conductor does retries with the alternatives
20:55:52 mriedem edleafe: yes, but what are you saying about duplicates?
20:55:59 mriedem wrt alternatives sent to the cell?
20:56:27 mriedem do you mean like 2 alternatives pointing at the same compute node for ram/cpu but different storage providers? (one local, one shared)?
20:56:59 mriedem jaypipes: also, if we call placement for allocation candidates for each instance, that would greatly slow things down wouldn't it?
20:57:24 dansmith that would suck for 100-instance batches, yeah
20:57:32 mriedem or is that a loop of (1) get allocatoin candidates, (2) filter/weigh (3) claim, (4) repeat
20:57:38 jaypipes mriedem: that scenario is actually *exactly* what I benchmarked here: https://github.com/jaypipes/placement-bench/
20:57:57 edleafe mriedem: if we're returning 100 sets of (selected host, alt1, alt2, ...altN) to the conductor, the same host could appear in multiple alternate lists
20:58:02 jaypipes mriedem: and I was able to achieve >200 claimed requests *per second*.
20:58:32 mriedem jaypipes: what is a 'claimed request'?
20:58:51 jaypipes mriedem: request for resources, get list of providers, claim for an instance against a selected provider.
20:58:53 mriedem that's the full thing through select_destinations to placement to filter/weigh to claim?
20:58:56 jaypipes right.
20:59:43 mriedem edleafe: yes agree
20:59:47 mriedem edleafe: by default it's 3

Earlier   Later