Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-07-10
19:38:42 dansmith I guess we're not likely to see one from sdague in the near future either, eh?
19:39:12 mriedem probably not. i plan on reviewing that quotas one.
19:39:26 jaypipes mriedem: I will put a patch up renaming delete_allocation_for_instance() to delete_allocations_for_consumer() to remedy my need for grammatical correctness.
19:43:34 sdague dansmith: yeh, not this week
19:44:00 sdague I can get my head back into it next week, but I don't know which dates we're racing against now
19:44:19 dansmith yeah, need it before next week I think
19:52:26 mriedem sdague: feature freeze is july 27
19:52:31 mriedem so we're less than 3 weeks from FF
19:53:09 mriedem jaypipes: a couple of comments in https://review.openstack.org/#/c/476632/
19:53:25 mriedem jaypipes: dansmith: in a nutshell on that one,
19:53:47 mriedem i think it means we potentially have problems for bursts of instances in the same request with that change until we start doing retries in the scheduler
19:54:08 mriedem and/or we start randomizing which allocation request match we pick for a given compute node
19:55:39 jaypipes mriedem: do you want me to do retries in this patch?
19:55:49 jaypipes mriedem: I was working on that in a separate followup.
19:56:04 jaypipes mriedem: but if you're concerned about this, I suppose I could put it all in one patch.
19:56:11 jaypipes mriedem: was trying to keep the LOCC low.
19:56:16 mriedem a single change makes it harder to review for sure
19:56:17 mriedem yeah
19:56:42 mriedem i guess we assume no one is doing CD
19:57:12 mriedem jaypipes: how about line up the retries change on top,
19:57:13 jaypipes mriedem: answered you.
19:57:16 mriedem and then we'll get them in order
19:57:22 jaypipes mriedem: sure thing.
19:57:22 mriedem so there isn't like a week between changes
19:57:30 jaypipes mriedem: gimme about an hour.
20:00:31 cdent is this canon: “i guess we assume no one is doing CD” ? If so, life just got a lot easier.
20:00:49 mriedem no, it's not
20:01:19 mriedem dims is supposed to make huawei public cloud doing CD in the next couple of weeks :)
20:01:25 cdent do you take bribes?
20:02:54 mriedem https://docs.openstack.org/nova/latest/project_scope.html#upgrade-expectations
20:03:01 mriedem "In addition, we also support users upgrading from the master branch, technically, between any two commits within the same release cycle."
20:03:39 cdent oh yeah, I know, I was just thinking maybe the world had collapsed
20:30:03 jaypipes mriedem: so... problem.
20:30:17 jaypipes mriedem: I need to do the alternatives stuff before the retries.
20:30:29 jaypipes mriedem: because the retries need to loop over the alternatives.
20:30:34 mriedem right, b/c we need the alternatives to do retries
20:30:35 mriedem yup
20:30:49 jaypipes mriedem: I don't necessarily need to *return* (over RPC) the alternative yet though.
20:34:20 mriedem correctamundo
20:34:35 mriedem so it's like 3 changes on top of this, maybe 2
20:34:40 mriedem 1. determine alternatives,
20:34:45 mriedem 2. retry on alternatives in scheduler
20:34:52 mriedem 3. pass alternatives to conductor
20:35:00 mriedem and then 4,5,6,7,8 do things with those in the cell
20:35:05 jaypipes yeah
20:38:50 jaypipes mriedem, dansmith: did we ever decide how many alternatives we wanted to return?
20:39:02 jaypipes mriedem, dansmith: limit to 10 per instance UUID maybe?
20:39:08 dansmith jaypipes: CONF.num_retries
20:39:13 dansmith times num_instances
20:39:13 jaypipes ah
20:39:14 jaypipes gotcha.
20:41:27 edleafe jaypipes: this was the consensus a couple of months ago: https://review.openstack.org/#/c/465767/
20:41:51 edleafe default to CONF.num_retries
20:42:17 edleafe also, won't the order need to be different?
20:43:06 edleafe IOW, claim fails on "best" host, so you try claiming next one. Once you succeed, you select N alternates from the next N hosts in the weighed list that are in the same cell as the successful claimed host
20:43:18 jaypipes edleafe: the consensus was to create a new CONF option build_max_attempts?
20:44:42 dansmith that sounds like a terrible idea
20:44:45 dansmith num_retries and build_max_attempts is highly confusing
20:45:08 dansmith we have the former, I see no reason for a billionth conf var
20:45:54 edleafe jaypipes: sorry, it was supposed to be 'max_attempts'
20:47:47 mriedem plus host_subset_size
20:48:05 openstackgerrit Merged openstack/nova master: placement: scheduler uses allocation candidates https://review.openstack.org/476631
20:48:13 mriedem we're also going to have to build in the filtering for the alternatives by cell
20:48:35 mriedem i'm guessing we're going to want to cache that as well
20:48:39 edleafe mriedem: host_subset_size only comes into play in the scheduler. The conductor wouldn't use it
20:48:42 mriedem so we don't lookup hostmappings every time we schedule an instance
20:48:55 mriedem edleafe: i was just commenting on the number of existing options
20:49:01 dansmith mriedem: we discussed that yeah, bauzas had a plan for just filtering the alternatives with matching cells
20:49:17 dansmith it was done in his set I think
20:49:36 edleafe it was also done here: https://review.openstack.org/#/c/465768/8/nova/scheduler/filter_scheduler.py
20:49:37 mriedem something else nagging me a bit here,
20:49:47 edleafe (that was before moving to scheduler manager)
20:50:03 jaypipes fuck the instance group shit.
20:50:10 mriedem say it's a create request with 100 servers
20:50:21 mriedem we get our allocation candidates and make claims on them
20:50:35 smcginnis jaypipes: How do you really feel?
20:50:47 mriedem then we're going to send some alternatives down, but couldn't those alternatives be full by the time we're doing with the claims?
20:50:49 jaypipes smcginnis: sorry, that wasn't meant for public consumption.
20:50:53 smcginnis :)
20:50:53 mriedem as the alternatives aren't really a fresh set?
20:51:18 mriedem s/doing/done/
20:51:19 edleafe mriedem: there could be dupes in the alternatives for each of the 100, too
20:51:20 jaypipes mriedem: yes, you're absolutely correct. this is the problem with num_instances > 1 :(
20:51:34 edleafe mriedem: just like the selected host could be the same for some of the 100
20:51:55 dansmith jaypipes: fwiw, I was just (minutes ago) helping a real nova user debug a problem with --num_instances=100 :/
20:52:01 mriedem edleafe: well filtering dupes from the list of alternatives seems trivial, just do a set based on rp uuid
20:52:14 edleafe mriedem: why?
20:52:23 mriedem i get the feeling that huawei's users are also lovers of multiple create
20:52:30 edleafe with the scheduler claiming, the only reason a build will fail will be a host issue
20:52:32 mriedem b/c i've seen bug reports from them about it
20:52:32 jaypipes it's an NFV thing...
20:52:51 edleafe so most will succeed, and we throw away the alternatives
20:52:55 mriedem edleafe: i assume you and i are talking about different things then
20:52:57 dansmith while googling for something, I also found a bug filed against another platform with "nova has --num_instances and I use it a lot, can't use your platform without it...."
20:53:09 mriedem ha, that's a first
20:53:16 mriedem 'every other cloud has spot instances but nova *doesn't* have it'
20:53:31 edleafe mriedem: what are you referring to as "alternatives"?
20:54:23 jaypipes so here's the thing... I could change the scheduler manager to get rid of all the "consume this stuff within the _schedule() filter scheduler loop" and call the placement API for allocation candidates once for each instance in num_instances.
20:54:44 jaypipes the problem with that is the "allocate 99 instances and the last one fails" problem that large Heat stacks have.
20:54:53 edleafe mriedem: I mean the set of N potential hosts we return with the selected host to the conductor. That's passed to the cell, and if the build fails, the cell conductor does retries with the alternatives

Earlier   Later