| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-07-10 | |||
| 20:00:31 | cdent | is this canon: “i guess we assume no one is doing CD” ? If so, life just got a lot easier. | |
| 20:00:49 | mriedem | no, it's not | |
| 20:01:19 | mriedem | dims is supposed to make huawei public cloud doing CD in the next couple of weeks :) | |
| 20:01:25 | cdent | do you take bribes? | |
| 20:02:54 | mriedem | https://docs.openstack.org/nova/latest/project_scope.html#upgrade-expectations | |
| 20:03:01 | mriedem | "In addition, we also support users upgrading from the master branch, technically, between any two commits within the same release cycle." | |
| 20:03:39 | cdent | oh yeah, I know, I was just thinking maybe the world had collapsed | |
| 20:30:03 | jaypipes | mriedem: so... problem. | |
| 20:30:17 | jaypipes | mriedem: I need to do the alternatives stuff before the retries. | |
| 20:30:29 | jaypipes | mriedem: because the retries need to loop over the alternatives. | |
| 20:30:34 | mriedem | right, b/c we need the alternatives to do retries | |
| 20:30:35 | mriedem | yup | |
| 20:30:49 | jaypipes | mriedem: I don't necessarily need to *return* (over RPC) the alternative yet though. | |
| 20:34:20 | mriedem | correctamundo | |
| 20:34:35 | mriedem | so it's like 3 changes on top of this, maybe 2 | |
| 20:34:40 | mriedem | 1. determine alternatives, | |
| 20:34:45 | mriedem | 2. retry on alternatives in scheduler | |
| 20:34:52 | mriedem | 3. pass alternatives to conductor | |
| 20:35:00 | mriedem | and then 4,5,6,7,8 do things with those in the cell | |
| 20:35:05 | jaypipes | yeah | |
| 20:38:50 | jaypipes | mriedem, dansmith: did we ever decide how many alternatives we wanted to return? | |
| 20:39:02 | jaypipes | mriedem, dansmith: limit to 10 per instance UUID maybe? | |
| 20:39:08 | dansmith | jaypipes: CONF.num_retries | |
| 20:39:13 | dansmith | times num_instances | |
| 20:39:13 | jaypipes | ah | |
| 20:39:14 | jaypipes | gotcha. | |
| 20:41:27 | edleafe | jaypipes: this was the consensus a couple of months ago: https://review.openstack.org/#/c/465767/ | |
| 20:41:51 | edleafe | default to CONF.num_retries | |
| 20:42:17 | edleafe | also, won't the order need to be different? | |
| 20:43:06 | edleafe | IOW, claim fails on "best" host, so you try claiming next one. Once you succeed, you select N alternates from the next N hosts in the weighed list that are in the same cell as the successful claimed host | |
| 20:43:18 | jaypipes | edleafe: the consensus was to create a new CONF option build_max_attempts? | |
| 20:44:42 | dansmith | that sounds like a terrible idea | |
| 20:44:45 | dansmith | num_retries and build_max_attempts is highly confusing | |
| 20:45:08 | dansmith | we have the former, I see no reason for a billionth conf var | |
| 20:45:54 | edleafe | jaypipes: sorry, it was supposed to be 'max_attempts' | |
| 20:47:47 | mriedem | plus host_subset_size | |
| 20:48:05 | openstackgerrit | Merged openstack/nova master: placement: scheduler uses allocation candidates https://review.openstack.org/476631 | |
| 20:48:13 | mriedem | we're also going to have to build in the filtering for the alternatives by cell | |
| 20:48:35 | mriedem | i'm guessing we're going to want to cache that as well | |
| 20:48:39 | edleafe | mriedem: host_subset_size only comes into play in the scheduler. The conductor wouldn't use it | |
| 20:48:42 | mriedem | so we don't lookup hostmappings every time we schedule an instance | |
| 20:48:55 | mriedem | edleafe: i was just commenting on the number of existing options | |
| 20:49:01 | dansmith | mriedem: we discussed that yeah, bauzas had a plan for just filtering the alternatives with matching cells | |
| 20:49:17 | dansmith | it was done in his set I think | |
| 20:49:36 | edleafe | it was also done here: https://review.openstack.org/#/c/465768/8/nova/scheduler/filter_scheduler.py | |
| 20:49:37 | mriedem | something else nagging me a bit here, | |
| 20:49:47 | edleafe | (that was before moving to scheduler manager) | |
| 20:50:03 | jaypipes | fuck the instance group shit. | |
| 20:50:10 | mriedem | say it's a create request with 100 servers | |
| 20:50:21 | mriedem | we get our allocation candidates and make claims on them | |
| 20:50:35 | smcginnis | jaypipes: How do you really feel? | |
| 20:50:47 | mriedem | then we're going to send some alternatives down, but couldn't those alternatives be full by the time we're doing with the claims? | |
| 20:50:49 | jaypipes | smcginnis: sorry, that wasn't meant for public consumption. | |
| 20:50:53 | smcginnis | :) | |
| 20:50:53 | mriedem | as the alternatives aren't really a fresh set? | |
| 20:51:18 | mriedem | s/doing/done/ | |
| 20:51:19 | edleafe | mriedem: there could be dupes in the alternatives for each of the 100, too | |
| 20:51:20 | jaypipes | mriedem: yes, you're absolutely correct. this is the problem with num_instances > 1 :( | |
| 20:51:34 | edleafe | mriedem: just like the selected host could be the same for some of the 100 | |
| 20:51:55 | dansmith | jaypipes: fwiw, I was just (minutes ago) helping a real nova user debug a problem with --num_instances=100 :/ | |
| 20:52:01 | mriedem | edleafe: well filtering dupes from the list of alternatives seems trivial, just do a set based on rp uuid | |
| 20:52:14 | edleafe | mriedem: why? | |
| 20:52:23 | mriedem | i get the feeling that huawei's users are also lovers of multiple create | |
| 20:52:30 | edleafe | with the scheduler claiming, the only reason a build will fail will be a host issue | |
| 20:52:32 | mriedem | b/c i've seen bug reports from them about it | |
| 20:52:32 | jaypipes | it's an NFV thing... | |
| 20:52:51 | edleafe | so most will succeed, and we throw away the alternatives | |
| 20:52:55 | mriedem | edleafe: i assume you and i are talking about different things then | |
| 20:52:57 | dansmith | while googling for something, I also found a bug filed against another platform with "nova has --num_instances and I use it a lot, can't use your platform without it...." | |
| 20:53:09 | mriedem | ha, that's a first | |
| 20:53:16 | mriedem | 'every other cloud has spot instances but nova *doesn't* have it' | |
| 20:53:31 | edleafe | mriedem: what are you referring to as "alternatives"? | |
| 20:54:23 | jaypipes | so here's the thing... I could change the scheduler manager to get rid of all the "consume this stuff within the _schedule() filter scheduler loop" and call the placement API for allocation candidates once for each instance in num_instances. | |
| 20:54:44 | jaypipes | the problem with that is the "allocate 99 instances and the last one fails" problem that large Heat stacks have. | |
| 20:54:53 | edleafe | mriedem: I mean the set of N potential hosts we return with the selected host to the conductor. That's passed to the cell, and if the build fails, the cell conductor does retries with the alternatives | |
| 20:55:52 | mriedem | edleafe: yes, but what are you saying about duplicates? | |
| 20:55:59 | mriedem | wrt alternatives sent to the cell? | |
| 20:56:27 | mriedem | do you mean like 2 alternatives pointing at the same compute node for ram/cpu but different storage providers? (one local, one shared)? | |
| 20:56:59 | mriedem | jaypipes: also, if we call placement for allocation candidates for each instance, that would greatly slow things down wouldn't it? | |
| 20:57:24 | dansmith | that would suck for 100-instance batches, yeah | |
| 20:57:32 | mriedem | or is that a loop of (1) get allocatoin candidates, (2) filter/weigh (3) claim, (4) repeat | |
| 20:57:38 | jaypipes | mriedem: that scenario is actually *exactly* what I benchmarked here: https://github.com/jaypipes/placement-bench/ | |
| 20:57:57 | edleafe | mriedem: if we're returning 100 sets of (selected host, alt1, alt2, ...altN) to the conductor, the same host could appear in multiple alternate lists | |
| 20:58:02 | jaypipes | mriedem: and I was able to achieve >200 claimed requests *per second*. | |
| 20:58:32 | mriedem | jaypipes: what is a 'claimed request'? | |
| 20:58:51 | jaypipes | mriedem: request for resources, get list of providers, claim for an instance against a selected provider. | |
| 20:58:53 | mriedem | that's the full thing through select_destinations to placement to filter/weigh to claim? | |
| 20:58:56 | jaypipes | right. | |
| 20:59:43 | mriedem | edleafe: yes agree | |
| 20:59:47 | mriedem | edleafe: by default it's 3 | |
| 20:59:48 | mriedem | but yes | |
| 20:59:51 | jaypipes | mriedem: I'm think maybe it would be better to scrap trying to make this square peg fit the filter scheduler round hole. | |
| 21:00:40 | mriedem | what is the square peg in this analogy? | |
| 21:00:43 | mriedem | multiple create? | |
| 21:00:48 | dansmith | edleafe | |
| 21:00:51 | dansmith | oh did I say that? | |
| 21:00:51 | mriedem | ha | |
| 21:00:55 | jaypipes | mriedem: and let me move the _schedule() internals into the caching scheduler and make the filter scheduler call placement each iteration of the num_instances loop... | |
| 21:01:02 | mriedem | he's no square daddy-o | |
| 21:01:37 | dansmith | I dunno, I'm highly skeptical of the efficiency of doing one call per instance with 100 instances | |