| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-07-10 | |||
| 20:49:01 | dansmith | mriedem: we discussed that yeah, bauzas had a plan for just filtering the alternatives with matching cells | |
| 20:49:17 | dansmith | it was done in his set I think | |
| 20:49:36 | edleafe | it was also done here: https://review.openstack.org/#/c/465768/8/nova/scheduler/filter_scheduler.py | |
| 20:49:37 | mriedem | something else nagging me a bit here, | |
| 20:49:47 | edleafe | (that was before moving to scheduler manager) | |
| 20:50:03 | jaypipes | fuck the instance group shit. | |
| 20:50:10 | mriedem | say it's a create request with 100 servers | |
| 20:50:21 | mriedem | we get our allocation candidates and make claims on them | |
| 20:50:35 | smcginnis | jaypipes: How do you really feel? | |
| 20:50:47 | mriedem | then we're going to send some alternatives down, but couldn't those alternatives be full by the time we're doing with the claims? | |
| 20:50:49 | jaypipes | smcginnis: sorry, that wasn't meant for public consumption. | |
| 20:50:53 | mriedem | as the alternatives aren't really a fresh set? | |
| 20:50:53 | smcginnis | :) | |
| 20:51:18 | mriedem | s/doing/done/ | |
| 20:51:19 | edleafe | mriedem: there could be dupes in the alternatives for each of the 100, too | |
| 20:51:20 | jaypipes | mriedem: yes, you're absolutely correct. this is the problem with num_instances > 1 :( | |
| 20:51:34 | edleafe | mriedem: just like the selected host could be the same for some of the 100 | |
| 20:51:55 | dansmith | jaypipes: fwiw, I was just (minutes ago) helping a real nova user debug a problem with --num_instances=100 :/ | |
| 20:52:01 | mriedem | edleafe: well filtering dupes from the list of alternatives seems trivial, just do a set based on rp uuid | |
| 20:52:14 | edleafe | mriedem: why? | |
| 20:52:23 | mriedem | i get the feeling that huawei's users are also lovers of multiple create | |
| 20:52:30 | edleafe | with the scheduler claiming, the only reason a build will fail will be a host issue | |
| 20:52:32 | jaypipes | it's an NFV thing... | |
| 20:52:32 | mriedem | b/c i've seen bug reports from them about it | |
| 20:52:51 | edleafe | so most will succeed, and we throw away the alternatives | |
| 20:52:55 | mriedem | edleafe: i assume you and i are talking about different things then | |
| 20:52:57 | dansmith | while googling for something, I also found a bug filed against another platform with "nova has --num_instances and I use it a lot, can't use your platform without it...." | |
| 20:53:09 | mriedem | ha, that's a first | |
| 20:53:16 | mriedem | 'every other cloud has spot instances but nova *doesn't* have it' | |
| 20:53:31 | edleafe | mriedem: what are you referring to as "alternatives"? | |
| 20:54:23 | jaypipes | so here's the thing... I could change the scheduler manager to get rid of all the "consume this stuff within the _schedule() filter scheduler loop" and call the placement API for allocation candidates once for each instance in num_instances. | |
| 20:54:44 | jaypipes | the problem with that is the "allocate 99 instances and the last one fails" problem that large Heat stacks have. | |
| 20:54:53 | edleafe | mriedem: I mean the set of N potential hosts we return with the selected host to the conductor. That's passed to the cell, and if the build fails, the cell conductor does retries with the alternatives | |
| 20:55:52 | mriedem | edleafe: yes, but what are you saying about duplicates? | |
| 20:55:59 | mriedem | wrt alternatives sent to the cell? | |
| 20:56:27 | mriedem | do you mean like 2 alternatives pointing at the same compute node for ram/cpu but different storage providers? (one local, one shared)? | |
| 20:56:59 | mriedem | jaypipes: also, if we call placement for allocation candidates for each instance, that would greatly slow things down wouldn't it? | |
| 20:57:24 | dansmith | that would suck for 100-instance batches, yeah | |
| 20:57:32 | mriedem | or is that a loop of (1) get allocatoin candidates, (2) filter/weigh (3) claim, (4) repeat | |
| 20:57:38 | jaypipes | mriedem: that scenario is actually *exactly* what I benchmarked here: https://github.com/jaypipes/placement-bench/ | |
| 20:57:57 | edleafe | mriedem: if we're returning 100 sets of (selected host, alt1, alt2, ...altN) to the conductor, the same host could appear in multiple alternate lists | |
| 20:58:02 | jaypipes | mriedem: and I was able to achieve >200 claimed requests *per second*. | |
| 20:58:32 | mriedem | jaypipes: what is a 'claimed request'? | |
| 20:58:51 | jaypipes | mriedem: request for resources, get list of providers, claim for an instance against a selected provider. | |
| 20:58:53 | mriedem | that's the full thing through select_destinations to placement to filter/weigh to claim? | |
| 20:58:56 | jaypipes | right. | |
| 20:59:43 | mriedem | edleafe: yes agree | |
| 20:59:47 | mriedem | edleafe: by default it's 3 | |
| 20:59:48 | mriedem | but yes | |
| 20:59:51 | jaypipes | mriedem: I'm think maybe it would be better to scrap trying to make this square peg fit the filter scheduler round hole. | |
| 21:00:40 | mriedem | what is the square peg in this analogy? | |
| 21:00:43 | mriedem | multiple create? | |
| 21:00:48 | dansmith | edleafe | |
| 21:00:51 | mriedem | ha | |
| 21:00:51 | dansmith | oh did I say that? | |
| 21:00:55 | jaypipes | mriedem: and let me move the _schedule() internals into the caching scheduler and make the filter scheduler call placement each iteration of the num_instances loop... | |
| 21:01:02 | mriedem | he's no square daddy-o | |
| 21:01:37 | dansmith | I dunno, I'm highly skeptical of the efficiency of doing one call per instance with 100 instances | |
| 21:01:49 | edleafe | dansmith: I sure feel like it | |
| 21:01:50 | jaypipes | dansmith: well, actually.. | |
| 21:02:08 | jaypipes | dansmith: I *am* returning the provider_summaries, which has the remaining capacity for various resources in it. | |
| 21:02:12 | dansmith | even if we *can* do it, that still might be a lot of network, a lot of CPU, etc, and if there's any latency between placement and scheduler... | |
| 21:02:25 | dansmith | jaypipes: right, see, what I would do is, | |
| 21:02:45 | dansmith | ask for some batch N, and consume from there until a claim fails, and then do another batch until you've processed num_instances | |
| 21:02:47 | jaypipes | deduct from the capacity on each loop... | |
| 21:03:05 | jaypipes | which is pretty much what the _schedule() method is doing right now for the host state objects. | |
| 21:03:30 | mriedem | that's "chosen_host.obj.consume_from_request(spec_obj)" right? | |
| 21:03:32 | dansmith | well, if you try to predict capacity, you might be missing some things that placement knows about, and definitely other schedulers are claiming | |
| 21:03:33 | dims | mriedem : LOL | |
| 21:03:48 | dansmith | so if you just claim until you get a failure and then ask for a new batch I think you're safer | |
| 21:03:58 | jaypipes | mriedem: yeah, kinda. | |
| 21:04:01 | dansmith | that could be every request if things are chaotic, but if not, then you've saved a bunch of work I think | |
| 21:04:43 | jaypipes | dansmith: that's my problem with doing that... we get the cache invalidation problem :( | |
| 21:05:14 | dansmith | right, so you claim until fail, then get another batch of candidates, IMHO | |
| 21:05:23 | dansmith | jaypipes: the other thing that sucks about --min, which is what I was debugging just now, | |
| 21:05:31 | dansmith | is that --min only works for quota, | |
| 21:05:42 | dansmith | and if your max is 100 and you can only fit 93, you fail the whole deal, | |
| 21:05:46 | dansmith | even if --min=1 | |
| 21:05:55 | dansmith | which is pretty stupid and not what the user expects | |
| 21:06:04 | dansmith | so it would be nice to fix that in the process of doing this, if we could | |
| 21:06:21 | jaypipes | this whole num_instances thing is stupid to begin with. | |
| 21:06:42 | dansmith | except for all the users that actively use it | |
| 21:06:47 | dansmith | https://bugs.launchpad.net/nova/+bug/1703457 | |
| 21:06:48 | openstack | Launchpad bug 1703457 in OpenStack Compute (nova) "RFE: More intuitive usage of --min and --max arguments in openstack server create" [Undecided,New] | |
| 21:06:53 | dansmith | not from an nfv workload ^ btw | |
| 21:07:14 | jaypipes | yeah, but it should be an orchestration layer that handles all the crazy. not the Nova scheduler. | |
| 21:07:34 | jaypipes | like I don't know.. Heat with convergence? :( | |
| 21:07:43 | dansmith | well, you can say that for sure. they're not using heat or anything else, and we have it in our api... | |
| 21:08:21 | dansmith | given the number of people that seem to want it, | |
| 21:08:42 | dansmith | I think we either keep supporting it, or we're going to be asked to do reservations for that orchestrator thing | |
| 21:10:08 | dansmith | at a minimum, we should either fix (or allow fixes) for the above kind of stupid thing, or we should cut the whole bit of it out of the api with a microversion | |
| 21:10:19 | melwitt | yeah, cfriesen opened a bug about failing a request where >= min_count instances were successful but it fails bc it couldn't satisfy max_count, two years ago https://bugs.launchpad.net/nova/+bug/1458122 | |
| 21:10:21 | openstack | Launchpad bug 1458122 in OpenStack Compute (nova) "nova shouldn't error if we can't schedule all of max_count instances at boot time" [Wishlist,Opinion] - Assigned to Chris Friesen (cbf123) | |
| 21:10:38 | mriedem | dansmith: even if you cut it out of the api with a microversion, there are all of the microversion before it where it lives | |
| 21:10:42 | dansmith | but for a cloud that supports an application and doesn't use quota at all, --min-instances is literally ineffectual | |
| 21:11:11 | dansmith | mriedem: well, I know, and I think we should just keep supporting it given the popularity, but if we're going to let it rot, we should signal that we're not going to fix things | |
| 21:11:19 | dansmith | melwitt: ah yep | |
| 21:12:43 | melwitt | I know they use the "request multiple instances" a lot at yahoo too | |
| 21:12:45 | dansmith | anyway, I gotta run let some creeper feel my eyeballs | |
| 21:12:47 | cdent | can somebody summarize/recapitulate the problem; i’ve been following along but it is coming in bits and pieces I’m not sure I get it. | |