| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-07-10 | |||
| 20:52:57 | dansmith | while googling for something, I also found a bug filed against another platform with "nova has --num_instances and I use it a lot, can't use your platform without it...." | |
| 20:53:09 | mriedem | ha, that's a first | |
| 20:53:16 | mriedem | 'every other cloud has spot instances but nova *doesn't* have it' | |
| 20:53:31 | edleafe | mriedem: what are you referring to as "alternatives"? | |
| 20:54:23 | jaypipes | so here's the thing... I could change the scheduler manager to get rid of all the "consume this stuff within the _schedule() filter scheduler loop" and call the placement API for allocation candidates once for each instance in num_instances. | |
| 20:54:44 | jaypipes | the problem with that is the "allocate 99 instances and the last one fails" problem that large Heat stacks have. | |
| 20:54:53 | edleafe | mriedem: I mean the set of N potential hosts we return with the selected host to the conductor. That's passed to the cell, and if the build fails, the cell conductor does retries with the alternatives | |
| 20:55:52 | mriedem | edleafe: yes, but what are you saying about duplicates? | |
| 20:55:59 | mriedem | wrt alternatives sent to the cell? | |
| 20:56:27 | mriedem | do you mean like 2 alternatives pointing at the same compute node for ram/cpu but different storage providers? (one local, one shared)? | |
| 20:56:59 | mriedem | jaypipes: also, if we call placement for allocation candidates for each instance, that would greatly slow things down wouldn't it? | |
| 20:57:24 | dansmith | that would suck for 100-instance batches, yeah | |
| 20:57:32 | mriedem | or is that a loop of (1) get allocatoin candidates, (2) filter/weigh (3) claim, (4) repeat | |
| 20:57:38 | jaypipes | mriedem: that scenario is actually *exactly* what I benchmarked here: https://github.com/jaypipes/placement-bench/ | |
| 20:57:57 | edleafe | mriedem: if we're returning 100 sets of (selected host, alt1, alt2, ...altN) to the conductor, the same host could appear in multiple alternate lists | |
| 20:58:02 | jaypipes | mriedem: and I was able to achieve >200 claimed requests *per second*. | |
| 20:58:32 | mriedem | jaypipes: what is a 'claimed request'? | |
| 20:58:51 | jaypipes | mriedem: request for resources, get list of providers, claim for an instance against a selected provider. | |
| 20:58:53 | mriedem | that's the full thing through select_destinations to placement to filter/weigh to claim? | |
| 20:58:56 | jaypipes | right. | |
| 20:59:43 | mriedem | edleafe: yes agree | |
| 20:59:47 | mriedem | edleafe: by default it's 3 | |
| 20:59:48 | mriedem | but yes | |
| 20:59:51 | jaypipes | mriedem: I'm think maybe it would be better to scrap trying to make this square peg fit the filter scheduler round hole. | |
| 21:00:40 | mriedem | what is the square peg in this analogy? | |
| 21:00:43 | mriedem | multiple create? | |
| 21:00:48 | dansmith | edleafe | |
| 21:00:51 | dansmith | oh did I say that? | |
| 21:00:51 | mriedem | ha | |
| 21:00:55 | jaypipes | mriedem: and let me move the _schedule() internals into the caching scheduler and make the filter scheduler call placement each iteration of the num_instances loop... | |
| 21:01:02 | mriedem | he's no square daddy-o | |
| 21:01:37 | dansmith | I dunno, I'm highly skeptical of the efficiency of doing one call per instance with 100 instances | |
| 21:01:49 | edleafe | dansmith: I sure feel like it | |
| 21:01:50 | jaypipes | dansmith: well, actually.. | |
| 21:02:08 | jaypipes | dansmith: I *am* returning the provider_summaries, which has the remaining capacity for various resources in it. | |
| 21:02:12 | dansmith | even if we *can* do it, that still might be a lot of network, a lot of CPU, etc, and if there's any latency between placement and scheduler... | |
| 21:02:25 | dansmith | jaypipes: right, see, what I would do is, | |
| 21:02:45 | dansmith | ask for some batch N, and consume from there until a claim fails, and then do another batch until you've processed num_instances | |
| 21:02:47 | jaypipes | deduct from the capacity on each loop... | |
| 21:03:05 | jaypipes | which is pretty much what the _schedule() method is doing right now for the host state objects. | |
| 21:03:30 | mriedem | that's "chosen_host.obj.consume_from_request(spec_obj)" right? | |
| 21:03:32 | dansmith | well, if you try to predict capacity, you might be missing some things that placement knows about, and definitely other schedulers are claiming | |
| 21:03:33 | dims | mriedem : LOL | |
| 21:03:48 | dansmith | so if you just claim until you get a failure and then ask for a new batch I think you're safer | |
| 21:03:58 | jaypipes | mriedem: yeah, kinda. | |
| 21:04:01 | dansmith | that could be every request if things are chaotic, but if not, then you've saved a bunch of work I think | |
| 21:04:43 | jaypipes | dansmith: that's my problem with doing that... we get the cache invalidation problem :( | |
| 21:05:14 | dansmith | right, so you claim until fail, then get another batch of candidates, IMHO | |
| 21:05:23 | dansmith | jaypipes: the other thing that sucks about --min, which is what I was debugging just now, | |
| 21:05:31 | dansmith | is that --min only works for quota, | |
| 21:05:42 | dansmith | and if your max is 100 and you can only fit 93, you fail the whole deal, | |
| 21:05:46 | dansmith | even if --min=1 | |
| 21:05:55 | dansmith | which is pretty stupid and not what the user expects | |
| 21:06:04 | dansmith | so it would be nice to fix that in the process of doing this, if we could | |
| 21:06:21 | jaypipes | this whole num_instances thing is stupid to begin with. | |
| 21:06:42 | dansmith | except for all the users that actively use it | |
| 21:06:47 | dansmith | https://bugs.launchpad.net/nova/+bug/1703457 | |
| 21:06:48 | openstack | Launchpad bug 1703457 in OpenStack Compute (nova) "RFE: More intuitive usage of --min and --max arguments in openstack server create" [Undecided,New] | |
| 21:06:53 | dansmith | not from an nfv workload ^ btw | |
| 21:07:14 | jaypipes | yeah, but it should be an orchestration layer that handles all the crazy. not the Nova scheduler. | |
| 21:07:34 | jaypipes | like I don't know.. Heat with convergence? :( | |
| 21:07:43 | dansmith | well, you can say that for sure. they're not using heat or anything else, and we have it in our api... | |
| 21:08:21 | dansmith | given the number of people that seem to want it, | |
| 21:08:42 | dansmith | I think we either keep supporting it, or we're going to be asked to do reservations for that orchestrator thing | |
| 21:10:08 | dansmith | at a minimum, we should either fix (or allow fixes) for the above kind of stupid thing, or we should cut the whole bit of it out of the api with a microversion | |
| 21:10:19 | melwitt | yeah, cfriesen opened a bug about failing a request where >= min_count instances were successful but it fails bc it couldn't satisfy max_count, two years ago https://bugs.launchpad.net/nova/+bug/1458122 | |
| 21:10:21 | openstack | Launchpad bug 1458122 in OpenStack Compute (nova) "nova shouldn't error if we can't schedule all of max_count instances at boot time" [Wishlist,Opinion] - Assigned to Chris Friesen (cbf123) | |
| 21:10:38 | mriedem | dansmith: even if you cut it out of the api with a microversion, there are all of the microversion before it where it lives | |
| 21:10:42 | dansmith | but for a cloud that supports an application and doesn't use quota at all, --min-instances is literally ineffectual | |
| 21:11:11 | dansmith | mriedem: well, I know, and I think we should just keep supporting it given the popularity, but if we're going to let it rot, we should signal that we're not going to fix things | |
| 21:11:19 | dansmith | melwitt: ah yep | |
| 21:12:43 | melwitt | I know they use the "request multiple instances" a lot at yahoo too | |
| 21:12:45 | dansmith | anyway, I gotta run let some creeper feel my eyeballs | |
| 21:12:47 | cdent | can somebody summarize/recapitulate the problem; i’ve been following along but it is coming in bits and pieces I’m not sure I get it. | |
| 21:12:53 | jaypipes | alright, I think I can do this with the provider_summaries thing. | |
| 21:14:01 | mriedem | cdent: we aren't accounting for allocations already claimed when processing a list of instances in the request | |
| 21:14:32 | mriedem | so we could essentially send alternatives to the cell that are already full | |
| 21:16:41 | melwitt | taking a step back, I think the usefulness of multi boot is with the legacy scheduler behavior that would race and pick full hosts because of claiming in compute. so with multi boot, you get the advantage of not having a lot of your requests fail out because of races | |
| 21:18:12 | cdent | melwitt: is that more important than “it’s just convenient to do it one step”? | |
| 21:18:17 | melwitt | aside from that, people do like the ease of saying "I want N instances" but I don't think that's the main reason people like it | |
| 21:18:27 | cdent | jinxish | |
| 21:18:29 | melwitt | (I could be wrong ... disclaimer) | |
| 21:18:47 | dansmith | melwitt: well, fwiw, I've never heard that use case for it | |
| 21:18:49 | dansmith | but maybe so | |
| 21:18:58 | melwitt | at yahoo I think that's the use case | |
| 21:19:17 | melwitt | when people do a for-loop of 100 instances, lots of them fail bc races | |
| 21:19:19 | dansmith | so you ask for 30 because you actually want 20? | |
| 21:20:27 | melwitt | hm, I don't think so. I haven't heard of that. they just want like 100 at a time and if they do it one-by-one a lot fail, if they do multi boot they get almost all | |
| 21:21:48 | melwitt | well, also, in VM clusters at yahoo retries are disabled. so that's why so many fail due to race | |
| 21:21:49 | openstackgerrit | Maciej Kucia proposed openstack/nova master: SR-IOV passthrough: Check PF only if VF is enabled https://review.openstack.org/476642 | |
| 21:22:52 | melwitt | (retries are enabled for baremetal clusters) | |
| 21:23:17 | mriedem | wouldn't you also do multiple create to get the instances in the same affinity group? | |
| 21:23:22 | mriedem | at once | |
| 21:23:36 | mriedem | like, i need 10 instances and need them all on the same host | |
| 21:24:04 | melwitt | you could. I think you could also pass the same group for each of 10 too right | |
| 21:24:32 | mriedem | yeah, but you might land 5 of 10 on that same host and then the 6th fails | |
| 21:24:44 | melwitt | yeah, true | |
| 21:24:47 | mriedem | so what do you do? delete them all and try again? | |
| 21:25:12 | mriedem | i believe that's why the soft affinity weigher was added | |
| 21:25:23 | mriedem | best effort to place the group on the same host, but not fail if you can't | |