| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-07-10 | |||
| 20:59:48 | mriedem | but yes | |
| 20:59:51 | jaypipes | mriedem: I'm think maybe it would be better to scrap trying to make this square peg fit the filter scheduler round hole. | |
| 21:00:40 | mriedem | what is the square peg in this analogy? | |
| 21:00:43 | mriedem | multiple create? | |
| 21:00:48 | dansmith | edleafe | |
| 21:00:51 | mriedem | ha | |
| 21:00:51 | dansmith | oh did I say that? | |
| 21:00:55 | jaypipes | mriedem: and let me move the _schedule() internals into the caching scheduler and make the filter scheduler call placement each iteration of the num_instances loop... | |
| 21:01:02 | mriedem | he's no square daddy-o | |
| 21:01:37 | dansmith | I dunno, I'm highly skeptical of the efficiency of doing one call per instance with 100 instances | |
| 21:01:49 | edleafe | dansmith: I sure feel like it | |
| 21:01:50 | jaypipes | dansmith: well, actually.. | |
| 21:02:08 | jaypipes | dansmith: I *am* returning the provider_summaries, which has the remaining capacity for various resources in it. | |
| 21:02:12 | dansmith | even if we *can* do it, that still might be a lot of network, a lot of CPU, etc, and if there's any latency between placement and scheduler... | |
| 21:02:25 | dansmith | jaypipes: right, see, what I would do is, | |
| 21:02:45 | dansmith | ask for some batch N, and consume from there until a claim fails, and then do another batch until you've processed num_instances | |
| 21:02:47 | jaypipes | deduct from the capacity on each loop... | |
| 21:03:05 | jaypipes | which is pretty much what the _schedule() method is doing right now for the host state objects. | |
| 21:03:30 | mriedem | that's "chosen_host.obj.consume_from_request(spec_obj)" right? | |
| 21:03:32 | dansmith | well, if you try to predict capacity, you might be missing some things that placement knows about, and definitely other schedulers are claiming | |
| 21:03:33 | dims | mriedem : LOL | |
| 21:03:48 | dansmith | so if you just claim until you get a failure and then ask for a new batch I think you're safer | |
| 21:03:58 | jaypipes | mriedem: yeah, kinda. | |
| 21:04:01 | dansmith | that could be every request if things are chaotic, but if not, then you've saved a bunch of work I think | |
| 21:04:43 | jaypipes | dansmith: that's my problem with doing that... we get the cache invalidation problem :( | |
| 21:05:14 | dansmith | right, so you claim until fail, then get another batch of candidates, IMHO | |
| 21:05:23 | dansmith | jaypipes: the other thing that sucks about --min, which is what I was debugging just now, | |
| 21:05:31 | dansmith | is that --min only works for quota, | |
| 21:05:42 | dansmith | and if your max is 100 and you can only fit 93, you fail the whole deal, | |
| 21:05:46 | dansmith | even if --min=1 | |
| 21:05:55 | dansmith | which is pretty stupid and not what the user expects | |
| 21:06:04 | dansmith | so it would be nice to fix that in the process of doing this, if we could | |
| 21:06:21 | jaypipes | this whole num_instances thing is stupid to begin with. | |
| 21:06:42 | dansmith | except for all the users that actively use it | |
| 21:06:47 | dansmith | https://bugs.launchpad.net/nova/+bug/1703457 | |
| 21:06:48 | openstack | Launchpad bug 1703457 in OpenStack Compute (nova) "RFE: More intuitive usage of --min and --max arguments in openstack server create" [Undecided,New] | |
| 21:06:53 | dansmith | not from an nfv workload ^ btw | |
| 21:07:14 | jaypipes | yeah, but it should be an orchestration layer that handles all the crazy. not the Nova scheduler. | |
| 21:07:34 | jaypipes | like I don't know.. Heat with convergence? :( | |
| 21:07:43 | dansmith | well, you can say that for sure. they're not using heat or anything else, and we have it in our api... | |
| 21:08:21 | dansmith | given the number of people that seem to want it, | |
| 21:08:42 | dansmith | I think we either keep supporting it, or we're going to be asked to do reservations for that orchestrator thing | |
| 21:10:08 | dansmith | at a minimum, we should either fix (or allow fixes) for the above kind of stupid thing, or we should cut the whole bit of it out of the api with a microversion | |
| 21:10:19 | melwitt | yeah, cfriesen opened a bug about failing a request where >= min_count instances were successful but it fails bc it couldn't satisfy max_count, two years ago https://bugs.launchpad.net/nova/+bug/1458122 | |
| 21:10:21 | openstack | Launchpad bug 1458122 in OpenStack Compute (nova) "nova shouldn't error if we can't schedule all of max_count instances at boot time" [Wishlist,Opinion] - Assigned to Chris Friesen (cbf123) | |
| 21:10:38 | mriedem | dansmith: even if you cut it out of the api with a microversion, there are all of the microversion before it where it lives | |
| 21:10:42 | dansmith | but for a cloud that supports an application and doesn't use quota at all, --min-instances is literally ineffectual | |
| 21:11:11 | dansmith | mriedem: well, I know, and I think we should just keep supporting it given the popularity, but if we're going to let it rot, we should signal that we're not going to fix things | |
| 21:11:19 | dansmith | melwitt: ah yep | |
| 21:12:43 | melwitt | I know they use the "request multiple instances" a lot at yahoo too | |
| 21:12:45 | dansmith | anyway, I gotta run let some creeper feel my eyeballs | |
| 21:12:47 | cdent | can somebody summarize/recapitulate the problem; i’ve been following along but it is coming in bits and pieces I’m not sure I get it. | |
| 21:12:53 | jaypipes | alright, I think I can do this with the provider_summaries thing. | |
| 21:14:01 | mriedem | cdent: we aren't accounting for allocations already claimed when processing a list of instances in the request | |
| 21:14:32 | mriedem | so we could essentially send alternatives to the cell that are already full | |
| 21:16:41 | melwitt | taking a step back, I think the usefulness of multi boot is with the legacy scheduler behavior that would race and pick full hosts because of claiming in compute. so with multi boot, you get the advantage of not having a lot of your requests fail out because of races | |
| 21:18:12 | cdent | melwitt: is that more important than “it’s just convenient to do it one step”? | |
| 21:18:17 | melwitt | aside from that, people do like the ease of saying "I want N instances" but I don't think that's the main reason people like it | |
| 21:18:27 | cdent | jinxish | |
| 21:18:29 | melwitt | (I could be wrong ... disclaimer) | |
| 21:18:47 | dansmith | melwitt: well, fwiw, I've never heard that use case for it | |
| 21:18:49 | dansmith | but maybe so | |
| 21:18:58 | melwitt | at yahoo I think that's the use case | |
| 21:19:17 | melwitt | when people do a for-loop of 100 instances, lots of them fail bc races | |
| 21:19:19 | dansmith | so you ask for 30 because you actually want 20? | |
| 21:20:27 | melwitt | hm, I don't think so. I haven't heard of that. they just want like 100 at a time and if they do it one-by-one a lot fail, if they do multi boot they get almost all | |
| 21:21:48 | melwitt | well, also, in VM clusters at yahoo retries are disabled. so that's why so many fail due to race | |
| 21:21:49 | openstackgerrit | Maciej Kucia proposed openstack/nova master: SR-IOV passthrough: Check PF only if VF is enabled https://review.openstack.org/476642 | |
| 21:22:52 | melwitt | (retries are enabled for baremetal clusters) | |
| 21:23:17 | mriedem | wouldn't you also do multiple create to get the instances in the same affinity group? | |
| 21:23:22 | mriedem | at once | |
| 21:23:36 | mriedem | like, i need 10 instances and need them all on the same host | |
| 21:24:04 | melwitt | you could. I think you could also pass the same group for each of 10 too right | |
| 21:24:32 | mriedem | yeah, but you might land 5 of 10 on that same host and then the 6th fails | |
| 21:24:44 | melwitt | yeah, true | |
| 21:24:47 | mriedem | so what do you do? delete them all and try again? | |
| 21:25:12 | mriedem | i believe that's why the soft affinity weigher was added | |
| 21:25:23 | mriedem | best effort to place the group on the same host, but not fail if you can't | |
| 21:26:00 | melwitt | I see | |
| 21:38:07 | mdrabe | Any cores willing to look at https://review.openstack.org/#/c/467774/5? Small bug fix that's been chillin | |
| 21:44:15 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Fix quota class set APIs https://review.openstack.org/467999 | |
| 21:47:18 | mriedem | mdrabe: done | |
| 21:48:05 | mdrabe | thnx | |
| 21:56:20 | mriedem | overall i don't know how great of an idea this is in general, or what side effects we might have, would like to see dansmith look at it but he's out for the rest of the day | |
| 22:01:28 | melwitt | mriedem: I wondered similar, like how/why aren't those resources cleaned up by local delete + init_host. but I'm not well-versed in what's special about evacuation resources | |
| 22:17:14 | openstackgerrit | Chris Dent proposed openstack/nova-specs master: Provide detailed error information for placement API https://review.openstack.org/418393 | |
| 22:21:42 | mdrabe | mriedem melwitt: It'll make init_host a little slower, but I don't know where else the source host instance deletion should be handled | |
| 22:32:48 | cdent | goodnight all | |
| 22:34:14 | melwitt | mdrabe: I think init_host is the right place for doing that. I had thought they'd be deleted by the "_cleanup_running_deleted_instances" periodic task. dansmith knows a lot about evacuate, so hopefully tomorrow he can look at the patch and let us know if there are any gotchas there | |
| 22:39:38 | mriedem | melwitt: _cleanup_running_deleted_instances gets the instances on that host, | |
| 22:39:48 | mriedem | so that won't work if the instance was evacuated and it's instance.host field was changed | |
| 22:40:09 | melwitt | ah, okay. that's what I was missing | |
| 22:40:13 | mriedem | that's why mdrabe's change is doing it via the migrations | |
| 22:51:43 | openstackgerrit | Matt Rabe proposed openstack/nova master: Query deleted instance records during _destroy_evacuated_instances https://review.openstack.org/467774 | |
| 23:18:27 | rm_work | I've noticed today that in a new devstack with cirros-0.3.5-x86_64-disk (and supposedly 0.3.4 too according to someone else who is seeing the same thing) the ssh keypair I'm configuring isn't ending up on the VM, there's no ~cirros/.ssh at all, so I have to log in with the default password | |
| 23:18:45 | rm_work | anyone else able to replicate that? | |
| 23:20:17 | openstackgerrit | melanie witt proposed openstack/nova master: Convert websocketproxy to use db for token validation https://review.openstack.org/333990 | |
| 23:34:50 | clarkb | rm_work: I'm pretty sure the gate tests ssh connectivity using ssh keys | |
| 23:35:04 | clarkb | rm_work: but maybe different versions of cirros? | |
| 23:35:18 | rm_work | hmmm | |