Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-07-10
21:00:43 mriedem multiple create?
21:00:48 dansmith edleafe
21:00:51 mriedem ha
21:00:51 dansmith oh did I say that?
21:00:55 jaypipes mriedem: and let me move the _schedule() internals into the caching scheduler and make the filter scheduler call placement each iteration of the num_instances loop...
21:01:02 mriedem he's no square daddy-o
21:01:37 dansmith I dunno, I'm highly skeptical of the efficiency of doing one call per instance with 100 instances
21:01:49 edleafe dansmith: I sure feel like it
21:01:50 jaypipes dansmith: well, actually..
21:02:08 jaypipes dansmith: I *am* returning the provider_summaries, which has the remaining capacity for various resources in it.
21:02:12 dansmith even if we *can* do it, that still might be a lot of network, a lot of CPU, etc, and if there's any latency between placement and scheduler...
21:02:25 dansmith jaypipes: right, see, what I would do is,
21:02:45 dansmith ask for some batch N, and consume from there until a claim fails, and then do another batch until you've processed num_instances
21:02:47 jaypipes deduct from the capacity on each loop...
21:03:05 jaypipes which is pretty much what the _schedule() method is doing right now for the host state objects.
21:03:30 mriedem that's "chosen_host.obj.consume_from_request(spec_obj)" right?
21:03:32 dansmith well, if you try to predict capacity, you might be missing some things that placement knows about, and definitely other schedulers are claiming
21:03:33 dims mriedem : LOL
21:03:48 dansmith so if you just claim until you get a failure and then ask for a new batch I think you're safer
21:03:58 jaypipes mriedem: yeah, kinda.
21:04:01 dansmith that could be every request if things are chaotic, but if not, then you've saved a bunch of work I think
21:04:43 jaypipes dansmith: that's my problem with doing that... we get the cache invalidation problem :(
21:05:14 dansmith right, so you claim until fail, then get another batch of candidates, IMHO
21:05:23 dansmith jaypipes: the other thing that sucks about --min, which is what I was debugging just now,
21:05:31 dansmith is that --min only works for quota,
21:05:42 dansmith and if your max is 100 and you can only fit 93, you fail the whole deal,
21:05:46 dansmith even if --min=1
21:05:55 dansmith which is pretty stupid and not what the user expects
21:06:04 dansmith so it would be nice to fix that in the process of doing this, if we could
21:06:21 jaypipes this whole num_instances thing is stupid to begin with.
21:06:42 dansmith except for all the users that actively use it
21:06:47 dansmith https://bugs.launchpad.net/nova/+bug/1703457
21:06:48 openstack Launchpad bug 1703457 in OpenStack Compute (nova) "RFE: More intuitive usage of --min and --max arguments in openstack server create" [Undecided,New]
21:06:53 dansmith not from an nfv workload ^ btw
21:07:14 jaypipes yeah, but it should be an orchestration layer that handles all the crazy. not the Nova scheduler.
21:07:34 jaypipes like I don't know.. Heat with convergence? :(
21:07:43 dansmith well, you can say that for sure. they're not using heat or anything else, and we have it in our api...
21:08:21 dansmith given the number of people that seem to want it,
21:08:42 dansmith I think we either keep supporting it, or we're going to be asked to do reservations for that orchestrator thing
21:10:08 dansmith at a minimum, we should either fix (or allow fixes) for the above kind of stupid thing, or we should cut the whole bit of it out of the api with a microversion
21:10:19 melwitt yeah, cfriesen opened a bug about failing a request where >= min_count instances were successful but it fails bc it couldn't satisfy max_count, two years ago https://bugs.launchpad.net/nova/+bug/1458122
21:10:21 openstack Launchpad bug 1458122 in OpenStack Compute (nova) "nova shouldn't error if we can't schedule all of max_count instances at boot time" [Wishlist,Opinion] - Assigned to Chris Friesen (cbf123)
21:10:38 mriedem dansmith: even if you cut it out of the api with a microversion, there are all of the microversion before it where it lives
21:10:42 dansmith but for a cloud that supports an application and doesn't use quota at all, --min-instances is literally ineffectual
21:11:11 dansmith mriedem: well, I know, and I think we should just keep supporting it given the popularity, but if we're going to let it rot, we should signal that we're not going to fix things
21:11:19 dansmith melwitt: ah yep
21:12:43 melwitt I know they use the "request multiple instances" a lot at yahoo too
21:12:45 dansmith anyway, I gotta run let some creeper feel my eyeballs
21:12:47 cdent can somebody summarize/recapitulate the problem; i’ve been following along but it is coming in bits and pieces I’m not sure I get it.
21:12:53 jaypipes alright, I think I can do this with the provider_summaries thing.
21:14:01 mriedem cdent: we aren't accounting for allocations already claimed when processing a list of instances in the request
21:14:32 mriedem so we could essentially send alternatives to the cell that are already full
21:16:41 melwitt taking a step back, I think the usefulness of multi boot is with the legacy scheduler behavior that would race and pick full hosts because of claiming in compute. so with multi boot, you get the advantage of not having a lot of your requests fail out because of races
21:18:12 cdent melwitt: is that more important than “it’s just convenient to do it one step”?
21:18:17 melwitt aside from that, people do like the ease of saying "I want N instances" but I don't think that's the main reason people like it
21:18:27 cdent jinxish
21:18:29 melwitt (I could be wrong ... disclaimer)
21:18:47 dansmith melwitt: well, fwiw, I've never heard that use case for it
21:18:49 dansmith but maybe so
21:18:58 melwitt at yahoo I think that's the use case
21:19:17 melwitt when people do a for-loop of 100 instances, lots of them fail bc races
21:19:19 dansmith so you ask for 30 because you actually want 20?
21:20:27 melwitt hm, I don't think so. I haven't heard of that. they just want like 100 at a time and if they do it one-by-one a lot fail, if they do multi boot they get almost all
21:21:48 melwitt well, also, in VM clusters at yahoo retries are disabled. so that's why so many fail due to race
21:21:49 openstackgerrit Maciej Kucia proposed openstack/nova master: SR-IOV passthrough: Check PF only if VF is enabled https://review.openstack.org/476642
21:22:52 melwitt (retries are enabled for baremetal clusters)
21:23:17 mriedem wouldn't you also do multiple create to get the instances in the same affinity group?
21:23:22 mriedem at once
21:23:36 mriedem like, i need 10 instances and need them all on the same host
21:24:04 melwitt you could. I think you could also pass the same group for each of 10 too right
21:24:32 mriedem yeah, but you might land 5 of 10 on that same host and then the 6th fails
21:24:44 melwitt yeah, true
21:24:47 mriedem so what do you do? delete them all and try again?
21:25:12 mriedem i believe that's why the soft affinity weigher was added
21:25:23 mriedem best effort to place the group on the same host, but not fail if you can't
21:26:00 melwitt I see
21:38:07 mdrabe Any cores willing to look at https://review.openstack.org/#/c/467774/5? Small bug fix that's been chillin
21:44:15 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Fix quota class set APIs https://review.openstack.org/467999
21:47:18 mriedem mdrabe: done
21:48:05 mdrabe thnx
21:56:20 mriedem overall i don't know how great of an idea this is in general, or what side effects we might have, would like to see dansmith look at it but he's out for the rest of the day
22:01:28 melwitt mriedem: I wondered similar, like how/why aren't those resources cleaned up by local delete + init_host. but I'm not well-versed in what's special about evacuation resources
22:17:14 openstackgerrit Chris Dent proposed openstack/nova-specs master: Provide detailed error information for placement API https://review.openstack.org/418393
22:21:42 mdrabe mriedem melwitt: It'll make init_host a little slower, but I don't know where else the source host instance deletion should be handled
22:32:48 cdent goodnight all
22:34:14 melwitt mdrabe: I think init_host is the right place for doing that. I had thought they'd be deleted by the "_cleanup_running_deleted_instances" periodic task. dansmith knows a lot about evacuate, so hopefully tomorrow he can look at the patch and let us know if there are any gotchas there
22:39:38 mriedem melwitt: _cleanup_running_deleted_instances gets the instances on that host,
22:39:48 mriedem so that won't work if the instance was evacuated and it's instance.host field was changed
22:40:09 melwitt ah, okay. that's what I was missing
22:40:13 mriedem that's why mdrabe's change is doing it via the migrations
22:51:43 openstackgerrit Matt Rabe proposed openstack/nova master: Query deleted instance records during _destroy_evacuated_instances https://review.openstack.org/467774
23:18:27 rm_work I've noticed today that in a new devstack with cirros-0.3.5-x86_64-disk (and supposedly 0.3.4 too according to someone else who is seeing the same thing) the ssh keypair I'm configuring isn't ending up on the VM, there's no ~cirros/.ssh at all, so I have to log in with the default password
23:18:45 rm_work anyone else able to replicate that?
23:20:17 openstackgerrit melanie witt proposed openstack/nova master: Convert websocketproxy to use db for token validation https://review.openstack.org/333990
23:34:50 clarkb rm_work: I'm pretty sure the gate tests ssh connectivity using ssh keys
23:35:04 clarkb rm_work: but maybe different versions of cirros?
23:35:18 rm_work hmmm
23:35:19 rm_work so, something about our config then?
23:35:28 rm_work do you know what version the gate uses?
23:35:38 clarkb maybe? is metadata service enabled? and if not are you bootign with config drive?

Earlier   Later