| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-09-13 | |||
| 14:50:22 | mriedem | bauzas: here is the fugly cross_az_attach=False bug fix i mentioned yesterday https://review.openstack.org/#/c/469675/ | |
| 14:50:49 | mriedem | note that i plan on writing an alternate, less tightly coupled approach to fixing this laid out in https://review.openstack.org/#/c/469675/4/nova/compute/api.py@918 | |
| 14:53:07 | lyarwood | bauzas: mind taking a look at https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:bug/1787606 this morning if you get a chance | |
| 14:53:27 | bauzas | mriedem: ta, will look at both | |
| 14:54:37 | mriedem | edleafe: you might want to take a look at https://review.openstack.org/#/c/593074/ from lyarwood above | |
| 14:54:38 | openstackgerrit | Claudiu Belu proposed openstack/nova master: conductor: add live_resize task https://review.openstack.org/248579 | |
| 14:54:39 | openstackgerrit | Claudiu Belu proposed openstack/nova master: API: Adds instance live-resize https://review.openstack.org/184584 | |
| 14:54:42 | bauzas | lyarwood: yup, saw your change yesterday | |
| 14:57:34 | lyarwood | bauzas: well rebase, I'll fix the failing func tests if it's a valid fix | |
| 14:57:40 | bauzas | lyarwood: mriedem: could we just one day say we deprecate multiple instances creation ? :) | |
| 14:58:31 | edleafe | mriedem: One of the early discussions on alternate hosts was the concern about providing an alternate that may have already been consumed by another instance from the request. If that's no longer a concern, then that patch is fine. | |
| 14:59:00 | bauzas | edleafe: the problem is that we don't know yet if we will consume the host | |
| 14:59:09 | mriedem | edleafe: yeah that's what i mentioned to lya | |
| 14:59:20 | mriedem | *lyarwood | |
| 14:59:28 | bauzas | I mean, when it goes to pick a destination, we don't know yet if the allocation is possible | |
| 15:02:51 | bauzas | nevermind, we consumed the host before we get the alternates | |
| 15:03:19 | mriedem | i think the point is we don't want several reschedules trying to clobber the same alternate | |
| 15:03:25 | mriedem | maybe we should randomize the alternats? | |
| 15:03:28 | edleafe | bauzas: the selected_host has already claimed the resources at that point. Alternates only are provided if something else has prevented the selected host from bulding the instance. The feeling expressed when I wrote that is that we didn't want to include a host that already had some of its resources consumed | |
| 15:03:28 | mriedem | *alternates? | |
| 15:03:48 | melwitt | gibi: where did you send a mail about the minimum bandwidth scheduling demo location? I didn't see it | |
| 15:06:57 | gibi | melwitt: http://lists.openstack.org/pipermail/openstack-dev/2018-September/134622.html | |
| 15:07:58 | bauzas | mriedem: edleafe: yeah, the problem is that we don't optimize how we sort the alternates | |
| 15:08:20 | melwitt | gibi: oh, yeah, ok. the same one I read /facepalm | |
| 15:08:29 | melwitt | yesterday. I was thinking there was some other new one | |
| 15:08:40 | bauzas | mriedem: edleafe: meaning, instances could be optimized to be claimed spreadly, but then they could have the same host as alternate | |
| 15:09:01 | gibi | melwitt: if there was a new one yesterday then I didn't see that :) | |
| 15:09:08 | bauzas | so, yeah, we could just randomize the alternates, or find a better strategy | |
| 15:09:57 | edleafe | lyarwood: I'm curious as to the motivation for this. Have you seen a number of build failures on hosts that would require more alternates? | |
| 15:10:09 | bauzas | edleafe: see the bug report | |
| 15:10:20 | bauzas | in a small env, this could fail | |
| 15:10:23 | bauzas | 5 instances, 2 nodes | |
| 15:10:58 | bauzas | so, yeah, we're over rejecting | |
| 15:11:19 | edleafe | Ah, when you are requesting more instances in a single build than you have available hosts | |
| 15:11:23 | bauzas | we should less reject, but we should also shuffle a little bit to spread more | |
| 15:11:33 | bauzas | edleafe: that's correct | |
| 15:12:06 | edleafe | Patient: "Doctor, it hurts when I do this" Doctor: "Then stop doing that!" | |
| 15:12:12 | edleafe | :) | |
| 15:14:18 | mriedem | then i'd make the logic conditional on whether or not we have num_instances > filtered_hosts | |
| 15:16:46 | bauzas | mriedem: edleafe: well | |
| 15:18:27 | bauzas | mriedem: edleafe: the above is a real bug if and only if what was claimed is not able to be spawned, right? | |
| 15:18:57 | bauzas | because we claimed before we look for alternates, so from a pure capacity point of view, we should be good | |
| 15:19:21 | edleafe | bauzas: it depends on the reason for the failure. | |
| 15:19:47 | bauzas | edleafe: we trigger alternates because something else but a capacitiy issue, right ? | |
| 15:19:47 | edleafe | bauzas: if it was because of insufficient resources, then we have a BIG problem | |
| 15:20:03 | edleafe | bauzas: yes, that's the design | |
| 15:20:29 | edleafe | We've already claimed resources before a host is designated as the selected_host | |
| 15:22:23 | bauzas | edleafe: yup, I'm just stating this | |
| 15:22:46 | bauzas | edleafe: alternates are for derailed requests that shouldn't be capacity problems | |
| 15:22:57 | bauzas | so | |
| 15:23:59 | bauzas | back to the possible solutions, I think lyarwood's proposal is good with the slight detail that we could just shuffle a bit | |
| 15:24:21 | mriedem | i just want to make sure that if i'm spawning 5 instances with 4 hosts, is the first alternate for all instances going to be the same host? | |
| 15:24:52 | mriedem | or is the answer "it depends" | |
| 15:26:11 | bauzas | mriedem: there is a TODO on this | |
| 15:27:04 | bauzas | mriedem: edleafe: https://github.com/openstack/nova/blob/master/nova/scheduler/filter_scheduler.py#L411 | |
| 15:27:33 | bauzas | mriedem: that's the shuffling bit I'd like to have | |
| 15:27:50 | bauzas | we just get the first element and we consider it to be the alternate | |
| 15:28:07 | bauzas | that's highly probabilistic that we gonna end on the same alternate | |
| 15:28:34 | bauzas | so, just fix the TODO and do a bit of a guess instead of getting the first element of the list | |
| 15:29:03 | bauzas | edleafe: alternates are all seen as viable targets, rightN | |
| 15:29:06 | bauzas | right? | |
| 15:29:14 | bauzas | from a capacity pov | |
| 15:29:34 | bauzas | so, there is no harm to pick a random one and not just the first in the list | |
| 15:29:39 | bauzas | edleafe: you agree ? | |
| 15:30:07 | bauzas | lyarwood: I made some very technical comments on the functional test | |
| 15:30:20 | bauzas | lyarwood: if you need help for a respin, lemme know | |
| 15:31:01 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Fix 500 error while passing 4-byte unicode data https://review.openstack.org/407514 | |
| 15:31:16 | edleafe | bauzas: that's a different version of shuffling, for the nested case | |
| 15:31:38 | edleafe | IOW, if a single RP has multiple allocation_candidates, just pick the first for now | |
| 15:31:56 | edleafe | The TODO is to evaluate each for "best fit" | |
| 15:35:19 | lyarwood | bauzas: thanks, I'll get to it after this session | |
| 15:35:33 | edleafe | bauzas: I don't see a problem with a random selection, but in the two-node setup described in the bug, it won't help :) | |
| 15:36:03 | bauzas | edleafe: well, it could help or not | |
| 15:36:45 | bauzas | edleafe: but at least, we would ensure that if we have more than 1 host as alternate, we don't pick the same for all the instances :) | |
| 15:36:45 | edleafe | bauzas: in a two-node setup, there is only one possible alternate :) | |
| 15:36:53 | bauzas | edleafe: yeah, sure, no disagreement on it :D | |
| 15:37:48 | bauzas | actually, my guess is that the example we have in the bug report will fail anyway | |
| 15:37:48 | edleafe | bauzas: the other thing to remember is that the list of hosts have been weighed at this point | |
| 15:38:07 | edleafe | So a random choice will ignore all weighting | |
| 15:38:14 | bauzas | ah good point | |
| 15:38:24 | bauzas | mmmm | |
| 15:38:43 | bauzas | ok, let's keep it simple then | |
| 15:39:19 | bauzas | if alternates follow the packing/spreading strategy given by the weighers, they're already shuffled | |
| 15:39:34 | bauzas | so, what lyarwood did is sufficient | |
| 15:43:30 | openstackgerrit | Eric Fried proposed openstack/os-traits master: Add OWNERSHIP_$SERVICE traits https://review.openstack.org/602160 | |
| 15:47:45 | bauzas | edleafe: wait a sec | |
| 15:48:09 | bauzas | edleafe: is https://github.com/openstack/nova/blob/master/nova/scheduler/filter_scheduler.py#L417 a weighed list ? | |
| 15:49:20 | edleafe | ARs are not weighed. The hosts they represent are | |
| 15:49:39 | bauzas | that's old greek to me :) | |
| 15:50:47 | bauzas | oh shit | |
| 15:51:05 | bauzas | I think I misunderstood the logic | |
| 15:51:28 | bauzas | we got selected hosts, we iterate over those and see whether they have an AR against each | |
| 15:51:36 | bauzas | gotcha | |
| 15:52:26 | bauzas | so, yeah, selected hosts is weighed | |
| 15:52:50 | edleafe | bauzas: this is where the alternates are created: https://github.com/openstack/nova/blob/master/nova/scheduler/filter_scheduler.py#L260, and the 'hosts' variable is defined above: https://github.com/openstack/nova/blob/master/nova/scheduler/filter_scheduler.py#L260 | |
| 15:54:24 | bauzas | yeah I got it | |
| 15:54:33 | edleafe | bauzas: oops: the second link should be https://github.com/openstack/nova/blob/master/nova/scheduler/filter_scheduler.py#L201 | |
| 15:54:49 | bauzas | https://github.com/openstack/nova/blob/master/nova/scheduler/filter_scheduler.py#L397 is the loop over hosts that are weighed | |
| 15:55:13 | bauzas | so, yeah, we basically either spread or pack that list | |
| 15:55:17 | bauzas | so, we're fine | |