| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-09-13 | |||
| 15:08:29 | melwitt | yesterday. I was thinking there was some other new one | |
| 15:08:40 | bauzas | mriedem: edleafe: meaning, instances could be optimized to be claimed spreadly, but then they could have the same host as alternate | |
| 15:09:01 | gibi | melwitt: if there was a new one yesterday then I didn't see that :) | |
| 15:09:08 | bauzas | so, yeah, we could just randomize the alternates, or find a better strategy | |
| 15:09:57 | edleafe | lyarwood: I'm curious as to the motivation for this. Have you seen a number of build failures on hosts that would require more alternates? | |
| 15:10:09 | bauzas | edleafe: see the bug report | |
| 15:10:20 | bauzas | in a small env, this could fail | |
| 15:10:23 | bauzas | 5 instances, 2 nodes | |
| 15:10:58 | bauzas | so, yeah, we're over rejecting | |
| 15:11:19 | edleafe | Ah, when you are requesting more instances in a single build than you have available hosts | |
| 15:11:23 | bauzas | we should less reject, but we should also shuffle a little bit to spread more | |
| 15:11:33 | bauzas | edleafe: that's correct | |
| 15:12:06 | edleafe | Patient: "Doctor, it hurts when I do this" Doctor: "Then stop doing that!" | |
| 15:12:12 | edleafe | :) | |
| 15:14:18 | mriedem | then i'd make the logic conditional on whether or not we have num_instances > filtered_hosts | |
| 15:16:46 | bauzas | mriedem: edleafe: well | |
| 15:18:27 | bauzas | mriedem: edleafe: the above is a real bug if and only if what was claimed is not able to be spawned, right? | |
| 15:18:57 | bauzas | because we claimed before we look for alternates, so from a pure capacity point of view, we should be good | |
| 15:19:21 | edleafe | bauzas: it depends on the reason for the failure. | |
| 15:19:47 | edleafe | bauzas: if it was because of insufficient resources, then we have a BIG problem | |
| 15:19:47 | bauzas | edleafe: we trigger alternates because something else but a capacitiy issue, right ? | |
| 15:20:03 | edleafe | bauzas: yes, that's the design | |
| 15:20:29 | edleafe | We've already claimed resources before a host is designated as the selected_host | |
| 15:22:23 | bauzas | edleafe: yup, I'm just stating this | |
| 15:22:46 | bauzas | edleafe: alternates are for derailed requests that shouldn't be capacity problems | |
| 15:22:57 | bauzas | so | |
| 15:23:59 | bauzas | back to the possible solutions, I think lyarwood's proposal is good with the slight detail that we could just shuffle a bit | |
| 15:24:21 | mriedem | i just want to make sure that if i'm spawning 5 instances with 4 hosts, is the first alternate for all instances going to be the same host? | |
| 15:24:52 | mriedem | or is the answer "it depends" | |
| 15:26:11 | bauzas | mriedem: there is a TODO on this | |
| 15:27:04 | bauzas | mriedem: edleafe: https://github.com/openstack/nova/blob/master/nova/scheduler/filter_scheduler.py#L411 | |
| 15:27:33 | bauzas | mriedem: that's the shuffling bit I'd like to have | |
| 15:27:50 | bauzas | we just get the first element and we consider it to be the alternate | |
| 15:28:07 | bauzas | that's highly probabilistic that we gonna end on the same alternate | |
| 15:28:34 | bauzas | so, just fix the TODO and do a bit of a guess instead of getting the first element of the list | |
| 15:29:03 | bauzas | edleafe: alternates are all seen as viable targets, rightN | |
| 15:29:06 | bauzas | right? | |
| 15:29:14 | bauzas | from a capacity pov | |
| 15:29:34 | bauzas | so, there is no harm to pick a random one and not just the first in the list | |
| 15:29:39 | bauzas | edleafe: you agree ? | |
| 15:30:07 | bauzas | lyarwood: I made some very technical comments on the functional test | |
| 15:30:20 | bauzas | lyarwood: if you need help for a respin, lemme know | |
| 15:31:01 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Fix 500 error while passing 4-byte unicode data https://review.openstack.org/407514 | |
| 15:31:16 | edleafe | bauzas: that's a different version of shuffling, for the nested case | |
| 15:31:38 | edleafe | IOW, if a single RP has multiple allocation_candidates, just pick the first for now | |
| 15:31:56 | edleafe | The TODO is to evaluate each for "best fit" | |
| 15:35:19 | lyarwood | bauzas: thanks, I'll get to it after this session | |
| 15:35:33 | edleafe | bauzas: I don't see a problem with a random selection, but in the two-node setup described in the bug, it won't help :) | |
| 15:36:03 | bauzas | edleafe: well, it could help or not | |
| 15:36:45 | edleafe | bauzas: in a two-node setup, there is only one possible alternate :) | |
| 15:36:45 | bauzas | edleafe: but at least, we would ensure that if we have more than 1 host as alternate, we don't pick the same for all the instances :) | |
| 15:36:53 | bauzas | edleafe: yeah, sure, no disagreement on it :D | |
| 15:37:48 | edleafe | bauzas: the other thing to remember is that the list of hosts have been weighed at this point | |
| 15:37:48 | bauzas | actually, my guess is that the example we have in the bug report will fail anyway | |
| 15:38:07 | edleafe | So a random choice will ignore all weighting | |
| 15:38:14 | bauzas | ah good point | |
| 15:38:24 | bauzas | mmmm | |
| 15:38:43 | bauzas | ok, let's keep it simple then | |
| 15:39:19 | bauzas | if alternates follow the packing/spreading strategy given by the weighers, they're already shuffled | |
| 15:39:34 | bauzas | so, what lyarwood did is sufficient | |
| 15:43:30 | openstackgerrit | Eric Fried proposed openstack/os-traits master: Add OWNERSHIP_$SERVICE traits https://review.openstack.org/602160 | |
| 15:47:45 | bauzas | edleafe: wait a sec | |
| 15:48:09 | bauzas | edleafe: is https://github.com/openstack/nova/blob/master/nova/scheduler/filter_scheduler.py#L417 a weighed list ? | |
| 15:49:20 | edleafe | ARs are not weighed. The hosts they represent are | |
| 15:49:39 | bauzas | that's old greek to me :) | |
| 15:50:47 | bauzas | oh shit | |
| 15:51:05 | bauzas | I think I misunderstood the logic | |
| 15:51:28 | bauzas | we got selected hosts, we iterate over those and see whether they have an AR against each | |
| 15:51:36 | bauzas | gotcha | |
| 15:52:26 | bauzas | so, yeah, selected hosts is weighed | |
| 15:52:50 | edleafe | bauzas: this is where the alternates are created: https://github.com/openstack/nova/blob/master/nova/scheduler/filter_scheduler.py#L260, and the 'hosts' variable is defined above: https://github.com/openstack/nova/blob/master/nova/scheduler/filter_scheduler.py#L260 | |
| 15:54:24 | bauzas | yeah I got it | |
| 15:54:33 | edleafe | bauzas: oops: the second link should be https://github.com/openstack/nova/blob/master/nova/scheduler/filter_scheduler.py#L201 | |
| 15:54:49 | bauzas | https://github.com/openstack/nova/blob/master/nova/scheduler/filter_scheduler.py#L397 is the loop over hosts that are weighed | |
| 15:55:13 | bauzas | so, yeah, we basically either spread or pack that list | |
| 15:55:17 | bauzas | so, we're fine | |
| 15:55:21 | edleafe | yup | |
| 15:55:22 | bauzas | no need to shuffle any bit | |
| 15:55:41 | bauzas | if the operator chose the packing strategy, that's their decision | |
| 16:01:43 | openstackgerrit | sean mooney proposed openstack/os-vif master: add support for generic tap device plug https://review.openstack.org/602384 | |
| 16:08:58 | bauzas | edleafe: you know what ? I'm horribly confused | |
| 16:11:20 | openstackgerrit | Eric Fried proposed openstack/os-traits master: Add OWNERSHIP_$SERVICE traits https://review.openstack.org/602160 | |
| 16:11:52 | bauzas | edleafe: how many times are we calling _get_alternate_hosts() if we have multiple instances ? one AFAICS | |
| 16:13:34 | mriedem | mdbooth: i'm not sure why but my evacuate script is thinking that we only have 1 hypervisor in this job but we should have 2 https://review.openstack.org/#/c/602174/ - i'm guessing it's a bug in my script doing the word count | |
| 16:14:34 | mriedem | oh i see why | |
| 16:14:43 | mriedem | well i think | |
| 16:15:35 | mriedem | 1 | |
| 16:15:35 | mriedem | $ echo "Missing value auth-url required for auth plugin password" | wc -l | |
| 16:29:31 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: consumer gen: more tests for delete allocation cases https://review.openstack.org/591811 | |
| 16:29:32 | edleafe | bauzas: We call it once for each requested instance | |
| 16:50:43 | openstackgerrit | Merged openstack/nova master: Fix mock.patch usage in unit tests https://review.openstack.org/602198 | |
| 16:55:34 | bauzas | edleafe: I tried to summarize my thoughts in one single comment https://review.openstack.org/#/c/593074/2/nova/scheduler/filter_scheduler.py@397 | |
| 17:17:26 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Update RequestSpec.flavor on resize_revert https://review.openstack.org/602403 | |
| 17:31:03 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Noop CantStartEngineError in targets_cell if API DB not configured https://review.openstack.org/581912 | |
| 17:32:05 | melwitt | efried: can you add some notes about the whitelist plan on the etherpad? I wasn't sure which line we're talking | |
| 17:33:48 | efried | melwitt: roger that. It's here: https://review.openstack.org/#/c/591037/ Where do you want it on which etherpad? :) | |
| 17:34:27 | melwitt | efried: oh, if it's already known, that's cool | |
| 17:34:41 | efried | melwitt: Agree it would be useful to have on one or both etherpads | |
| 17:35:19 | melwitt | efried: yeah, I wanted there to be a reference for cyborg team to see how that part will be handled. I don't see a relevant section in the etherpad https://etherpad.openstack.org/p/stein-ptg.cyborg-nova-new so far | |
| 17:35:37 | efried | oh, yeah, that spec is in a topic of its own later on.... | |