| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-04-12 | |||
| 15:10:29 | bauzas | efried: so we could chime in that about the possible implementation options we have | |
| 15:11:40 | efried | bauzas: Yes. I see us at some point needing to do filtering of GET /allocation_candidates results. This would be a good use case for setting that up. | |
| 15:11:59 | efried | bauzas: By the way, I assume a weigher has the same issue. It just weighs hosts, not allocation requests. | |
| 15:12:12 | efried | bauzas: I think that needs to be re-imagined in the same way. | |
| 15:13:19 | sahid | gameon: if you set a specific model you have to configure cpu_mode=custom, that is what you did? | |
| 15:14:01 | gameon | sahid: Yeah I did do that, no dice :( | |
| 15:14:28 | sahid | gameon: that with master? | |
| 15:14:34 | openstackgerrit | Jay Pipes proposed openstack/nova master: mirror nova host aggregate members to placement https://review.openstack.org/553597 | |
| 15:14:45 | gameon | sahid: only on hypervisors, not the scheduler or api node | |
| 15:15:06 | sahid | gameon: i mean you are using master branch of nova? | |
| 15:15:27 | gameon | Pike | |
| 15:16:38 | bauzas | efried: I don't like the wording "reimagined" for filters and weighers | |
| 15:17:04 | bauzas | efried: given it's one of the most custom pieces we have in Nova, with a ton of operators having their own filters, changing that isn't trivial | |
| 15:17:20 | bauzas | efried: but adding some extra field to the RequestSpec object seems fine with me | |
| 15:17:32 | efried | bauzas: yes, fair enough; it may have to be a new/different kind of filter. | |
| 15:17:35 | bauzas | so that filters can opt-in and check if necessary | |
| 15:17:46 | efried | bauzas: Because filtering at the host level is simply not going to be enough long-term. | |
| 15:18:15 | efried | bauzas: It doesn't have to be customizable or opt-in for the use case we're talking about. | |
| 15:18:36 | efried | bauzas: Other than in the sense they're customizing/opting-in based on what they put in their flavor | |
| 15:18:45 | jaypipes | bauzas: Besides "check my switch to see if it's on before scheduling to this node", what are the custom filters/weighers you have seen from operators? | |
| 15:18:48 | bauzas | efried: the problem is really because of backwards compatibility you know | |
| 15:19:13 | bauzas | efried: for the VGPU usecase, I don't care about that because scheduler will pick one allocation for me | |
| 15:19:31 | efried | bauzas: I contend that this is the same. | |
| 15:19:32 | bauzas | I don't have to make sure to mimic any pre-existing logic | |
| 15:19:41 | melwitt | dansmith, mriedem: ah, dammit. I messed up thinking the cert one was supposed to end on 2018-04-11. sorry | |
| 15:19:46 | efried | bauzas: The scheduler has to pick one allocation request based on the numa topology in the flavor. | |
| 15:21:28 | gameon | sahid: OK - setting them to kvm64 has seemed to have a different result. Maybe I didn't restart the service. Now I have 'Live Migration failure: unsupported configuration: Unable to find security driver for model apparmor: libvirtError: unsupported configuration: Unable to find security driver for model apparmor' | |
| 15:22:33 | bauzas | efried: for VGPUs ? | |
| 15:22:42 | efried | bauzas: For NUMA | |
| 15:22:47 | sahid | gameon: ok in same time i was trying to find a patch which address an issue with compare CPU but it seems that it's already on Pike https://review.openstack.org/#/c/53746/ | |
| 15:23:20 | bauzas | efried: I'm confused | |
| 15:23:27 | efried | bauzas: So maybe "filter" is the wrong word for it. The scheduler is going to need to pick from among the allocation requests returned by GET /a_c. It's gotta employ *some* kind of logic to do that. | |
| 15:23:39 | sahid | gameon: I can't help you for the issue with apparmar perhaps you could try #virt in OFTC | |
| 15:23:45 | bauzas | efried: I think we agreed something was necessary for keeping the existing behaviour | |
| 15:23:55 | gameon | sahid: Thank you for pointing me in the right direction, much appriciated | |
| 15:24:01 | bauzas | efried: because of some assumption from the filter | |
| 15:24:08 | bauzas | efried: that's usecase #1 in my spec | |
| 15:24:18 | efried | bauzas: In the case we're talking about, the NUMATopologyFilter may (or may not) have already filtered down to a certain subset of hosts. Now the scheduler knows it can ignore allocation requests related to those hosts. But it still has to employ additional logic to ignore allocation requests that don't have the right number of numa nodes in them. | |
| 15:24:26 | bauzas | efried: for usecase #2 (which also matches the SR-IOV usecase), we don't care | |
| 15:24:41 | bauzas | efried: right, I don't disagree with that | |
| 15:25:08 | efried | bauzas: That being the case, I'm not sure NUMATopologyFilter is actually doing us any good here. | |
| 15:25:29 | bauzas | that's the stephenfin vs. sahid point | |
| 15:25:54 | efried | bauzas: Because the scheduler is having to do the same logic anyway. The difference is that NTF had to throw away a lot of its work to just say "host is valid or not". | |
| 15:25:56 | bauzas | if we want to keep existing behaviour, we need something to force allocation requests to be disregarded by the filter | |
| 15:26:09 | efried | bauzas: Or the scheduler. And eff the filter. | |
| 15:27:50 | cfriesen | jaypipes: currently we have a "does this compute node have at least as good a CPU model as we asked for" filter | |
| 15:29:32 | cfriesen | jaypipes: we also have a "does this node have access to the physical networks required by the instance" filter | |
| 15:29:52 | melwitt | jackie-truong: hey, if you had noticed earlier that your blueprint got removed from the runway, that was an accident, sorry. I put it back. end date is EOD on April 16 | |
| 15:30:49 | jackie-truong | melwitt: No problem! I added a documentation patch to the list, too | |
| 15:30:58 | melwitt | k, cool | |
| 15:31:38 | cfriesen | efried: the filter is part of the scheduler | |
| 15:32:04 | cfriesen | efried: or are you using different terminology than I'm used to? | |
| 15:32:34 | efried | cfriesen: But IIUC, filters like NUMATopologyFilter are opt-in, and have a predefined interface (so folks can create custom ones), and only let you say "this host is good or bad". | |
| 15:32:40 | cfriesen | efried: to me the scheduler is placement+filters+weighers | |
| 15:33:18 | efried | cfriesen: I'm saying what we need here to make the world sane is a piece of code that "filters" allocation requests out of the result of GET /allocation_candidates. | |
| 15:33:56 | efried | cfriesen: In order for that code to be in a "filter" (as described above), we would have to extend/reinvent that predefined interface. | |
| 15:34:16 | efried | cfriesen: But if we do that filtering outside of that interface, we're not restricted. | |
| 15:34:35 | efried | cfriesen: Somewhere in the scheduler there is a piece of code that looks at the list of allocation requests we get back from placement, and picks one. | |
| 15:34:37 | efried | (or three) | |
| 15:35:32 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Don't persist RequestSpec.retry https://review.openstack.org/560143 | |
| 15:35:39 | efried | cfriesen: We need to get our dirty hands into that algorithm in order to satisfy this use case - unless we can find a way for the filtering to be done 100% in placement. | |
| 15:37:50 | edleafe | efried: I really don't like any approach that doesn't treat allocation_candidates as opaque | |
| 15:38:22 | efried | edleafe: Brace yourself. It's going to happen eventually. If not for this, then for something. | |
| 15:38:37 | cfriesen | efried: looks like SchedulerManager.select_destinations() calls out to placement, then passes the returned information down to FilterScheduler.select_destinations() | |
| 15:38:46 | efried | edleafe: Because we can't expect to implement *all* weighing on the placement side. | |
| 15:39:05 | efried | cfriesen: Is a "destination" a host in that context? | |
| 15:39:20 | efried | (trying not to look at code; have a pile of reviews I'm already in the middle of) | |
| 15:39:48 | cfriesen | efried: yes. Looks like when we call down to the actual filters (via _get_sorted_hosts() in this case, we don't pass the allocation candidates) | |
| 15:39:51 | edleafe | efried: it doesn't have to happen | |
| 15:40:29 | edleafe | I really oppose making the internal structure of an a-c part of the API contract | |
| 15:43:59 | efried | edleafe: /me predicts that that will be the path of least resistance when the alternative is implementing every conceivable filtering/weighing algorithm natively in placement. | |
| 15:44:06 | cfriesen | edleafe: I can see what efried is saying though...placement is returning allocation candidates, not hosts. Something in the rest of the code needs to understand how to map that allocation candidate to the actual resources being used. | |
| 15:44:46 | efried | cfriesen: That's slightly different. The format of the *allocation* is part of the contract, and the virt driver (or whatever) can introspect it to map to actual resources. | |
| 15:44:56 | gameon | sahid: Thanks again for your help - I have resolved the issue. It turns out Broadwell doesn't have all Haswell features and vice versa. So I've used IvyBridge | |
| 15:45:27 | efried | cfriesen: Although it amounts to the same thing, really. | |
| 15:45:35 | cfriesen | gameon: what was in haswell that's not in broadwell? | |
| 15:45:38 | efried | cfriesen: So yeah, edleafe there's your counterargument. | |
| 15:46:16 | sahid | gameon: cool :) | |
| 15:47:31 | edleafe | efried: I would much prefer modifying the provider info than the a-c | |
| 15:47:58 | efried | edleafe: But our problem isn't picking a provider. It's picking an allocation request. | |
| 15:48:07 | gameon | cfriesen: this was generated by running virsh capabilities - I the features reported by 'cat /proc/cpuinfo' are the same... http://paste.openstack.org/show/719064/ | |
| 15:48:18 | efried | edleafe: ...from among possibly many for a given provider. | |
| 15:48:41 | efried | edleafe: Besides, we're not talking about modifying anything about the format of the response. | |
| 15:48:48 | gameon | cfriesen: happy to hear alternative solutions to setting model to IvyBridge there's a better way of allowing LM between these set of hosts | |
| 15:49:23 | efried | edleafe: But we are talking about introspecting that payload. Which nothing in the doc implies we should avoid... | |
| 15:49:50 | edleafe | efried: but it also doesn't guarantee that the payload structure will never change | |
| 15:50:07 | edleafe | "Introspect at your own risk" :) | |
| 15:50:31 | cfriesen | gameon: according to my libvirt cpu model definitions Broadwell should be a superset of Haswell. | |
| 15:50:46 | bauzas | edleafe: just a bit of explanation | |
| 15:50:48 | cfriesen | gameon: and you might want to use the "noTSX" versions | |
| 15:51:09 | bauzas | edleafe: we discussed since 3 hours with efried about the possible implementation for NUMA | |
| 15:51:19 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Don't persist RequestSpec.retry https://review.openstack.org/560146 | |
| 15:51:32 | bauzas | edleafe: the problem is that the NUMA filter makes some assumptions that are not supported by the numbered request groups feature | |
| 15:51:48 | gameon | cfriesen: I'll give it a go, let's see. So set it to Broadwell or Haswell do you think? | |
| 15:51:58 | bauzas | edleafe: the one in particular is that it checks whether the host has the exact same topology than the asked guest one | |
| 15:52:07 | bauzas | edleafe: which seems weird to me, but meh | |
| 15:52:44 | cfriesen | gameon: interesting that it doesn't report your Haswell as a Haswell. | |
| 15:52:52 | bauzas | edleafe: accordingly, that means that placement will return some allocation requests that are valid from his point of view, but not from the filter's PoV | |
| 15:53:17 | bauzas | edleafe: hence the need of some post-filtering that would only assure the same behaviour | |
| 15:53:20 | bauzas | but | |