Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-04-12
15:18:45 jaypipes bauzas: Besides "check my switch to see if it's on before scheduling to this node", what are the custom filters/weighers you have seen from operators?
15:18:48 bauzas efried: the problem is really because of backwards compatibility you know
15:19:13 bauzas efried: for the VGPU usecase, I don't care about that because scheduler will pick one allocation for me
15:19:31 efried bauzas: I contend that this is the same.
15:19:32 bauzas I don't have to make sure to mimic any pre-existing logic
15:19:41 melwitt dansmith, mriedem: ah, dammit. I messed up thinking the cert one was supposed to end on 2018-04-11. sorry
15:19:46 efried bauzas: The scheduler has to pick one allocation request based on the numa topology in the flavor.
15:21:28 gameon sahid: OK - setting them to kvm64 has seemed to have a different result. Maybe I didn't restart the service. Now I have 'Live Migration failure: unsupported configuration: Unable to find security driver for model apparmor: libvirtError: unsupported configuration: Unable to find security driver for model apparmor'
15:22:33 bauzas efried: for VGPUs ?
15:22:42 efried bauzas: For NUMA
15:22:47 sahid gameon: ok in same time i was trying to find a patch which address an issue with compare CPU but it seems that it's already on Pike https://review.openstack.org/#/c/53746/
15:23:20 bauzas efried: I'm confused
15:23:27 efried bauzas: So maybe "filter" is the wrong word for it. The scheduler is going to need to pick from among the allocation requests returned by GET /a_c. It's gotta employ *some* kind of logic to do that.
15:23:39 sahid gameon: I can't help you for the issue with apparmar perhaps you could try #virt in OFTC
15:23:45 bauzas efried: I think we agreed something was necessary for keeping the existing behaviour
15:23:55 gameon sahid: Thank you for pointing me in the right direction, much appriciated
15:24:01 bauzas efried: because of some assumption from the filter
15:24:08 bauzas efried: that's usecase #1 in my spec
15:24:18 efried bauzas: In the case we're talking about, the NUMATopologyFilter may (or may not) have already filtered down to a certain subset of hosts. Now the scheduler knows it can ignore allocation requests related to those hosts. But it still has to employ additional logic to ignore allocation requests that don't have the right number of numa nodes in them.
15:24:26 bauzas efried: for usecase #2 (which also matches the SR-IOV usecase), we don't care
15:24:41 bauzas efried: right, I don't disagree with that
15:25:08 efried bauzas: That being the case, I'm not sure NUMATopologyFilter is actually doing us any good here.
15:25:29 bauzas that's the stephenfin vs. sahid point
15:25:54 efried bauzas: Because the scheduler is having to do the same logic anyway. The difference is that NTF had to throw away a lot of its work to just say "host is valid or not".
15:25:56 bauzas if we want to keep existing behaviour, we need something to force allocation requests to be disregarded by the filter
15:26:09 efried bauzas: Or the scheduler. And eff the filter.
15:27:50 cfriesen jaypipes: currently we have a "does this compute node have at least as good a CPU model as we asked for" filter
15:29:32 cfriesen jaypipes: we also have a "does this node have access to the physical networks required by the instance" filter
15:29:52 melwitt jackie-truong: hey, if you had noticed earlier that your blueprint got removed from the runway, that was an accident, sorry. I put it back. end date is EOD on April 16
15:30:49 jackie-truong melwitt: No problem! I added a documentation patch to the list, too
15:30:58 melwitt k, cool
15:31:38 cfriesen efried: the filter is part of the scheduler
15:32:04 cfriesen efried: or are you using different terminology than I'm used to?
15:32:34 efried cfriesen: But IIUC, filters like NUMATopologyFilter are opt-in, and have a predefined interface (so folks can create custom ones), and only let you say "this host is good or bad".
15:32:40 cfriesen efried: to me the scheduler is placement+filters+weighers
15:33:18 efried cfriesen: I'm saying what we need here to make the world sane is a piece of code that "filters" allocation requests out of the result of GET /allocation_candidates.
15:33:56 efried cfriesen: In order for that code to be in a "filter" (as described above), we would have to extend/reinvent that predefined interface.
15:34:16 efried cfriesen: But if we do that filtering outside of that interface, we're not restricted.
15:34:35 efried cfriesen: Somewhere in the scheduler there is a piece of code that looks at the list of allocation requests we get back from placement, and picks one.
15:34:37 efried (or three)
15:35:32 openstackgerrit Matt Riedemann proposed openstack/nova stable/queens: Don't persist RequestSpec.retry https://review.openstack.org/560143
15:35:39 efried cfriesen: We need to get our dirty hands into that algorithm in order to satisfy this use case - unless we can find a way for the filtering to be done 100% in placement.
15:37:50 edleafe efried: I really don't like any approach that doesn't treat allocation_candidates as opaque
15:38:22 efried edleafe: Brace yourself. It's going to happen eventually. If not for this, then for something.
15:38:37 cfriesen efried: looks like SchedulerManager.select_destinations() calls out to placement, then passes the returned information down to FilterScheduler.select_destinations()
15:38:46 efried edleafe: Because we can't expect to implement *all* weighing on the placement side.
15:39:05 efried cfriesen: Is a "destination" a host in that context?
15:39:20 efried (trying not to look at code; have a pile of reviews I'm already in the middle of)
15:39:48 cfriesen efried: yes. Looks like when we call down to the actual filters (via _get_sorted_hosts() in this case, we don't pass the allocation candidates)
15:39:51 edleafe efried: it doesn't have to happen
15:40:29 edleafe I really oppose making the internal structure of an a-c part of the API contract
15:43:59 efried edleafe: /me predicts that that will be the path of least resistance when the alternative is implementing every conceivable filtering/weighing algorithm natively in placement.
15:44:06 cfriesen edleafe: I can see what efried is saying though...placement is returning allocation candidates, not hosts. Something in the rest of the code needs to understand how to map that allocation candidate to the actual resources being used.
15:44:46 efried cfriesen: That's slightly different. The format of the *allocation* is part of the contract, and the virt driver (or whatever) can introspect it to map to actual resources.
15:44:56 gameon sahid: Thanks again for your help - I have resolved the issue. It turns out Broadwell doesn't have all Haswell features and vice versa. So I've used IvyBridge
15:45:27 efried cfriesen: Although it amounts to the same thing, really.
15:45:35 cfriesen gameon: what was in haswell that's not in broadwell?
15:45:38 efried cfriesen: So yeah, edleafe there's your counterargument.
15:46:16 sahid gameon: cool :)
15:47:31 edleafe efried: I would much prefer modifying the provider info than the a-c
15:47:58 efried edleafe: But our problem isn't picking a provider. It's picking an allocation request.
15:48:07 gameon cfriesen: this was generated by running virsh capabilities - I the features reported by 'cat /proc/cpuinfo' are the same... http://paste.openstack.org/show/719064/
15:48:18 efried edleafe: ...from among possibly many for a given provider.
15:48:41 efried edleafe: Besides, we're not talking about modifying anything about the format of the response.
15:48:48 gameon cfriesen: happy to hear alternative solutions to setting model to IvyBridge there's a better way of allowing LM between these set of hosts
15:49:23 efried edleafe: But we are talking about introspecting that payload. Which nothing in the doc implies we should avoid...
15:49:50 edleafe efried: but it also doesn't guarantee that the payload structure will never change
15:50:07 edleafe "Introspect at your own risk" :)
15:50:31 cfriesen gameon: according to my libvirt cpu model definitions Broadwell should be a superset of Haswell.
15:50:46 bauzas edleafe: just a bit of explanation
15:50:48 cfriesen gameon: and you might want to use the "noTSX" versions
15:51:09 bauzas edleafe: we discussed since 3 hours with efried about the possible implementation for NUMA
15:51:19 openstackgerrit Matt Riedemann proposed openstack/nova stable/pike: Don't persist RequestSpec.retry https://review.openstack.org/560146
15:51:32 bauzas edleafe: the problem is that the NUMA filter makes some assumptions that are not supported by the numbered request groups feature
15:51:48 gameon cfriesen: I'll give it a go, let's see. So set it to Broadwell or Haswell do you think?
15:51:58 bauzas edleafe: the one in particular is that it checks whether the host has the exact same topology than the asked guest one
15:52:07 bauzas edleafe: which seems weird to me, but meh
15:52:44 cfriesen gameon: interesting that it doesn't report your Haswell as a Haswell.
15:52:52 bauzas edleafe: accordingly, that means that placement will return some allocation requests that are valid from his point of view, but not from the filter's PoV
15:53:17 bauzas edleafe: hence the need of some post-filtering that would only assure the same behaviour
15:53:20 bauzas but
15:53:21 gameon cfriesen: Yes I noticed this, reported as Nehalem which is a bit puzzling...
15:53:23 bauzas there is a but
15:53:25 cfriesen gameon: According to what you've got there I'd use Nehalem, unless you need the newer features.
15:53:57 bauzas edleafe: one other option could be to consider that placement should return the exact same candidates as the ones that the filter would agree
15:54:10 bauzas edleafe: in that case, it would mean some new query param for Placement
15:54:21 bauzas thoughts ?
15:55:05 gameon cfriesen: Nehalem works :)
15:56:27 efried edleafe: The way the doc is written, it *is* guaranteeing that the format won't change (without a new microversion).
15:56:53 efried which is as it should be.
15:57:10 edleafe efried: about to run off to a meeting
15:57:17 efried I've never understood the reasoning behind that payload needing to be opaque, btw.
15:57:50 edleafe efried: all it was supposed to be was something you could send back to allocate/claim
15:57:53 cfriesen bauzas: it seems to me that we have two options. give the filters access to the allocation candidates so they can rule out ones they don't like, or have enough flexibility in placement that we can ensure we never get back invalid candidates.
15:58:03 edleafe it could have been a hash, or a uuid, or...
15:58:22 efried edleafe: And you're supposed to pick one based on... what?
15:58:32 cfriesen edleafe: for that to work we need enough flexibility in what we can request from placement to ensure that all the candidates are valid
15:58:33 edleafe now we have nested, which brings in multiple a-cs per rp
15:58:35 melwitt efried: when you get a chance, wanna write some notes on how the update-provider-tree runway review went at L118? https://etherpad.openstack.org/p/nova-runways-rocky
15:58:47 efried melwitt: ack

Earlier   Later