| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-07-03 | |||
| 17:40:45 | dansmith | storing the child providers in the numa object and a weigher to look at it seems like (a) it might be useful for other things and (b) a weigher isn't a huge amount of debt | |
| 17:41:28 | dansmith | like, I can imagine the "show me the instance topology" thing might use some info in that child object to construct a view of where your assigned gpu (or something else) fits into the topop | |
| 17:56:10 | efried | dansmith: We're talking about coding a weigher that would provide a subset of the capability, and be 100% obsolete when we do placement-based topo modeling & affinity. | |
| 17:56:19 | efried | reading material, like the placement side specs & patches? | |
| 17:58:09 | dansmith | efried: even if placement returns an AC with hierarchical allocations for the proper gpus, we'd still have weigher activity to decide which of those we want right? | |
| 17:58:29 | efried | what would we be weighing in that case? | |
| 17:58:52 | efried | like, what's a use case where "pick the first one" isn't right. | |
| 17:58:55 | dansmith | efried: there is always a decision to be made about which allocation we want right? | |
| 17:59:42 | sean-k-mooney | we are weighign host that can fufile numa affitny above those that cant | |
| 18:00:12 | dansmith | placement may return two ACs, one where the gpu is on a node with the minimum amount of memory, and one where there's a bunch extra, and a weigher would choose the latter I would think | |
| 18:00:39 | dansmith | or the weigher would opt for the AC with the best combination of cpu, memory and gpu, especially where there are more than one gpu and you want as even of a distribution as possible | |
| 18:01:09 | sean-k-mooney | the other usecase for the weigher would be to avoid placeing instace on a host with a gpu if they did not request one | |
| 18:01:10 | efried | I thought bauzas's spec proposes that we binarily weigh "satisfies affinity requirements" higher than "doesn't" | |
| 18:01:18 | sean-k-mooney | by weighering them lower | |
| 18:01:26 | dansmith | sean-k-mooney: that's what bauzas' proposed weigher is doing yeah, but I'm saying that may evolve to something still useful after placement is returning ACs with gpu elements | |
| 18:01:26 | sean-k-mooney | we do that in the pci weigher i think | |
| 18:01:42 | sean-k-mooney | right | |
| 18:01:42 | efried | I'm certainly not saying there's no value in a weigher | |
| 18:02:11 | sean-k-mooney | when modeled in placmenet if we have two AC on the same host it coudl weigh the ACs based on some other parmater | |
| 18:02:12 | efried | but weighing "satisfies affinity requirements" is a thing we can do very soon with placement, if we're willing to put the work in it. | |
| 18:02:17 | dansmith | I'm just saying I expect that the bones of the proposed weigher would still be useful after placement is returning more data | |
| 18:03:01 | dansmith | efried: and also, we've been on the verge of being able to do this for a while, so it's easy to never move forward on a sub-optimal approach in favor of something else that never comes, | |
| 18:03:26 | dansmith | so if there's something re-usable about this after the better thing comes, I'm not as concerned about the "100% debt after merge" concern | |
| 18:03:31 | efried | okay, so let's talk about how close we are to being able to move forward: | |
| 18:05:31 | sean-k-mooney | i see tetsuro pushed up https://review.opendev.org/#/c/668376/ on monday but he has -w it so i assume it still have more work to do. | |
| 18:06:14 | efried | For NUMA we need | |
| 18:06:14 | efried | rg-to-rp mappings (implemented, microversion 1.34) https://docs.openstack.org/placement/latest/specs/train/approved/placement-resource-provider-request-group-mapping-in-allocation-candidates.html | |
| 18:06:14 | efried | and pieces from https://docs.openstack.org/placement/latest/specs/train/approved/2005575-nested-magic-1.html as follows | |
| 18:06:14 | efried | arbitrary group suffixes (implemented, microversion 1.33) | |
| 18:06:14 | efried | same_subtree + resourceless request groups (https://review.opendev.org/#/c/668376/ -W as sean-k-mooney says, but pretty close) | |
| 18:06:36 | efried | and then obviously we have to do the things in nova to 1) reshape and 2) request accordingly. | |
| 18:06:59 | dansmith | yeah the reshape is what I'm concerned about.. that delaying us another cycle | |
| 18:07:01 | efried | I feel it is very well understood what 1 and 2 will look like. | |
| 18:07:29 | sean-k-mooney | 1 is https://review.opendev.org/#/q/topic:story/2006068+(status:open+OR+status:merged) right | |
| 18:07:50 | dansmith | those are all placement, | |
| 18:07:53 | dansmith | the reshape is in nova | |
| 18:08:13 | sean-k-mooney | oh ya but the reshap cant happen until the placment stuff is done | |
| 18:08:37 | dansmith | sure, just being clear | |
| 18:08:52 | dansmith | the reshape is more complicated (IMHO, and IME), so unless that's "close" I'd be more concerned | |
| 18:08:58 | sean-k-mooney | im not sure we fully underdand what the reshap will looklike | |
| 18:09:00 | efried | yeah, those are a minor bugfix for rg-to-rp mappings. Really only cdent's is needed, the rest are what-ifs. | |
| 18:09:29 | efried | The nested magic spec has several examples of what the topo will look like | |
| 18:09:48 | dansmith | I think knowing what it will look like is not so much the problem | |
| 18:10:38 | efried | deciding which resources need to go where? | |
| 18:10:50 | sean-k-mooney | yes | |
| 18:11:09 | dansmith | the actual mechanics of doing the reshape.. I know it's doable and we know what the end result will look like | |
| 18:11:25 | sean-k-mooney | for exampel i think cpus are going to need to be moded in a nested resouce provider of the numa node not as an inventory of the numa node | |
| 18:12:11 | sean-k-mooney | hugepages should proably be on the numa node but as a new resouce class per pagesize or as somthign else? | |
| 18:12:30 | dansmith | efried: case in point ^ :) | |
| 18:12:31 | sean-k-mooney | dose memroy_mb stay on the compute resouce proveiider? | |
| 18:12:41 | dansmith | still plenty of discussion to derail the reshape | |
| 18:15:48 | dansmith | efried: anyway, I think you oughta make a clear decision on this point, and -2 bauzas' thing if so, so that the messaging is clear what to expect and focus on | |
| 18:16:19 | dansmith | I'm not saying that's a terrible thing, I just think it's not terrible to continue with both | |
| 18:16:35 | dansmith | obviously I want the pure-placement approach for all the right reasons | |
| 18:17:02 | efried | "not terrible to continue with both" is why I haven't downvoted | |
| 18:17:25 | dansmith | okay | |
| 18:17:33 | efried | but "want the pure-placement approach" is why I haven't upvoted. | |
| 18:17:43 | efried | Who's going to work the code for bauzas's spec. | |
| 18:17:44 | efried | ? | |
| 18:17:52 | sean-k-mooney | bauzas: | |
| 18:17:55 | dansmith | um, bauzas I hope :D | |
| 18:18:15 | sean-k-mooney | assumign its approved if not he will bw working on other things | |
| 18:18:23 | dansmith | mriedem: any thoughts on this? if you and efried are both -0.9 then it's probably not worth expecting it to happen | |
| 18:18:26 | efried | If that person were dedicated instead to working on the topo reshape, that would get us closer | |
| 18:18:35 | efried | faster | |
| 18:18:42 | efried | even if it doesn't all come together in train. | |
| 18:19:04 | efried | whereas I'm not sure what we gain by having the separate weigher. | |
| 18:19:44 | sean-k-mooney | we get numa support for gpus in trian which will be the basis of our next longer support release donwstream | |
| 18:19:45 | dansmith | the weigher gets us a lot closer to the correct behavior with a minimal amount of code and pretty predictable amount of work | |
| 18:19:50 | dansmith | it doesn't get us to perfection, obviously | |
| 18:20:43 | efried | closer to the correct behavior... for just vgpu, right? | |
| 18:21:05 | dansmith | well, the bulk of this is reusable for other child devices, | |
| 18:21:07 | sean-k-mooney | we already do the right thing for pci pasthough | |
| 18:21:16 | sean-k-mooney | and for hugepages and cpu pinning | |
| 18:21:16 | dansmith | although I expect the weigher would probably be specific to gpus | |
| 18:21:24 | efried | other child devices - that we don't model in placement yet | |
| 18:21:31 | efried | so n/a? | |
| 18:22:08 | efried | and please let's not start modeling devices in non-NUMA-aware placement and adding weigher code to do the NUMA stuff for that. | |
| 18:22:16 | dansmith | efried: other things we hang off of the compute RP that aren't specific in placement, but yeah, nothing pending, just saying this isn't adding a bunch of stuff to do gpu-specific things | |
| 18:22:18 | sean-k-mooney | we dont track them in plamcneet and can provide better placmente and affintiy because of that | |
| 18:22:53 | efried | until we track them *properly* in placement, right. | |
| 18:23:07 | sean-k-mooney | which has been promsied for 2 years at this point | |
| 18:23:25 | efried | good, so let's get it the f done instead of keep kicking it down the road | |
| 18:23:32 | dansmith | efried: just FYI I hate myself for even arguing about this, because pure-placement is obviously the right way, I'm just hedging because of the pain of things not actually being in train | |
| 18:23:59 | dansmith | because seriously... years. | |
| 18:24:39 | efried | I've said before (or maybe just thought it) vgpu may have been nice as a foot in the door proving device modeling in placement, but premature. And now we're trying to retrofit to make it work instead of shelving it until we can make the infrastructure support it properly. | |
| 18:25:03 | efried | but | |
| 18:25:19 | efried | I still don't feel strongly enough to block | |
| 18:25:28 | sean-k-mooney | efried: for context the CAT stuff im working on here https://review.opendev.org/#/c/662264/ which you would also like me to track only with numa awer placment was requested for pike | |
| 18:25:38 | dansmith | if we didn't have basic gpu stuff that we have today at this point, I think we'd probably be having a much different conversation.. and not a fun one. | |
| 18:26:17 | efried | Tell ya, though, if sam_subtree lands pretty soon, as appears likely to do, that may be enough to make me -1 the vgpu affinity spec. | |
| 18:26:22 | dansmith | I thought you were the "don't wait forever for perfect" guy? :) | |
| 18:26:57 | efried | Totally. And I have no illusions that numa-in-placement will be perfect right away | |
| 18:27:19 | dansmith | well, if you think that's going to happen, you should just preemtively -1 it, or | |
| 18:27:21 | efried | which btw is why we should not be arguing about hugepages and splitting CPUs finer than per-numa-node. | |
| 18:27:28 | dansmith | or -1 it contingent on that merging when you think it will | |
| 18:27:44 | dansmith | because really, messaging the likelihood of this being a thing is pretty important to us | |
| 18:28:24 | efried | and why we punted on can_split | |
| 18:28:50 | efried | "messaging the likelihood of this being a thing" is actually the best argument I've heard so far. | |
| 18:29:52 | dansmith | it's the most important I think | |