| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-07-03 | |||
| 18:02:17 | dansmith | I'm just saying I expect that the bones of the proposed weigher would still be useful after placement is returning more data | |
| 18:03:01 | dansmith | efried: and also, we've been on the verge of being able to do this for a while, so it's easy to never move forward on a sub-optimal approach in favor of something else that never comes, | |
| 18:03:26 | dansmith | so if there's something re-usable about this after the better thing comes, I'm not as concerned about the "100% debt after merge" concern | |
| 18:03:31 | efried | okay, so let's talk about how close we are to being able to move forward: | |
| 18:05:31 | sean-k-mooney | i see tetsuro pushed up https://review.opendev.org/#/c/668376/ on monday but he has -w it so i assume it still have more work to do. | |
| 18:06:14 | efried | For NUMA we need | |
| 18:06:14 | efried | rg-to-rp mappings (implemented, microversion 1.34) https://docs.openstack.org/placement/latest/specs/train/approved/placement-resource-provider-request-group-mapping-in-allocation-candidates.html | |
| 18:06:14 | efried | and pieces from https://docs.openstack.org/placement/latest/specs/train/approved/2005575-nested-magic-1.html as follows | |
| 18:06:14 | efried | arbitrary group suffixes (implemented, microversion 1.33) | |
| 18:06:14 | efried | same_subtree + resourceless request groups (https://review.opendev.org/#/c/668376/ -W as sean-k-mooney says, but pretty close) | |
| 18:06:36 | efried | and then obviously we have to do the things in nova to 1) reshape and 2) request accordingly. | |
| 18:06:59 | dansmith | yeah the reshape is what I'm concerned about.. that delaying us another cycle | |
| 18:07:01 | efried | I feel it is very well understood what 1 and 2 will look like. | |
| 18:07:29 | sean-k-mooney | 1 is https://review.opendev.org/#/q/topic:story/2006068+(status:open+OR+status:merged) right | |
| 18:07:50 | dansmith | those are all placement, | |
| 18:07:53 | dansmith | the reshape is in nova | |
| 18:08:13 | sean-k-mooney | oh ya but the reshap cant happen until the placment stuff is done | |
| 18:08:37 | dansmith | sure, just being clear | |
| 18:08:52 | dansmith | the reshape is more complicated (IMHO, and IME), so unless that's "close" I'd be more concerned | |
| 18:08:58 | sean-k-mooney | im not sure we fully underdand what the reshap will looklike | |
| 18:09:00 | efried | yeah, those are a minor bugfix for rg-to-rp mappings. Really only cdent's is needed, the rest are what-ifs. | |
| 18:09:29 | efried | The nested magic spec has several examples of what the topo will look like | |
| 18:09:48 | dansmith | I think knowing what it will look like is not so much the problem | |
| 18:10:38 | efried | deciding which resources need to go where? | |
| 18:10:50 | sean-k-mooney | yes | |
| 18:11:09 | dansmith | the actual mechanics of doing the reshape.. I know it's doable and we know what the end result will look like | |
| 18:11:25 | sean-k-mooney | for exampel i think cpus are going to need to be moded in a nested resouce provider of the numa node not as an inventory of the numa node | |
| 18:12:11 | sean-k-mooney | hugepages should proably be on the numa node but as a new resouce class per pagesize or as somthign else? | |
| 18:12:30 | dansmith | efried: case in point ^ :) | |
| 18:12:31 | sean-k-mooney | dose memroy_mb stay on the compute resouce proveiider? | |
| 18:12:41 | dansmith | still plenty of discussion to derail the reshape | |
| 18:15:48 | dansmith | efried: anyway, I think you oughta make a clear decision on this point, and -2 bauzas' thing if so, so that the messaging is clear what to expect and focus on | |
| 18:16:19 | dansmith | I'm not saying that's a terrible thing, I just think it's not terrible to continue with both | |
| 18:16:35 | dansmith | obviously I want the pure-placement approach for all the right reasons | |
| 18:17:02 | efried | "not terrible to continue with both" is why I haven't downvoted | |
| 18:17:25 | dansmith | okay | |
| 18:17:33 | efried | but "want the pure-placement approach" is why I haven't upvoted. | |
| 18:17:43 | efried | Who's going to work the code for bauzas's spec. | |
| 18:17:44 | efried | ? | |
| 18:17:52 | sean-k-mooney | bauzas: | |
| 18:17:55 | dansmith | um, bauzas I hope :D | |
| 18:18:15 | sean-k-mooney | assumign its approved if not he will bw working on other things | |
| 18:18:23 | dansmith | mriedem: any thoughts on this? if you and efried are both -0.9 then it's probably not worth expecting it to happen | |
| 18:18:26 | efried | If that person were dedicated instead to working on the topo reshape, that would get us closer | |
| 18:18:35 | efried | faster | |
| 18:18:42 | efried | even if it doesn't all come together in train. | |
| 18:19:04 | efried | whereas I'm not sure what we gain by having the separate weigher. | |
| 18:19:44 | sean-k-mooney | we get numa support for gpus in trian which will be the basis of our next longer support release donwstream | |
| 18:19:45 | dansmith | the weigher gets us a lot closer to the correct behavior with a minimal amount of code and pretty predictable amount of work | |
| 18:19:50 | dansmith | it doesn't get us to perfection, obviously | |
| 18:20:43 | efried | closer to the correct behavior... for just vgpu, right? | |
| 18:21:05 | dansmith | well, the bulk of this is reusable for other child devices, | |
| 18:21:07 | sean-k-mooney | we already do the right thing for pci pasthough | |
| 18:21:16 | sean-k-mooney | and for hugepages and cpu pinning | |
| 18:21:16 | dansmith | although I expect the weigher would probably be specific to gpus | |
| 18:21:24 | efried | other child devices - that we don't model in placement yet | |
| 18:21:31 | efried | so n/a? | |
| 18:22:08 | efried | and please let's not start modeling devices in non-NUMA-aware placement and adding weigher code to do the NUMA stuff for that. | |
| 18:22:16 | dansmith | efried: other things we hang off of the compute RP that aren't specific in placement, but yeah, nothing pending, just saying this isn't adding a bunch of stuff to do gpu-specific things | |
| 18:22:18 | sean-k-mooney | we dont track them in plamcneet and can provide better placmente and affintiy because of that | |
| 18:22:53 | efried | until we track them *properly* in placement, right. | |
| 18:23:07 | sean-k-mooney | which has been promsied for 2 years at this point | |
| 18:23:25 | efried | good, so let's get it the f done instead of keep kicking it down the road | |
| 18:23:32 | dansmith | efried: just FYI I hate myself for even arguing about this, because pure-placement is obviously the right way, I'm just hedging because of the pain of things not actually being in train | |
| 18:23:59 | dansmith | because seriously... years. | |
| 18:24:39 | efried | I've said before (or maybe just thought it) vgpu may have been nice as a foot in the door proving device modeling in placement, but premature. And now we're trying to retrofit to make it work instead of shelving it until we can make the infrastructure support it properly. | |
| 18:25:03 | efried | but | |
| 18:25:19 | efried | I still don't feel strongly enough to block | |
| 18:25:28 | sean-k-mooney | efried: for context the CAT stuff im working on here https://review.opendev.org/#/c/662264/ which you would also like me to track only with numa awer placment was requested for pike | |
| 18:25:38 | dansmith | if we didn't have basic gpu stuff that we have today at this point, I think we'd probably be having a much different conversation.. and not a fun one. | |
| 18:26:17 | efried | Tell ya, though, if sam_subtree lands pretty soon, as appears likely to do, that may be enough to make me -1 the vgpu affinity spec. | |
| 18:26:22 | dansmith | I thought you were the "don't wait forever for perfect" guy? :) | |
| 18:26:57 | efried | Totally. And I have no illusions that numa-in-placement will be perfect right away | |
| 18:27:19 | dansmith | well, if you think that's going to happen, you should just preemtively -1 it, or | |
| 18:27:21 | efried | which btw is why we should not be arguing about hugepages and splitting CPUs finer than per-numa-node. | |
| 18:27:28 | dansmith | or -1 it contingent on that merging when you think it will | |
| 18:27:44 | dansmith | because really, messaging the likelihood of this being a thing is pretty important to us | |
| 18:28:24 | efried | and why we punted on can_split | |
| 18:28:50 | efried | "messaging the likelihood of this being a thing" is actually the best argument I've heard so far. | |
| 18:29:52 | dansmith | it's the most important I think | |
| 18:29:58 | sean-k-mooney | well if we cant have memory fine then a numa node we can implemenat CAT | |
| 18:30:17 | sean-k-mooney | the cpus inventories and cache inventores shoudl be in the same RP | |
| 18:30:26 | sean-k-mooney | and cache regins are finer then numa node | |
| 18:30:40 | sean-k-mooney | we can have mulplie l3 cache regins in the same numa node | |
| 18:30:49 | efried | understood. | |
| 18:30:56 | efried | So | |
| 18:31:23 | sean-k-mooney | it would basicaly be a 3 level tree | |
| 18:31:25 | efried | when we implement CAT *after* having implemented NUMA topo as its own thing, we can reshape the CPUs and L3s down into grandchild providers. | |
| 18:31:45 | sean-k-mooney | yes we could | |
| 18:32:00 | efried | but we should not be trying to solve CAT preemptively. | |
| 18:32:02 | sean-k-mooney | or we could do that first and reshap them under the numa node when they are added | |
| 18:32:28 | efried | I owe you a response in the pqos spec, but that doesn't buy us anything at this point. | |
| 18:32:48 | sean-k-mooney | we talked about it a little today in our squad call | |
| 18:33:02 | sean-k-mooney | im going to simplfy it and proably add a depency on some of the placmenet stuff | |
| 18:33:24 | sean-k-mooney | i will likely make memory bandwith depend on the numa in placment spec | |
| 18:33:38 | sean-k-mooney | i would like to keep cache not depenent on tha tbut may on same tree | |
| 18:34:34 | sean-k-mooney | well it depends i have to see is there a way to us more of the plamcent stuff | |
| 18:35:05 | sean-k-mooney | it significantly incresase the risk we wont have CAT in train however | |
| 18:35:53 | efried | I'm addressing similar concerns in the providers.yaml spec | |
| 18:36:47 | sean-k-mooney | ill admit i have nto really been following that one. i kindof got burt out on that topic last cycle | |