| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-01-09 | |||
| 15:09:57 | dansmith | efried: the same argument goes fo "some vendor shows up with a P2V'd ancient appliance that requires a specific hardware topology and we spent years trying to implement that instead of actually competing on something that matters" | |
| 15:10:04 | sean-k-mooney | so as an operate i can create a flavor with 8 cores and pinnnge and then as a user i can upload an image the says make the vm have 1 socket and, 2 thread per core, and 1 nuam node please | |
| 15:10:25 | efried | dansmith: I dig. But NUMA isn't going away. | |
| 15:10:52 | efried | and edge tuning is like the big thing, isn't it? | |
| 15:11:01 | dansmith | efried: no, numa isn't, that's not the point.. whether or not we need to provide this level of control over what the topology looks like instead of just providing a sane one with the asks we were given is. | |
| 15:11:26 | dansmith | efried: mostly because of legacy appliances afaik :) | |
| 15:11:32 | efried | if a good-enough VM was good enough, sure. | |
| 15:12:01 | efried | but in edge IIUC you need to be able to squeeze every cycle out of your hardware, no? | |
| 15:12:15 | sean-k-mooney | well if we were to provide a sane topology by default we would have to change our curren policy of 1 cpu socket per vcpu | |
| 15:12:25 | sean-k-mooney | but i get what you mean | |
| 15:12:27 | dansmith | that's not generally the reason they need to strictly control the topology.. it seems like the reason, but it's often not :) | |
| 15:12:33 | sean-k-mooney | sane default would be nice | |
| 15:12:48 | efried | I think we have sane defaults, don't we? | |
| 15:12:49 | dansmith | anyway, I'm not trying to stop the train on those grounds, | |
| 15:13:10 | sean-k-mooney | efried: i would not call the cpu toplogy nova generates sane | |
| 15:13:15 | efried | heh | |
| 15:13:22 | efried | well, it could be | |
| 15:13:34 | dansmith | I'm more looking for a signpost in the distance of what level of abstraction others have chosen | |
| 15:13:41 | sean-k-mooney | if you have a 16 VCPU guest it will give you a guest with 16 cpu socket each wtih a 1 core 1 thread cpu | |
| 15:14:11 | sean-k-mooney | and for software licence that chages per socket that is expensive | |
| 15:14:27 | sean-k-mooney | that iw why the cpu topology extra specs where added | |
| 15:14:35 | dansmith | exactly :) | |
| 15:14:40 | dansmith | not for perf tuning :) | |
| 15:14:45 | sean-k-mooney | windows server used to be changed per socket not per core | |
| 15:15:04 | dansmith | and nvidia has some similar things | |
| 15:15:16 | sean-k-mooney | so if you are sane you would do hw:cpu_socket=1 hw:cpu_theads=2 | |
| 15:15:19 | dansmith | and closed source P2V'd ancient telco appliances are far far worse | |
| 15:15:24 | efried | dansmith: I think I see your point. It should be *possible* to say "give me $p pinned and $s shared, you go figure out how to distribute them" without *having* to say "I want CPUs 0,3,7 pinned and 1,2,4-6 shared". | |
| 15:15:44 | dansmith | efried: pretty much | |
| 15:15:58 | sean-k-mooney | efried: yes so that is what we wanted to discuss in the meeting when we said count vs mask | |
| 15:16:01 | efried | But I would argue that, if you care about a pinned vs shared split, you care enough about performance that having to do the extra syntax isn't a big jump. | |
| 15:16:04 | dansmith | there's also the device-to-numa affinity which I understand is important, so that your nic is affined to your dedicated cpus | |
| 15:16:12 | efried | mm | |
| 15:16:48 | sean-k-mooney | dansmith: that is out of scope of this spec but yes that is a factor | |
| 15:17:06 | dansmith | certainly you know I'm not arguing that operators of multi-billion-dollar clouds shouldn't have to write 255 characters of ascii syntax for this right? | |
| 15:17:22 | dansmith | sean-k-mooney: no, I know, I was just commenting that I know asking for the abstract isn't enough | |
| 15:17:30 | dansmith | sean-k-mooney: or rather, asking for his abstract example | |
| 15:17:50 | sean-k-mooney | so if we take numa as an example | |
| 15:17:55 | sean-k-mooney | we actully support both | |
| 15:18:04 | sean-k-mooney | if you do hw:numa_nodes=2 | |
| 15:18:17 | sean-k-mooney | and say nothing esle we devied the cpu and ram evenly between the numa nodes | |
| 15:18:49 | sean-k-mooney | but you can also use hw:numa_cpu.0=2 .... to speciy the numa of cpus or ram per numa node | |
| 15:19:04 | sean-k-mooney | most peopel jsut hw:numa_nodes and let nova do it | |
| 15:19:58 | sean-k-mooney | i guess the diffferen with numa id they can detect the cpu and ram in the gues via the kernel jsut as if it was real hardware | |
| 15:20:40 | sean-k-mooney | pining vs floaing is a qualtive change ranter then quantative so if they care about that then need another openstack sepcifc way to determin that | |
| 15:22:04 | sean-k-mooney | the primary use case for this by the way that im aware of is for realtime guests where they want all the realtime cores to be pinned and the non realtime managment core to be floating so that they can get the most desity out of a host | |
| 15:22:08 | dansmith | efried: so I guess to close, | |
| 15:23:18 | dansmith | efried: my preference (and what I thought we were moving towards) was a future where resource (amounts) are specified in placement syntax, and topologies and the like are assigned sanely. Tweaks to how those look (like I really need this thing pinned here) would be hw:things but where the hw: is the optional | |
| 15:23:52 | dansmith | meaning, usually you just ask for the amount of stuff you want and we give it to you in a sane arrangement which is fine for most people, | |
| 15:24:07 | dansmith | and hw:tweak your way to a specific arrangement if that's important | |
| 15:24:20 | dansmith | a more clear delineation of which thing does what | |
| 15:24:40 | dansmith | with the hw: being the primary, then I'm not sure how you know when you do or don't or can't specify some resource count as well | |
| 15:25:13 | dansmith | but anyway, I grok the complexity on both sides and I'm sure both will have warts | |
| 15:25:37 | sean-k-mooney | i could see that working if we had a speartion between the placment query and the resouces*:* element in the extra specs | |
| 15:25:56 | efried | We're very close to agreeing here. I want the clear delineation to be that placement-ese syntax doesn't have side effects. That's the same delineation as you describe in *most* cases. | |
| 15:26:43 | dansmith | well, in what I said above, asking for PCPUS:2 would get you two pinned cpus, which I think you're describing as a side effect | |
| 15:26:59 | sean-k-mooney | dansmith: in your world would you see the need to supprot the number group syntax in the flavor for example | |
| 15:27:55 | dansmith | sean-k-mooney: I will totally admit that I don't (ever) have the resource group numbering quagmire swapped into my head, but I assume yeah, you might need an extension to placement syntax to group those things if that's what you mean | |
| 15:28:02 | dansmith | resources:2:PCPUS=2 I assume | |
| 15:28:53 | sean-k-mooney | the numbered sysntax means that all resouce requst in that group must come from the same RP | |
| 15:29:19 | sean-k-mooney | so if you have resources:2:PCPUS=2,MEMORY_MB=1024 | |
| 15:29:34 | dansmith | ack | |
| 15:29:39 | sean-k-mooney | then you need 2 pinned cpus and 1G of memory form the same resouce provider | |
| 15:29:43 | dansmith | yeah, I dunno, that's encoding a little bit of topology into the placement syntax, | |
| 15:29:54 | sean-k-mooney | so if you use that and we move to numa in placmenet it will break | |
| 15:30:05 | dansmith | where we should be able to parse the hw: things and alter the actual things we ask placement for inside nova I think | |
| 15:30:16 | efried | dansmith: exactly ^ | |
| 15:30:36 | sean-k-mooney | yes so if you only used the unmbered syntax | |
| 15:30:50 | sean-k-mooney | and had the hw:* things tweek the toplogy i can see it working | |
| 15:30:52 | efried | The existing (unsuffixed) placement syntax is already pretty complicated to understand. Trying to add group identifiers to that will make it worse. | |
| 15:30:56 | sean-k-mooney | as nova can alter it interally | |
| 15:31:14 | efried | and then you have to understand *how* we model numa in placement | |
| 15:31:24 | sean-k-mooney | well we already support grouping of bothe the resouce and trait requests | |
| 15:31:26 | dansmith | sean-k-mooney: right, so I'd say do that.. placement syntax for totals, hw syntax for details, let nova alter what we ask placement for based on any tweaks you asked for | |
| 15:31:29 | efried | and how group identifiers correspond to that. | |
| 15:31:44 | dansmith | efried: I totes don't understand how current placement syntax is hard to understand | |
| 15:31:54 | sean-k-mooney | dansmith: ya i could see that working as we aready alter stuff with the prefilters now | |
| 15:32:17 | efried | dansmith: maybe you're too close to it | |
| 15:32:36 | sean-k-mooney | dansmith: its complicated if you use the numbered groups | |
| 15:32:49 | sean-k-mooney | if you dont ist not that bad and its consitent | |
| 15:33:33 | dansmith | sean-k-mooney: but we don't use numbered groups in the flavor placement syntax today right? | |
| 15:33:35 | sean-k-mooney | your are forced to use the groups today if you want to apply trait requirement to specific resouces | |
| 15:33:43 | sean-k-mooney | dansmith: we do | |
| 15:33:47 | efried | I think we support it, yes | |
| 15:33:50 | sean-k-mooney | well you can use them | |
| 15:33:56 | efried | we just don't have many use cases that need it today. | |
| 15:34:01 | efried | perhaps... none? | |
| 15:34:04 | dansmith | efried: well, I've had to explain it a few times and it has always been really nice, but okay | |
| 15:34:21 | dansmith | okay, I didn't know we even supported the group numbering today | |
| 15:34:36 | sean-k-mooney | it was added so you could say these traits are for my nic and these other traits are for my cpu | |
| 15:35:10 | sean-k-mooney | but since we dont have traiats that apply to mulple resouce classes today? do we? i dont think its stictly required | |
| 15:35:18 | dansmith | sean-k-mooney: is the reason we need that an artifact of how placement works or something? | |
| 15:35:39 | dansmith | because I would think that with no traits that apply to both cpus and nics that wouldn't be a thing I need to specify | |
| 15:35:48 | efried | dansmith: I get what you're saying. For most cases, the placement-ese syntax isn't any more complicated than the hw:-ese syntax. I'm asserting that we will for sure need the latter, and understanding *both* is harder than understanding *one* of them. | |
| 15:35:57 | sean-k-mooney | well you cant do resocues:<resouce class>=X; <traits for that RC> | |
| 15:36:32 | sean-k-mooney | so you have to do resources1:PCPU traits1=AVX=required | |
| 15:36:39 | dansmith | efried: I'm saying I think placement syntax (not including groups) is simpler and more consistent than hw syntax and is easier to understand as a whole DSL on that basis | |
| 15:36:57 | sean-k-mooney | where the sufix 1 in this case pairs the trait and request group | |
| 15:37:21 | efried | sean-k-mooney: which only matters if we have multiple providers, which today is only the case for VGPU and bandwidth. | |
| 15:37:28 | efried | right? | |