| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-09-17 | |||
| 15:47:24 | bauzas | dansmith: that's right | |
| 15:47:43 | bauzas | dansmith: in my spec, I'm proposing the PCI ID as a way to know which card is for which type | |
| 15:47:56 | mriedem | so we're going to have 1 vgpu child provider per card with a total inventory of 1? | |
| 15:48:02 | bauzas | dansmith: what is uncovered in my spec is how an user can ask a certain type | |
| 15:48:04 | dansmith | mriedem: no, | |
| 15:48:14 | dansmith | mriedem: one card with however many vgpus it can support, right? | |
| 15:48:34 | dansmith | one card per provider I mean | |
| 15:48:36 | bauzas | mriedem: dansmith: no, it will create N children inventories, one per type | |
| 15:48:47 | sean-k-mooney | dansmith: well even with out numa restictions that would be required as the shader units cannot pysically acess the ram on another card. at least not without an explictit direct dma | |
| 15:48:55 | dansmith | bauzas: right, N inventories per M providers for M cards, yes? | |
| 15:48:55 | mriedem | i guess i just need someone to tell me if https://review.openstack.org/599208 is the correct direction or not | |
| 15:48:56 | bauzas | where the total of that inventory being the aggregated amount of capacity for each card | |
| 15:48:58 | mriedem | for existing inventory/allocations | |
| 15:49:11 | bauzas | dansmith: one PCI ID only supporting one type | |
| 15:49:15 | dansmith | sean-k-mooney: ack yeah, and the allocations are not fungible.. | |
| 15:49:29 | dansmith | bauzas: I think you're confusing what I'm asking | |
| 15:50:00 | bauzas | probably | |
| 15:50:15 | bauzas | :) | |
| 15:50:21 | dansmith | sean-k-mooney: btw, my inventory of 4 crunchie bars has been 75% allocated, with 25% now free | |
| 15:50:47 | bauzas | dansmith: lemme try to clarify my spec for multiple types | |
| 15:50:52 | sean-k-mooney | dansmith: haha next time ill have to bring a larger consignment | |
| 15:51:00 | bauzas | dansmith: the operator defines which types he wants | |
| 15:51:09 | dansmith | sean-k-mooney: get your importer license | |
| 15:51:20 | dansmith | bauzas: I don't think we're talking about your spec | |
| 15:51:25 | bauzas | dansmith: for each type, he has to mention which PCI IDs will be supported | |
| 15:51:27 | bauzas | oh | |
| 15:51:31 | bauzas | then I'm confused :) | |
| 15:51:32 | dansmith | I think we're talking about how to reshape from the controller (or not) | |
| 15:51:45 | bauzas | mriedem's patch ? | |
| 15:51:52 | bauzas | it's simple in my mind | |
| 15:51:59 | bauzas | we only support one type | |
| 15:52:18 | bauzas | reshaping will just create a child with the exact same inventory | |
| 15:52:27 | bauzas | and move allocations against it | |
| 15:52:40 | sean-k-mooney | bauzas: do we today supprot 2+ pgpus on a host | |
| 15:52:58 | bauzas | sean-k-mooney: yup, but they are hidden for libvirt | |
| 15:53:04 | mriedem | bauzas: and that's what i've implemented | |
| 15:53:10 | bauzas | mriedem: I know | |
| 15:53:13 | bauzas | hence my +1 | |
| 15:53:17 | bauzas | so i don't get the concerns | |
| 15:53:27 | bauzas | having multiple cards isn't a problem | |
| 15:53:40 | bauzas | or I'm missing a crucial point | |
| 15:53:52 | bauzas | I mean, I can be wrong but I don't get the problem | |
| 15:54:00 | dansmith | bauzas: presumably we want to reshape inventories onto providers that represent actual cards, right? | |
| 15:54:09 | bauzas | dansmith: no | |
| 15:54:10 | sean-k-mooney | bauzas: i would have assumed that as part of reshaper we would have had to devide that compinded inventory across the pGPUs no ? | |
| 15:54:11 | dansmith | not just reshape onto a provider that matches the one type we support today? | |
| 15:54:21 | dansmith | bauzas: why not? | |
| 15:54:46 | bauzas | dansmith: no, because once we support multiple types, they will still hide the fact that each type can be supported on N possible pGPUs | |
| 15:55:02 | dansmith | how is that helpful? | |
| 15:55:04 | dansmith | meaning, | |
| 15:55:14 | dansmith | how does that help us represent the physical numa affinity? | |
| 15:55:15 | sean-k-mooney | bauzas: you can hide that but you can also get the info via libvirt and expose it | |
| 15:55:34 | dansmith | and, presumably a guest that wants 4 vgpus won't (presumably) be happy with 2 from 2 separate cards? | |
| 15:56:00 | sean-k-mooney | dansmith: actully i think 4vgpus form 2 cards should be ok | |
| 15:56:21 | bauzas | dansmith: ah shit, for NUMA affinity, topology is important | |
| 15:56:28 | dansmith | sean-k-mooney: not if they look to be on the same NUMA node but aren't, right? | |
| 15:56:36 | bauzas | shit shit shit | |
| 15:56:40 | dansmith | ... | |
| 15:56:45 | sean-k-mooney | dansmith: right if they are split across numa nodes that would be bad | |
| 15:56:45 | dansmith | isn't this the whole discussion we had in denver? | |
| 15:56:53 | dansmith | sean-k-mooney: right | |
| 15:57:10 | dansmith | I'm confused about why we had that discussion and are now having it again :) | |
| 15:57:15 | sean-k-mooney | dansmith: but in general they will appear to be 4 different cards to the guest driver so it wont be able to tell the difference bar numa effects | |
| 15:57:20 | dansmith | but I have to jump on a call in a few minutes | |
| 15:57:26 | dansmith | sean-k-mooney: sure | |
| 15:57:29 | bauzas | dansmith: in https://specs.openstack.org/openstack/nova-specs/specs/queens/implemented/add-support-for-vgpu.html#proposed-change we always thought on GPU types | |
| 15:58:02 | bauzas | dansmith: yes we barely discussed on NUMA affinity for GPUs | |
| 15:58:12 | dansmith | barely/ | |
| 15:58:15 | bauzas | dansmith: but in my mind, those were GPU types, and not cards | |
| 15:58:27 | dansmith | what was the whole reason to talk about the reshape happening on the compute node, AFAIK :) | |
| 15:58:55 | dansmith | anyway, /me -> call | |
| 15:59:20 | johnthetubaguy | +1 | |
| 15:59:47 | bauzas | mriedem: I just feel I'll necessarly have to help you on the reshaper patch | |
| 16:00:25 | bauzas | but I first need to consider whether a child being a pGPU and not a GPU type would change | |
| 16:02:00 | sean-k-mooney | bauzas: i think the pGPUs would be childeren of the numa nodes with inventoies of vgpu(types) | |
| 16:02:30 | sean-k-mooney | bauzas: we might also want to group pgpus into nvlink aggretates at some point but not now | |
| 16:03:58 | bauzas | sean-k-mooney: shhhhhhhhhhhht | |
| 16:04:00 | mriedem | bauzas: clearly. i only shat this out b/c it seemed trivial. i never made the link between the numa node stuff in the vgpu reshaper discussions. | |
| 16:04:12 | bauzas | sean-k-mooney: some customer could hear us and ask for this feature | |
| 16:04:27 | bauzas | mriedem: and my brain fucked on it, I'm sorry | |
| 16:05:09 | bauzas | that's where I think it's crucial to see the topology now and in the future | |
| 16:05:14 | bauzas | in order to prevent reshapes | |
| 16:05:27 | bauzas | we could implement chidren as GPU types and do reshapes later | |
| 16:05:38 | bauzas | or we could reshape it now and do the necessary work now | |
| 16:05:53 | bauzas | I think given FFU pain, the less reshapes we have, the better it will be | |
| 16:06:10 | bauzas | mriedem: permission to play with your change ? | |
| 16:06:36 | mriedem | bauzas: sure | |
| 16:06:46 | sean-k-mooney | bauzas: /window splitv | |
| 16:06:58 | sean-k-mooney | ... ignore that | |
| 16:07:37 | bauzas | call me vim | |
| 16:08:42 | sean-k-mooney | bauzas: can you do that in vim? i was setting back up my irc client "weechat" | |
| 16:08:57 | bauzas | you can split for sure | |
| 16:10:44 | bauzas | mriedem: I'm just afraid of Xen having different RP structure | |
| 16:11:02 | bauzas | I know we said the drivers are responsible, but that makes us differ | |
| 16:11:37 | sean-k-mooney | bauzas: well im sure hyperv and powervm will likely have different stutures too | |
| 16:13:27 | sean-k-mooney | bauzas: is your consern that you would not know the numa affinty of xen. xen report and supports numa affinity and cpu pinning via libvirt. we just dnot have the supprot in the nova driver | |
| 16:15:03 | bauzas | sean-k-mooney: no, I'm talking of XenServer driver not reporting GPUs like libvirt | |
| 16:15:54 | sean-k-mooney | bauzas: ah ok. i think that is ok provided that does not require use to change the flaovr in some way | |
| 16:16:06 | bauzas | right | |
| 16:16:40 | bauzas | anyway, now /me has to figure out the best way to lookup which GPU is attached to each running instance | |