Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-09-17
15:48:14 dansmith mriedem: one card with however many vgpus it can support, right?
15:48:34 dansmith one card per provider I mean
15:48:36 bauzas mriedem: dansmith: no, it will create N children inventories, one per type
15:48:47 sean-k-mooney dansmith: well even with out numa restictions that would be required as the shader units cannot pysically acess the ram on another card. at least not without an explictit direct dma
15:48:55 dansmith bauzas: right, N inventories per M providers for M cards, yes?
15:48:55 mriedem i guess i just need someone to tell me if https://review.openstack.org/599208 is the correct direction or not
15:48:56 bauzas where the total of that inventory being the aggregated amount of capacity for each card
15:48:58 mriedem for existing inventory/allocations
15:49:11 bauzas dansmith: one PCI ID only supporting one type
15:49:15 dansmith sean-k-mooney: ack yeah, and the allocations are not fungible..
15:49:29 dansmith bauzas: I think you're confusing what I'm asking
15:50:00 bauzas probably
15:50:15 bauzas :)
15:50:21 dansmith sean-k-mooney: btw, my inventory of 4 crunchie bars has been 75% allocated, with 25% now free
15:50:47 bauzas dansmith: lemme try to clarify my spec for multiple types
15:50:52 sean-k-mooney dansmith: haha next time ill have to bring a larger consignment
15:51:00 bauzas dansmith: the operator defines which types he wants
15:51:09 dansmith sean-k-mooney: get your importer license
15:51:20 dansmith bauzas: I don't think we're talking about your spec
15:51:25 bauzas dansmith: for each type, he has to mention which PCI IDs will be supported
15:51:27 bauzas oh
15:51:31 bauzas then I'm confused :)
15:51:32 dansmith I think we're talking about how to reshape from the controller (or not)
15:51:45 bauzas mriedem's patch ?
15:51:52 bauzas it's simple in my mind
15:51:59 bauzas we only support one type
15:52:18 bauzas reshaping will just create a child with the exact same inventory
15:52:27 bauzas and move allocations against it
15:52:40 sean-k-mooney bauzas: do we today supprot 2+ pgpus on a host
15:52:58 bauzas sean-k-mooney: yup, but they are hidden for libvirt
15:53:04 mriedem bauzas: and that's what i've implemented
15:53:10 bauzas mriedem: I know
15:53:13 bauzas hence my +1
15:53:17 bauzas so i don't get the concerns
15:53:27 bauzas having multiple cards isn't a problem
15:53:40 bauzas or I'm missing a crucial point
15:53:52 bauzas I mean, I can be wrong but I don't get the problem
15:54:00 dansmith bauzas: presumably we want to reshape inventories onto providers that represent actual cards, right?
15:54:09 bauzas dansmith: no
15:54:10 sean-k-mooney bauzas: i would have assumed that as part of reshaper we would have had to devide that compinded inventory across the pGPUs no ?
15:54:11 dansmith not just reshape onto a provider that matches the one type we support today?
15:54:21 dansmith bauzas: why not?
15:54:46 bauzas dansmith: no, because once we support multiple types, they will still hide the fact that each type can be supported on N possible pGPUs
15:55:02 dansmith how is that helpful?
15:55:04 dansmith meaning,
15:55:14 dansmith how does that help us represent the physical numa affinity?
15:55:15 sean-k-mooney bauzas: you can hide that but you can also get the info via libvirt and expose it
15:55:34 dansmith and, presumably a guest that wants 4 vgpus won't (presumably) be happy with 2 from 2 separate cards?
15:56:00 sean-k-mooney dansmith: actully i think 4vgpus form 2 cards should be ok
15:56:21 bauzas dansmith: ah shit, for NUMA affinity, topology is important
15:56:28 dansmith sean-k-mooney: not if they look to be on the same NUMA node but aren't, right?
15:56:36 bauzas shit shit shit
15:56:40 dansmith ...
15:56:45 sean-k-mooney dansmith: right if they are split across numa nodes that would be bad
15:56:45 dansmith isn't this the whole discussion we had in denver?
15:56:53 dansmith sean-k-mooney: right
15:57:10 dansmith I'm confused about why we had that discussion and are now having it again :)
15:57:15 sean-k-mooney dansmith: but in general they will appear to be 4 different cards to the guest driver so it wont be able to tell the difference bar numa effects
15:57:20 dansmith but I have to jump on a call in a few minutes
15:57:26 dansmith sean-k-mooney: sure
15:57:29 bauzas dansmith: in https://specs.openstack.org/openstack/nova-specs/specs/queens/implemented/add-support-for-vgpu.html#proposed-change we always thought on GPU types
15:58:02 bauzas dansmith: yes we barely discussed on NUMA affinity for GPUs
15:58:12 dansmith barely/
15:58:15 bauzas dansmith: but in my mind, those were GPU types, and not cards
15:58:27 dansmith what was the whole reason to talk about the reshape happening on the compute node, AFAIK :)
15:58:55 dansmith anyway, /me -> call
15:59:20 johnthetubaguy +1
15:59:47 bauzas mriedem: I just feel I'll necessarly have to help you on the reshaper patch
16:00:25 bauzas but I first need to consider whether a child being a pGPU and not a GPU type would change
16:02:00 sean-k-mooney bauzas: i think the pGPUs would be childeren of the numa nodes with inventoies of vgpu(types)
16:02:30 sean-k-mooney bauzas: we might also want to group pgpus into nvlink aggretates at some point but not now
16:03:58 bauzas sean-k-mooney: shhhhhhhhhhhht
16:04:00 mriedem bauzas: clearly. i only shat this out b/c it seemed trivial. i never made the link between the numa node stuff in the vgpu reshaper discussions.
16:04:12 bauzas sean-k-mooney: some customer could hear us and ask for this feature
16:04:27 bauzas mriedem: and my brain fucked on it, I'm sorry
16:05:09 bauzas that's where I think it's crucial to see the topology now and in the future
16:05:14 bauzas in order to prevent reshapes
16:05:27 bauzas we could implement chidren as GPU types and do reshapes later
16:05:38 bauzas or we could reshape it now and do the necessary work now
16:05:53 bauzas I think given FFU pain, the less reshapes we have, the better it will be
16:06:10 bauzas mriedem: permission to play with your change ?
16:06:36 mriedem bauzas: sure
16:06:46 sean-k-mooney bauzas: /window splitv
16:06:58 sean-k-mooney ... ignore that
16:07:37 bauzas call me vim
16:08:42 sean-k-mooney bauzas: can you do that in vim? i was setting back up my irc client "weechat"
16:08:57 bauzas you can split for sure
16:10:44 bauzas mriedem: I'm just afraid of Xen having different RP structure
16:11:02 bauzas I know we said the drivers are responsible, but that makes us differ
16:11:37 sean-k-mooney bauzas: well im sure hyperv and powervm will likely have different stutures too
16:13:27 sean-k-mooney bauzas: is your consern that you would not know the numa affinty of xen. xen report and supports numa affinity and cpu pinning via libvirt. we just dnot have the supprot in the nova driver
16:15:03 bauzas sean-k-mooney: no, I'm talking of XenServer driver not reporting GPUs like libvirt
16:15:54 sean-k-mooney bauzas: ah ok. i think that is ok provided that does not require use to change the flaovr in some way
16:16:06 bauzas right
16:16:40 bauzas anyway, now /me has to figure out the best way to lookup which GPU is attached to each running instance
16:22:31 openstackgerrit Matt Riedemann proposed openstack/nova master: libvirt: implement reshaper for vgpu https://review.openstack.org/599208
16:36:44 openstackgerrit John Garbutt proposed openstack/nova-specs master: Add Unified Limits Spec https://review.openstack.org/602201
16:41:33 openstackgerrit John Garbutt proposed openstack/nova-specs master: Add Unified Limits Spec https://review.openstack.org/602201
16:43:41 johnthetubaguy melwitt alex_xu I have attempted to capture the unified limits discussion here in here, still not complete, but getting closer: https://review.openstack.org/602201
16:50:34 bauzas mriedem: thanks for the last update on libvirt reshape, I now see your TODO(sbauza) and the direction it goes

Earlier   Later