Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-09-17
15:43:51 dansmith yeah,
15:44:07 dansmith so can't we have created some vgpus for guests on different cards that now need to be represented separately but weren't before?
15:44:15 dansmith and without a view of the virt driver, that might not be doable
15:45:08 sean-k-mooney dansmith: perhaps i belive today bauzas has confied the compute agent to only createing 1 vgpu type per host so it might not be be an issue
15:45:16 bauzas dansmith: you can get the supported type by looking up the conf optionj
15:45:56 dansmith type per host
15:45:57 bauzas sean-k-mooney: that's correct, we only support one type
15:45:58 mriedem if you changed the configured support vgpu type after creating some inventory/allocations but before reshaping, then we'd have a problem
15:46:09 bauzas mriedem: yeah that's a limitation
15:46:13 dansmith bauzas: one type per host, but potentially across multiple cards right?
15:46:20 bauzas dansmith: correct
15:46:27 bauzas libvirt only gives the total
15:46:30 mriedem isn't the multiple cards just the total?
15:46:30 mriedem yeah
15:46:33 sean-k-mooney bauzas: you are proposing a whitelist model where the operator figures out what are compatible ahead of time rather then dynamicaly handeling this in nova/cyborg. that is deffinetly a step forward but may not cover all usecases well
15:46:33 dansmith so can we know externally which instance should go to which new (separate) card provider?
15:46:38 bauzas it can be sharded across cards
15:46:50 mriedem tbc,
15:46:56 dansmith presumably post-reshape, each allocation is against a single card for affinity reasons, right?
15:46:57 mriedem my patch isn't creating multiple vgpu providers
15:47:04 dansmith since they will eventually be tied to numa nodes
15:47:24 bauzas dansmith: that's right
15:47:43 bauzas dansmith: in my spec, I'm proposing the PCI ID as a way to know which card is for which type
15:47:56 mriedem so we're going to have 1 vgpu child provider per card with a total inventory of 1?
15:48:02 bauzas dansmith: what is uncovered in my spec is how an user can ask a certain type
15:48:04 dansmith mriedem: no,
15:48:14 dansmith mriedem: one card with however many vgpus it can support, right?
15:48:34 dansmith one card per provider I mean
15:48:36 bauzas mriedem: dansmith: no, it will create N children inventories, one per type
15:48:47 sean-k-mooney dansmith: well even with out numa restictions that would be required as the shader units cannot pysically acess the ram on another card. at least not without an explictit direct dma
15:48:55 mriedem i guess i just need someone to tell me if https://review.openstack.org/599208 is the correct direction or not
15:48:55 dansmith bauzas: right, N inventories per M providers for M cards, yes?
15:48:56 bauzas where the total of that inventory being the aggregated amount of capacity for each card
15:48:58 mriedem for existing inventory/allocations
15:49:11 bauzas dansmith: one PCI ID only supporting one type
15:49:15 dansmith sean-k-mooney: ack yeah, and the allocations are not fungible..
15:49:29 dansmith bauzas: I think you're confusing what I'm asking
15:50:00 bauzas probably
15:50:15 bauzas :)
15:50:21 dansmith sean-k-mooney: btw, my inventory of 4 crunchie bars has been 75% allocated, with 25% now free
15:50:47 bauzas dansmith: lemme try to clarify my spec for multiple types
15:50:52 sean-k-mooney dansmith: haha next time ill have to bring a larger consignment
15:51:00 bauzas dansmith: the operator defines which types he wants
15:51:09 dansmith sean-k-mooney: get your importer license
15:51:20 dansmith bauzas: I don't think we're talking about your spec
15:51:25 bauzas dansmith: for each type, he has to mention which PCI IDs will be supported
15:51:27 bauzas oh
15:51:31 bauzas then I'm confused :)
15:51:32 dansmith I think we're talking about how to reshape from the controller (or not)
15:51:45 bauzas mriedem's patch ?
15:51:52 bauzas it's simple in my mind
15:51:59 bauzas we only support one type
15:52:18 bauzas reshaping will just create a child with the exact same inventory
15:52:27 bauzas and move allocations against it
15:52:40 sean-k-mooney bauzas: do we today supprot 2+ pgpus on a host
15:52:58 bauzas sean-k-mooney: yup, but they are hidden for libvirt
15:53:04 mriedem bauzas: and that's what i've implemented
15:53:10 bauzas mriedem: I know
15:53:13 bauzas hence my +1
15:53:17 bauzas so i don't get the concerns
15:53:27 bauzas having multiple cards isn't a problem
15:53:40 bauzas or I'm missing a crucial point
15:53:52 bauzas I mean, I can be wrong but I don't get the problem
15:54:00 dansmith bauzas: presumably we want to reshape inventories onto providers that represent actual cards, right?
15:54:09 bauzas dansmith: no
15:54:10 sean-k-mooney bauzas: i would have assumed that as part of reshaper we would have had to devide that compinded inventory across the pGPUs no ?
15:54:11 dansmith not just reshape onto a provider that matches the one type we support today?
15:54:21 dansmith bauzas: why not?
15:54:46 bauzas dansmith: no, because once we support multiple types, they will still hide the fact that each type can be supported on N possible pGPUs
15:55:02 dansmith how is that helpful?
15:55:04 dansmith meaning,
15:55:14 dansmith how does that help us represent the physical numa affinity?
15:55:15 sean-k-mooney bauzas: you can hide that but you can also get the info via libvirt and expose it
15:55:34 dansmith and, presumably a guest that wants 4 vgpus won't (presumably) be happy with 2 from 2 separate cards?
15:56:00 sean-k-mooney dansmith: actully i think 4vgpus form 2 cards should be ok
15:56:21 bauzas dansmith: ah shit, for NUMA affinity, topology is important
15:56:28 dansmith sean-k-mooney: not if they look to be on the same NUMA node but aren't, right?
15:56:36 bauzas shit shit shit
15:56:40 dansmith ...
15:56:45 dansmith isn't this the whole discussion we had in denver?
15:56:45 sean-k-mooney dansmith: right if they are split across numa nodes that would be bad
15:56:53 dansmith sean-k-mooney: right
15:57:10 dansmith I'm confused about why we had that discussion and are now having it again :)
15:57:15 sean-k-mooney dansmith: but in general they will appear to be 4 different cards to the guest driver so it wont be able to tell the difference bar numa effects
15:57:20 dansmith but I have to jump on a call in a few minutes
15:57:26 dansmith sean-k-mooney: sure
15:57:29 bauzas dansmith: in https://specs.openstack.org/openstack/nova-specs/specs/queens/implemented/add-support-for-vgpu.html#proposed-change we always thought on GPU types
15:58:02 bauzas dansmith: yes we barely discussed on NUMA affinity for GPUs
15:58:12 dansmith barely/
15:58:15 bauzas dansmith: but in my mind, those were GPU types, and not cards
15:58:27 dansmith what was the whole reason to talk about the reshape happening on the compute node, AFAIK :)
15:58:55 dansmith anyway, /me -> call
15:59:20 johnthetubaguy +1
15:59:47 bauzas mriedem: I just feel I'll necessarly have to help you on the reshaper patch
16:00:25 bauzas but I first need to consider whether a child being a pGPU and not a GPU type would change
16:02:00 sean-k-mooney bauzas: i think the pGPUs would be childeren of the numa nodes with inventoies of vgpu(types)
16:02:30 sean-k-mooney bauzas: we might also want to group pgpus into nvlink aggretates at some point but not now
16:03:58 bauzas sean-k-mooney: shhhhhhhhhhhht
16:04:00 mriedem bauzas: clearly. i only shat this out b/c it seemed trivial. i never made the link between the numa node stuff in the vgpu reshaper discussions.
16:04:12 bauzas sean-k-mooney: some customer could hear us and ask for this feature

Earlier   Later