Posted	Nick	Remark
#openstack-nova - 2018-09-17
15:41:37	dansmith	but you still need data from the compute, not just the types the compute is going to use, AFAIK
15:41:45	mriedem	isn't that already in placement?
15:41:50	mriedem	via the inventory record?
15:42:07	dansmith	for one type, but not the others
15:42:10	dansmith	bauzas: right?
15:42:34	mriedem	we don't support multiple types yet
15:42:44	mriedem	so one vgpu inventory record at most per compute node provider
15:42:46	dansmith	right, but this is to get us there right?
15:42:51	mriedem	yes
15:42:52	dansmith	or your idea is to reshape just the one inventory,
15:42:55	mriedem	yes
15:43:00	mriedem	that's what my patch above does
15:43:01	dansmith	and then let the compute fill it itn?
15:43:27	dansmith	so, that might work,
15:43:35	sean-k-mooney	dansmith: one of the issues with vgpus is that while you can have multilpel vgpus on the same pcpu like vm flavors dependng on what you have already created you may not be able to create others since they all share the same resouces
15:43:51	dansmith	yeah,
15:44:07	dansmith	so can't we have created some vgpus for guests on different cards that now need to be represented separately but weren't before?
15:44:15	dansmith	and without a view of the virt driver, that might not be doable
15:45:08	sean-k-mooney	dansmith: perhaps i belive today bauzas has confied the compute agent to only createing 1 vgpu type per host so it might not be be an issue
15:45:16	bauzas	dansmith: you can get the supported type by looking up the conf optionj
15:45:56	dansmith	type per host
15:45:57	bauzas	sean-k-mooney: that's correct, we only support one type
15:45:58	mriedem	if you changed the configured support vgpu type after creating some inventory/allocations but before reshaping, then we'd have a problem
15:46:09	bauzas	mriedem: yeah that's a limitation
15:46:13	dansmith	bauzas: one type per host, but potentially across multiple cards right?
15:46:20	bauzas	dansmith: correct
15:46:27	bauzas	libvirt only gives the total
15:46:30	mriedem	yeah
15:46:30	mriedem	isn't the multiple cards just the total?
15:46:33	dansmith	so can we know externally which instance should go to which new (separate) card provider?
15:46:33	sean-k-mooney	bauzas: you are proposing a whitelist model where the operator figures out what are compatible ahead of time rather then dynamicaly handeling this in nova/cyborg. that is deffinetly a step forward but may not cover all usecases well
15:46:38	bauzas	it can be sharded across cards
15:46:50	mriedem	tbc,
15:46:56	dansmith	presumably post-reshape, each allocation is against a single card for affinity reasons, right?
15:46:57	mriedem	my patch isn't creating multiple vgpu providers
15:47:04	dansmith	since they will eventually be tied to numa nodes
15:47:24	bauzas	dansmith: that's right
15:47:43	bauzas	dansmith: in my spec, I'm proposing the PCI ID as a way to know which card is for which type
15:47:56	mriedem	so we're going to have 1 vgpu child provider per card with a total inventory of 1?
15:48:02	bauzas	dansmith: what is uncovered in my spec is how an user can ask a certain type
15:48:04	dansmith	mriedem: no,
15:48:14	dansmith	mriedem: one card with however many vgpus it can support, right?
15:48:34	dansmith	one card per provider I mean
15:48:36	bauzas	mriedem: dansmith: no, it will create N children inventories, one per type
15:48:47	sean-k-mooney	dansmith: well even with out numa restictions that would be required as the shader units cannot pysically acess the ram on another card. at least not without an explictit direct dma
15:48:55	dansmith	bauzas: right, N inventories per M providers for M cards, yes?
15:48:55	mriedem	i guess i just need someone to tell me if https://review.openstack.org/599208 is the correct direction or not
15:48:56	bauzas	where the total of that inventory being the aggregated amount of capacity for each card
15:48:58	mriedem	for existing inventory/allocations
15:49:11	bauzas	dansmith: one PCI ID only supporting one type
15:49:15	dansmith	sean-k-mooney: ack yeah, and the allocations are not fungible..
15:49:29	dansmith	bauzas: I think you're confusing what I'm asking
15:50:00	bauzas	probably
15:50:15	bauzas	:)
15:50:21	dansmith	sean-k-mooney: btw, my inventory of 4 crunchie bars has been 75% allocated, with 25% now free
15:50:47	bauzas	dansmith: lemme try to clarify my spec for multiple types
15:50:52	sean-k-mooney	dansmith: haha next time ill have to bring a larger consignment
15:51:00	bauzas	dansmith: the operator defines which types he wants
15:51:09	dansmith	sean-k-mooney: get your importer license
15:51:20	dansmith	bauzas: I don't think we're talking about your spec
15:51:25	bauzas	dansmith: for each type, he has to mention which PCI IDs will be supported
15:51:27	bauzas	oh
15:51:31	bauzas	then I'm confused :)
15:51:32	dansmith	I think we're talking about how to reshape from the controller (or not)
15:51:45	bauzas	mriedem's patch ?
15:51:52	bauzas	it's simple in my mind
15:51:59	bauzas	we only support one type
15:52:18	bauzas	reshaping will just create a child with the exact same inventory
15:52:27	bauzas	and move allocations against it
15:52:40	sean-k-mooney	bauzas: do we today supprot 2+ pgpus on a host
15:52:58	bauzas	sean-k-mooney: yup, but they are hidden for libvirt
15:53:04	mriedem	bauzas: and that's what i've implemented
15:53:10	bauzas	mriedem: I know
15:53:13	bauzas	hence my +1
15:53:17	bauzas	so i don't get the concerns
15:53:27	bauzas	having multiple cards isn't a problem
15:53:40	bauzas	or I'm missing a crucial point
15:53:52	bauzas	I mean, I can be wrong but I don't get the problem
15:54:00	dansmith	bauzas: presumably we want to reshape inventories onto providers that represent actual cards, right?
15:54:09	bauzas	dansmith: no
15:54:10	sean-k-mooney	bauzas: i would have assumed that as part of reshaper we would have had to devide that compinded inventory across the pGPUs no ?
15:54:11	dansmith	not just reshape onto a provider that matches the one type we support today?
15:54:21	dansmith	bauzas: why not?
15:54:46	bauzas	dansmith: no, because once we support multiple types, they will still hide the fact that each type can be supported on N possible pGPUs
15:55:02	dansmith	how is that helpful?
15:55:04	dansmith	meaning,
15:55:14	dansmith	how does that help us represent the physical numa affinity?
15:55:15	sean-k-mooney	bauzas: you can hide that but you can also get the info via libvirt and expose it
15:55:34	dansmith	and, presumably a guest that wants 4 vgpus won't (presumably) be happy with 2 from 2 separate cards?
15:56:00	sean-k-mooney	dansmith: actully i think 4vgpus form 2 cards should be ok
15:56:21	bauzas	dansmith: ah shit, for NUMA affinity, topology is important
15:56:28	dansmith	sean-k-mooney: not if they look to be on the same NUMA node but aren't, right?
15:56:36	bauzas	shit shit shit
15:56:40	dansmith	...
15:56:45	sean-k-mooney	dansmith: right if they are split across numa nodes that would be bad
15:56:45	dansmith	isn't this the whole discussion we had in denver?
15:56:53	dansmith	sean-k-mooney: right
15:57:10	dansmith	I'm confused about why we had that discussion and are now having it again :)
15:57:15	sean-k-mooney	dansmith: but in general they will appear to be 4 different cards to the guest driver so it wont be able to tell the difference bar numa effects
15:57:20	dansmith	but I have to jump on a call in a few minutes

Earlier   Later