Posted	Nick	Remark
#openstack-nova - 2018-09-17
15:23:47	bauzas	mriedem: so there is litterally no formatting
15:23:48	mriedem	and i'm passing it through to placement
15:23:57	bauzas	and passing to operators
15:24:05	bauzas	that's awesome
15:24:11	mriedem	well the operator is the one that configures nova with the type
15:24:12	mriedem	so yeah
15:26:08	jaypipes	bauzas: done.
15:26:55	jaypipes	bauzas: by "done", I mean I've re-reviewed the Stein vGPU types spec.
15:28:24	bauzas	jaypipes: gotcha, and thanks
15:28:34	bauzas	jaypipes: yeah, I think your comment is all good with me
15:30:08	bauzas	jaypipes: tbc, I think I can discover certain things like framebuffer size and what's described in https://docs.nvidia.com/grid/6.0/grid-vgpu-user-guide/index.html#vgpu-types-tesla-m60 and others
15:30:26	bauzas	jaypipes: but GPU capabilities will have to be described by operators directly
15:31:56	jaypipes	bauzas: what is the user asking for? I've only seen requests for things like "my application is built with CUDA library X and therefore can take advantage of NVIDIA Compute Capability Y hardware, so make sure I get on a GPU that has that Compute Capability".
15:32:26	bauzas	yeah probably
15:32:41	bauzas	jaypipes: that's where your idea of an YAML inventory could help
15:33:05	bauzas	but for the moment, I'm not happy with nova supporting vendor-specific features by code
15:33:24	bauzas	anyway, speaking of code is better with code
15:36:36	openstackgerrit	Merged openstack/nova-specs master: Add support specify volume type when boot instance https://review.openstack.org/579520
15:38:00	mriedem	libvirt reshaper patch updated
15:38:08	openstackgerrit	Matt Riedemann proposed openstack/nova master: libvirt: implement reshaper for vgpu https://review.openstack.org/599208
15:38:13	mriedem	i'm pondering how useful a pre-upgrade check could be for this
15:38:30	mriedem	as in, i'm not sure how useful it would be since it's not something you can run offline
15:39:01	mriedem	a pre-upgrade check would mostly be for reporting - yes you have x number of providers of these things and you're going to need to upgrade those via reshaper
15:40:16	mriedem	although...why can't we run this offline? if the vgpu inventory is already on the root compute node provider and has allocations, we can call the reshaper API - the only thing we'd need is the gpu type and we can get that from config
15:40:26	mriedem	lyarwood: dansmith: ^
15:40:52	dansmith	each compute node can have different types, right?
15:41:09	dansmith	we need the quantity from the virt driver at least
15:41:15	dansmith	we know which types, but not how many of each
15:41:20	mriedem	you'd have to run the data migration per compute yes
15:41:20	dansmith	right now we expose N of one type, IIRC
15:41:29	mriedem	like the ironic instance flavor one
15:41:37	dansmith	but you still need data from the compute, not just the types the compute is going to use, AFAIK
15:41:45	mriedem	isn't that already in placement?
15:41:50	mriedem	via the inventory record?
15:42:07	dansmith	for one type, but not the others
15:42:10	dansmith	bauzas: right?
15:42:34	mriedem	we don't support multiple types yet
15:42:44	mriedem	so one vgpu inventory record at most per compute node provider
15:42:46	dansmith	right, but this is to get us there right?
15:42:51	mriedem	yes
15:42:52	dansmith	or your idea is to reshape just the one inventory,
15:42:55	mriedem	yes
15:43:00	mriedem	that's what my patch above does
15:43:01	dansmith	and then let the compute fill it itn?
15:43:27	dansmith	so, that might work,
15:43:35	sean-k-mooney	dansmith: one of the issues with vgpus is that while you can have multilpel vgpus on the same pcpu like vm flavors dependng on what you have already created you may not be able to create others since they all share the same resouces
15:43:51	dansmith	yeah,
15:44:07	dansmith	so can't we have created some vgpus for guests on different cards that now need to be represented separately but weren't before?
15:44:15	dansmith	and without a view of the virt driver, that might not be doable
15:45:08	sean-k-mooney	dansmith: perhaps i belive today bauzas has confied the compute agent to only createing 1 vgpu type per host so it might not be be an issue
15:45:16	bauzas	dansmith: you can get the supported type by looking up the conf optionj
15:45:56	dansmith	type per host
15:45:57	bauzas	sean-k-mooney: that's correct, we only support one type
15:45:58	mriedem	if you changed the configured support vgpu type after creating some inventory/allocations but before reshaping, then we'd have a problem
15:46:09	bauzas	mriedem: yeah that's a limitation
15:46:13	dansmith	bauzas: one type per host, but potentially across multiple cards right?
15:46:20	bauzas	dansmith: correct
15:46:27	bauzas	libvirt only gives the total
15:46:30	mriedem	isn't the multiple cards just the total?
15:46:30	mriedem	yeah
15:46:33	sean-k-mooney	bauzas: you are proposing a whitelist model where the operator figures out what are compatible ahead of time rather then dynamicaly handeling this in nova/cyborg. that is deffinetly a step forward but may not cover all usecases well
15:46:33	dansmith	so can we know externally which instance should go to which new (separate) card provider?
15:46:38	bauzas	it can be sharded across cards
15:46:50	mriedem	tbc,
15:46:56	dansmith	presumably post-reshape, each allocation is against a single card for affinity reasons, right?
15:46:57	mriedem	my patch isn't creating multiple vgpu providers
15:47:04	dansmith	since they will eventually be tied to numa nodes
15:47:24	bauzas	dansmith: that's right
15:47:43	bauzas	dansmith: in my spec, I'm proposing the PCI ID as a way to know which card is for which type
15:47:56	mriedem	so we're going to have 1 vgpu child provider per card with a total inventory of 1?
15:48:02	bauzas	dansmith: what is uncovered in my spec is how an user can ask a certain type
15:48:04	dansmith	mriedem: no,
15:48:14	dansmith	mriedem: one card with however many vgpus it can support, right?
15:48:34	dansmith	one card per provider I mean
15:48:36	bauzas	mriedem: dansmith: no, it will create N children inventories, one per type
15:48:47	sean-k-mooney	dansmith: well even with out numa restictions that would be required as the shader units cannot pysically acess the ram on another card. at least not without an explictit direct dma
15:48:55	mriedem	i guess i just need someone to tell me if https://review.openstack.org/599208 is the correct direction or not
15:48:55	dansmith	bauzas: right, N inventories per M providers for M cards, yes?
15:48:56	bauzas	where the total of that inventory being the aggregated amount of capacity for each card
15:48:58	mriedem	for existing inventory/allocations
15:49:11	bauzas	dansmith: one PCI ID only supporting one type
15:49:15	dansmith	sean-k-mooney: ack yeah, and the allocations are not fungible..
15:49:29	dansmith	bauzas: I think you're confusing what I'm asking
15:50:00	bauzas	probably
15:50:15	bauzas	:)
15:50:21	dansmith	sean-k-mooney: btw, my inventory of 4 crunchie bars has been 75% allocated, with 25% now free
15:50:47	bauzas	dansmith: lemme try to clarify my spec for multiple types
15:50:52	sean-k-mooney	dansmith: haha next time ill have to bring a larger consignment
15:51:00	bauzas	dansmith: the operator defines which types he wants
15:51:09	dansmith	sean-k-mooney: get your importer license
15:51:20	dansmith	bauzas: I don't think we're talking about your spec
15:51:25	bauzas	dansmith: for each type, he has to mention which PCI IDs will be supported
15:51:27	bauzas	oh
15:51:31	bauzas	then I'm confused :)
15:51:32	dansmith	I think we're talking about how to reshape from the controller (or not)
15:51:45	bauzas	mriedem's patch ?
15:51:52	bauzas	it's simple in my mind
15:51:59	bauzas	we only support one type
15:52:18	bauzas	reshaping will just create a child with the exact same inventory
15:52:27	bauzas	and move allocations against it

Earlier   Later