| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-09-17 | |||
| 15:30:26 | bauzas | jaypipes: but GPU capabilities will have to be described by operators directly | |
| 15:31:56 | jaypipes | bauzas: what is the user asking for? I've only seen requests for things like "my application is built with CUDA library X and therefore can take advantage of NVIDIA Compute Capability Y hardware, so make sure I get on a GPU that has that Compute Capability". | |
| 15:32:26 | bauzas | yeah probably | |
| 15:32:41 | bauzas | jaypipes: that's where your idea of an YAML inventory could help | |
| 15:33:05 | bauzas | but for the moment, I'm not happy with nova supporting vendor-specific features by code | |
| 15:33:24 | bauzas | anyway, speaking of code is better with code | |
| 15:36:36 | openstackgerrit | Merged openstack/nova-specs master: Add support specify volume type when boot instance https://review.openstack.org/579520 | |
| 15:38:00 | mriedem | libvirt reshaper patch updated | |
| 15:38:08 | openstackgerrit | Matt Riedemann proposed openstack/nova master: libvirt: implement reshaper for vgpu https://review.openstack.org/599208 | |
| 15:38:13 | mriedem | i'm pondering how useful a pre-upgrade check could be for this | |
| 15:38:30 | mriedem | as in, i'm not sure how useful it would be since it's not something you can run offline | |
| 15:39:01 | mriedem | a pre-upgrade check would mostly be for reporting - yes you have x number of providers of these things and you're going to need to upgrade those via reshaper | |
| 15:40:16 | mriedem | although...why can't we run this offline? if the vgpu inventory is already on the root compute node provider and has allocations, we can call the reshaper API - the only thing we'd need is the gpu type and we can get that from config | |
| 15:40:26 | mriedem | lyarwood: dansmith: ^ | |
| 15:40:52 | dansmith | each compute node can have different types, right? | |
| 15:41:09 | dansmith | we need the quantity from the virt driver at least | |
| 15:41:15 | dansmith | we know which types, but not how many of each | |
| 15:41:20 | mriedem | you'd have to run the data migration per compute yes | |
| 15:41:20 | dansmith | right now we expose N of one type, IIRC | |
| 15:41:29 | mriedem | like the ironic instance flavor one | |
| 15:41:37 | dansmith | but you still need data from the compute, not just the types the compute is going to use, AFAIK | |
| 15:41:45 | mriedem | isn't that already in placement? | |
| 15:41:50 | mriedem | via the inventory record? | |
| 15:42:07 | dansmith | for one type, but not the others | |
| 15:42:10 | dansmith | bauzas: right? | |
| 15:42:34 | mriedem | we don't support multiple types yet | |
| 15:42:44 | mriedem | so one vgpu inventory record at most per compute node provider | |
| 15:42:46 | dansmith | right, but this is to get us there right? | |
| 15:42:51 | mriedem | yes | |
| 15:42:52 | dansmith | or your idea is to reshape just the one inventory, | |
| 15:42:55 | mriedem | yes | |
| 15:43:00 | mriedem | that's what my patch above does | |
| 15:43:01 | dansmith | and then let the compute fill it itn? | |
| 15:43:27 | dansmith | so, that might work, | |
| 15:43:35 | sean-k-mooney | dansmith: one of the issues with vgpus is that while you can have multilpel vgpus on the same pcpu like vm flavors dependng on what you have already created you may not be able to create others since they all share the same resouces | |
| 15:43:51 | dansmith | yeah, | |
| 15:44:07 | dansmith | so can't we have created some vgpus for guests on different cards that now need to be represented separately but weren't before? | |
| 15:44:15 | dansmith | and without a view of the virt driver, that might not be doable | |
| 15:45:08 | sean-k-mooney | dansmith: perhaps i belive today bauzas has confied the compute agent to only createing 1 vgpu type per host so it might not be be an issue | |
| 15:45:16 | bauzas | dansmith: you can get the supported type by looking up the conf optionj | |
| 15:45:56 | dansmith | type per host | |
| 15:45:57 | bauzas | sean-k-mooney: that's correct, we only support one type | |
| 15:45:58 | mriedem | if you changed the configured support vgpu type after creating some inventory/allocations but before reshaping, then we'd have a problem | |
| 15:46:09 | bauzas | mriedem: yeah that's a limitation | |
| 15:46:13 | dansmith | bauzas: one type per host, but potentially across multiple cards right? | |
| 15:46:20 | bauzas | dansmith: correct | |
| 15:46:27 | bauzas | libvirt only gives the total | |
| 15:46:30 | mriedem | isn't the multiple cards just the total? | |
| 15:46:30 | mriedem | yeah | |
| 15:46:33 | sean-k-mooney | bauzas: you are proposing a whitelist model where the operator figures out what are compatible ahead of time rather then dynamicaly handeling this in nova/cyborg. that is deffinetly a step forward but may not cover all usecases well | |
| 15:46:33 | dansmith | so can we know externally which instance should go to which new (separate) card provider? | |
| 15:46:38 | bauzas | it can be sharded across cards | |
| 15:46:50 | mriedem | tbc, | |
| 15:46:56 | dansmith | presumably post-reshape, each allocation is against a single card for affinity reasons, right? | |
| 15:46:57 | mriedem | my patch isn't creating multiple vgpu providers | |
| 15:47:04 | dansmith | since they will eventually be tied to numa nodes | |
| 15:47:24 | bauzas | dansmith: that's right | |
| 15:47:43 | bauzas | dansmith: in my spec, I'm proposing the PCI ID as a way to know which card is for which type | |
| 15:47:56 | mriedem | so we're going to have 1 vgpu child provider per card with a total inventory of 1? | |
| 15:48:02 | bauzas | dansmith: what is uncovered in my spec is how an user can ask a certain type | |
| 15:48:04 | dansmith | mriedem: no, | |
| 15:48:14 | dansmith | mriedem: one card with however many vgpus it can support, right? | |
| 15:48:34 | dansmith | one card per provider I mean | |
| 15:48:36 | bauzas | mriedem: dansmith: no, it will create N children inventories, one per type | |
| 15:48:47 | sean-k-mooney | dansmith: well even with out numa restictions that would be required as the shader units cannot pysically acess the ram on another card. at least not without an explictit direct dma | |
| 15:48:55 | mriedem | i guess i just need someone to tell me if https://review.openstack.org/599208 is the correct direction or not | |
| 15:48:55 | dansmith | bauzas: right, N inventories per M providers for M cards, yes? | |
| 15:48:56 | bauzas | where the total of that inventory being the aggregated amount of capacity for each card | |
| 15:48:58 | mriedem | for existing inventory/allocations | |
| 15:49:11 | bauzas | dansmith: one PCI ID only supporting one type | |
| 15:49:15 | dansmith | sean-k-mooney: ack yeah, and the allocations are not fungible.. | |
| 15:49:29 | dansmith | bauzas: I think you're confusing what I'm asking | |
| 15:50:00 | bauzas | probably | |
| 15:50:15 | bauzas | :) | |
| 15:50:21 | dansmith | sean-k-mooney: btw, my inventory of 4 crunchie bars has been 75% allocated, with 25% now free | |
| 15:50:47 | bauzas | dansmith: lemme try to clarify my spec for multiple types | |
| 15:50:52 | sean-k-mooney | dansmith: haha next time ill have to bring a larger consignment | |
| 15:51:00 | bauzas | dansmith: the operator defines which types he wants | |
| 15:51:09 | dansmith | sean-k-mooney: get your importer license | |
| 15:51:20 | dansmith | bauzas: I don't think we're talking about your spec | |
| 15:51:25 | bauzas | dansmith: for each type, he has to mention which PCI IDs will be supported | |
| 15:51:27 | bauzas | oh | |
| 15:51:31 | bauzas | then I'm confused :) | |
| 15:51:32 | dansmith | I think we're talking about how to reshape from the controller (or not) | |
| 15:51:45 | bauzas | mriedem's patch ? | |
| 15:51:52 | bauzas | it's simple in my mind | |
| 15:51:59 | bauzas | we only support one type | |
| 15:52:18 | bauzas | reshaping will just create a child with the exact same inventory | |
| 15:52:27 | bauzas | and move allocations against it | |
| 15:52:40 | sean-k-mooney | bauzas: do we today supprot 2+ pgpus on a host | |
| 15:52:58 | bauzas | sean-k-mooney: yup, but they are hidden for libvirt | |
| 15:53:04 | mriedem | bauzas: and that's what i've implemented | |
| 15:53:10 | bauzas | mriedem: I know | |
| 15:53:13 | bauzas | hence my +1 | |
| 15:53:17 | bauzas | so i don't get the concerns | |
| 15:53:27 | bauzas | having multiple cards isn't a problem | |
| 15:53:40 | bauzas | or I'm missing a crucial point | |
| 15:53:52 | bauzas | I mean, I can be wrong but I don't get the problem | |
| 15:54:00 | dansmith | bauzas: presumably we want to reshape inventories onto providers that represent actual cards, right? | |
| 15:54:09 | bauzas | dansmith: no | |