| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-11-02 | |||
| 13:46:54 | bauzas | now, with SR-IOV, instead of having one single pci device that have available_instances = 2 (and then only one RP), | |
| 13:47:11 | bauzas | then we have *two* VFs, each of them having available_instances = 1 | |
| 13:47:15 | bauzas | but,n | |
| 13:47:37 | bauzas | since this is type-dependent, nvidia doesn't create only 2 VFs but 16 | |
| 13:47:44 | bauzas | (for that single GPU) | |
| 13:47:57 | sean-k-mooney | yes | |
| 13:48:01 | bauzas | (if you have A2, IIRC this will create 32 virtual functions) | |
| 13:48:16 | dansmith | can someone answer the question: that's a function of nvidia-472 right? | |
| 13:48:26 | bauzas | so, say you define nvidia-472, this means that only 2 virtual functions have to be used | |
| 13:48:36 | sean-k-mooney | dansmith: the VF can support many mdev types | |
| 13:48:45 | dansmith | and if they are slicing differently with (nvidia-xxx right)? then the math would be different/ | |
| 13:48:55 | bauzas | if you use other type (say nvidia-473), then you could use *four* virtual functions | |
| 13:49:02 | bauzas | dansmith: correct | |
| 13:49:03 | dansmith | yeah | |
| 13:49:09 | sean-k-mooney | dansmith: yep the math change depenign on which mdev type you create | |
| 13:49:15 | sean-k-mooney | but the number of VF is fixed | |
| 13:49:17 | bauzas | the fact is | |
| 13:49:23 | sean-k-mooney | so not all of them can actully be allcoated | |
| 13:49:24 | bauzas | given the type, | |
| 13:49:29 | sean-k-mooney | for all types | |
| 13:49:39 | bauzas | once you create mdevs for all the VFs corresponding to the type, | |
| 13:49:51 | dansmith | right, and could it be some combination of two types? or only one type of slicing per card? | |
| 13:49:52 | bauzas | then other VFs start magically having available_instances=0 | |
| 13:50:00 | sean-k-mooney | dansmith: that depend on the card | |
| 13:50:08 | bauzas | you can't mix types per GPU, nvidia doesn't support it | |
| 13:50:12 | bauzas | well, | |
| 13:50:15 | sean-k-mooney | actully it does | |
| 13:50:16 | bauzas | for time-sliced GPUs | |
| 13:50:31 | bauzas | for MIG GPUs, you can, but that's a waaaaay different matter | |
| 13:50:35 | sean-k-mooney | but only in s specific mode for sepcific type on specific cards | |
| 13:50:40 | sean-k-mooney | yep | |
| 13:50:41 | bauzas | let's not overcomplicate this | |
| 13:50:51 | sean-k-mooney | lol | |
| 13:50:58 | sean-k-mooney | do you have a time machine | |
| 13:50:58 | dansmith | well, my point is, | |
| 13:51:20 | dansmith | it seems really uncool to expose all these single-inventory RPs in general | |
| 13:51:31 | dansmith | like it feels as though we modeled it wrong | |
| 13:51:51 | sean-k-mooney | ya so we shoudl be grouping per PF | |
| 13:52:09 | sean-k-mooney | as each phsical devce shoudl be its own RP | |
| 13:52:22 | sean-k-mooney | but since we never actully added offical supprot for sriov mode | |
| 13:52:30 | sean-k-mooney | what we did instead was tell people | |
| 13:52:33 | bauzas | dansmith: the problem is that we don't know preemptively which type consumes how many VFs | |
| 13:52:38 | sean-k-mooney | jsut use the pci adress of the vf instead of hte pf | |
| 13:52:57 | bauzas | so the simpliest solution is to tell people to statically allocate mdevs | |
| 13:53:01 | sean-k-mooney | bauzas: thats not really the probelm but it is one of the issues | |
| 13:53:17 | bauzas | sean-k-mooney: that *IS* the problem in our case | |
| 13:53:22 | dansmith | bauzas: yeah, I think there are ways we could handle that, but yeah it's not ideal for sure | |
| 13:53:32 | dansmith | unless we make people annotate the config in that way | |
| 13:53:45 | sean-k-mooney | well we agreed in the ptg to make them do that ^ | |
| 13:53:45 | bauzas | if I was able to get a clue of how many VFs are available for a type, I could prevent nova to create RPs for the 16 of them | |
| 13:54:02 | sean-k-mooney | we agreed to make device_adress required | |
| 13:54:10 | bauzas | sean-k-mooney: no we said at the PTG that allocating a mdev is cheap | |
| 13:54:14 | bauzas | so we could continue to do this | |
| 13:54:24 | sean-k-mooney | bauzas: yes we can that not what i said above | |
| 13:54:41 | sean-k-mooney | we agreed that going forward then need to specificy which devices we create inventoies form | |
| 13:54:46 | sean-k-mooney | by requiring the device_adress | |
| 13:54:47 | bauzas | sean-k-mooney: sure | |
| 13:54:58 | sean-k-mooney | nova can continue to od the mdev creation | |
| 13:54:59 | bauzas | but this doesn't prevent operator fat fingers, right? | |
| 13:55:09 | dansmith | sean-k-mooney: agreed to make who do what? | |
| 13:55:11 | sean-k-mooney | the key is just not list more vf then can be created | |
| 13:55:44 | sean-k-mooney | dansmith: https://docs.openstack.org/nova/latest/configuration/config.html#devices | |
| 13:55:45 | bauzas | sean-k-mooney: also, with SR-IOV, this will be hard for operators to know which VFs to allocate btw. | |
| 13:56:01 | bauzas | the fucking nvidia docs never tell about nvidia-XXX | |
| 13:56:02 | sean-k-mooney | for legacy reasons (only for the first mdev type) specifiv device_address is optional | |
| 13:56:25 | dansmith | sean-k-mooney: meaning in [mdev_nvidia-35]/this_takes_n_vfs=10 right? | |
| 13:56:29 | dansmith | that sounds good to me | |
| 13:56:30 | bauzas | the only way for an operator to know what a nvidia type is doing is by getting the /name value and match it with docs | |
| 13:56:37 | dansmith | default to 1 and if they don't fill it in, then the math will be wrong | |
| 13:56:38 | sean-k-mooney | dansmith: not quite | |
| 13:57:07 | bauzas | dansmith: no, sean-k-mooney is mentioning to explicitely define in nova.conf all the pci addresses of the VFs | |
| 13:57:09 | sean-k-mooney | dansmith: you would look at the mdev type and list n vfs | |
| 13:57:23 | bauzas | no new option | |
| 13:57:29 | dansmith | sean-k-mooney: isn't that what I just said? | |
| 13:57:30 | sean-k-mooney | we could do it diffently however | |
| 13:57:38 | bauzas | [devices] enabled_mdev_types = nvidia-35, nvidia-36 [mdev_nvidia-35] device_addresses = 0000:84:00.0,0000:85:00.0 [vgpu_nvidia-36] device_addresses = 0000:86:00.0 | |
| 13:57:53 | bauzas | you would specify then on nova.conf all the VFs | |
| 13:58:01 | bauzas | at least the ones you gonna allocate | |
| 13:58:31 | sean-k-mooney | dansmith: so we could do something better | |
| 13:58:51 | bauzas | sean-k-mooney: so sean, you gonna prefer to -1 my fix and say "please rather statically allocate all VFs" ? | |
| 13:59:04 | bauzas | https://review.opendev.org/c/openstack/nova/+/899625 | |
| 13:59:13 | dansmith | -1 to static, I can't imagine that's really going to make people happy | |
| 13:59:19 | bauzas | not sure our operators will appreciate | |
| 13:59:23 | bauzas | yeah | |
| 13:59:39 | sean-k-mooney | dansmith: what we could do is per mdev type declare how many that can have | |
| 13:59:49 | sean-k-mooney | and then specify that device_address shoudl be the PF | |
| 14:00:00 | bauzas | that's a risky math | |
| 14:00:08 | sean-k-mooney | so then we would have one device address (and placment inventoy) per phsyical gpu | |
| 14:00:17 | dansmith | I still don't understand how that's not what I said above and why that's not a good solution | |
| 14:00:21 | bauzas | because nothing in the sysfs structure is telling us which parent PF is the VF | |
| 14:00:31 | sean-k-mooney | and we would just not use aviable_isntance at all for the placment total | |
| 14:00:42 | sean-k-mooney | bauzas: it really should | |
| 14:00:49 | dansmith | suuuurely it does | |
| 14:00:55 | bauzas | sean-k-mooney: if you don't use available_instances, then you're asking people to precreate the mdevs | |
| 14:00:55 | sean-k-mooney | that might be missing in libvirt but it will be in sysfs | |
| 14:01:04 | sean-k-mooney | bauzas: no im not | |
| 14:01:11 | sean-k-mooney | im asking them to read the nvidia docs | |
| 14:01:14 | dansmith | I have to do another meeting for a bit | |
| 14:01:19 | sean-k-mooney | and tell us how many it can allcoate | |
| 14:01:44 | dansmith | surely we could also have a generic listing for the common types | |