| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-11-02 | |||
| 13:36:58 | bauzas | thanks for having started to look at it, btw. | |
| 13:37:12 | bauzas | (me also needs to look at your spec) | |
| 13:37:20 | dansmith | yeah sorry I forgot you were off yesterday until after I pinged | |
| 13:37:21 | bauzas | so, shoot your questions | |
| 13:37:31 | bauzas | have you seen the bug report btw.? | |
| 13:37:43 | bauzas | I tried to summarize the problem in there | |
| 13:37:48 | dansmith | my questions are actually *about* the stuff in the bug :) | |
| 13:37:52 | bauzas | https://bugs.launchpad.net/nova/+bug/2041519 | |
| 13:37:58 | bauzas | okay | |
| 13:38:10 | bauzas | and why I did that, I guess ? | |
| 13:38:23 | dansmith | so I get the inventory before, then after one allocation, one RP goes to zero | |
| 13:38:37 | bauzas | more than one RP but yeah | |
| 13:38:48 | dansmith | no, only one according to your report | |
| 13:38:57 | bauzas | and this is not exacly *always* after | |
| 13:39:23 | bauzas | the fact is that we automatically create a mdev if none of them are not used yet | |
| 13:39:25 | dansmith | but what i don't get is why after the second allocation they *all* go to zero.. that would imply we're using inventory wrong | |
| 13:39:43 | bauzas | okay, gonna clarify | |
| 13:39:56 | bauzas | we are not counting mdevs for inventories | |
| 13:40:31 | bauzas | what we count as VGPU resources are *either unused mdevs *or* available_instances* | |
| 13:40:44 | dansmith | oh, hang on | |
| 13:40:46 | bauzas | which means something you can allocate to a guest | |
| 13:41:08 | dansmith | re-reading this with fresh eyes, I think I was confusing the very similar sysfs and placement dumps | |
| 13:41:25 | dansmith | so it's the mdev availability that goes off a cliff | |
| 13:42:29 | bauzas | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L8090-L8133 | |
| 13:42:38 | bauzas | that's where we calculate inventories | |
| 13:42:53 | dansmith | yep | |
| 13:43:08 | bauzas | available_instances is a value that the kernel driver gives | |
| 13:43:12 | bauzas | but, | |
| 13:43:13 | dansmith | right | |
| 13:43:24 | bauzas | once you create a mdev, the value decreases by 1 | |
| 13:43:33 | bauzas | that doesn't mean that the mdev will be used by a guest | |
| 13:43:53 | sean-k-mooney | bauzas: yep shich is why you are ment to be internally tracking which devices the mdev are created form | |
| 13:43:56 | bauzas | if you basically precreate all mdev resources, then available_instances is 0 for all the gpus | |
| 13:44:11 | sean-k-mooney | and using that to compute the inventoy total as (aviablie_instnace + mdevs form that device) | |
| 13:44:16 | dansmith | does nvidia-472 imply that more than 1/total share of the card is used? | |
| 13:44:28 | bauzas | but the inventory still tracks that you can bind N vGPUs | |
| 13:44:49 | bauzas | dansmith: that's the crux of the problem | |
| 13:45:01 | bauzas | the number of vGPUs can differ per type | |
| 13:45:04 | sean-k-mooney | bauzas: that problem is sovled by enforcing that the device adress | |
| 13:45:10 | dansmith | that feels like we're using placement incorrectly to have inventory of unequal resources | |
| 13:45:15 | bauzas | nvidia-472 can create up to two vGPUs | |
| 13:45:32 | bauzas | so there are three possibilities | |
| 13:45:41 | bauzas | either available_instances = 2 | |
| 13:45:56 | bauzas | or, available_instances = 1 and one mdev exists that'not used by a guest | |
| 13:46:06 | bauzas | or, there are two mdevs available, which are both unused | |
| 13:46:17 | bauzas | with SR-IOV, things change | |
| 13:46:19 | sean-k-mooney | bauzas: the decreasing inventory behavoir is a nvidia specirci issue with vfs form teh same card | |
| 13:46:54 | bauzas | now, with SR-IOV, instead of having one single pci device that have available_instances = 2 (and then only one RP), | |
| 13:47:11 | bauzas | then we have *two* VFs, each of them having available_instances = 1 | |
| 13:47:15 | bauzas | but,n | |
| 13:47:37 | bauzas | since this is type-dependent, nvidia doesn't create only 2 VFs but 16 | |
| 13:47:44 | bauzas | (for that single GPU) | |
| 13:47:57 | sean-k-mooney | yes | |
| 13:48:01 | bauzas | (if you have A2, IIRC this will create 32 virtual functions) | |
| 13:48:16 | dansmith | can someone answer the question: that's a function of nvidia-472 right? | |
| 13:48:26 | bauzas | so, say you define nvidia-472, this means that only 2 virtual functions have to be used | |
| 13:48:36 | sean-k-mooney | dansmith: the VF can support many mdev types | |
| 13:48:45 | dansmith | and if they are slicing differently with (nvidia-xxx right)? then the math would be different/ | |
| 13:48:55 | bauzas | if you use other type (say nvidia-473), then you could use *four* virtual functions | |
| 13:49:02 | bauzas | dansmith: correct | |
| 13:49:03 | dansmith | yeah | |
| 13:49:09 | sean-k-mooney | dansmith: yep the math change depenign on which mdev type you create | |
| 13:49:15 | sean-k-mooney | but the number of VF is fixed | |
| 13:49:17 | bauzas | the fact is | |
| 13:49:23 | sean-k-mooney | so not all of them can actully be allcoated | |
| 13:49:24 | bauzas | given the type, | |
| 13:49:29 | sean-k-mooney | for all types | |
| 13:49:39 | bauzas | once you create mdevs for all the VFs corresponding to the type, | |
| 13:49:51 | dansmith | right, and could it be some combination of two types? or only one type of slicing per card? | |
| 13:49:52 | bauzas | then other VFs start magically having available_instances=0 | |
| 13:50:00 | sean-k-mooney | dansmith: that depend on the card | |
| 13:50:08 | bauzas | you can't mix types per GPU, nvidia doesn't support it | |
| 13:50:12 | bauzas | well, | |
| 13:50:15 | sean-k-mooney | actully it does | |
| 13:50:16 | bauzas | for time-sliced GPUs | |
| 13:50:31 | bauzas | for MIG GPUs, you can, but that's a waaaaay different matter | |
| 13:50:35 | sean-k-mooney | but only in s specific mode for sepcific type on specific cards | |
| 13:50:40 | sean-k-mooney | yep | |
| 13:50:41 | bauzas | let's not overcomplicate this | |
| 13:50:51 | sean-k-mooney | lol | |
| 13:50:58 | sean-k-mooney | do you have a time machine | |
| 13:50:58 | dansmith | well, my point is, | |
| 13:51:20 | dansmith | it seems really uncool to expose all these single-inventory RPs in general | |
| 13:51:31 | dansmith | like it feels as though we modeled it wrong | |
| 13:51:51 | sean-k-mooney | ya so we shoudl be grouping per PF | |
| 13:52:09 | sean-k-mooney | as each phsical devce shoudl be its own RP | |
| 13:52:22 | sean-k-mooney | but since we never actully added offical supprot for sriov mode | |
| 13:52:30 | sean-k-mooney | what we did instead was tell people | |
| 13:52:33 | bauzas | dansmith: the problem is that we don't know preemptively which type consumes how many VFs | |
| 13:52:38 | sean-k-mooney | jsut use the pci adress of the vf instead of hte pf | |
| 13:52:57 | bauzas | so the simpliest solution is to tell people to statically allocate mdevs | |
| 13:53:01 | sean-k-mooney | bauzas: thats not really the probelm but it is one of the issues | |
| 13:53:17 | bauzas | sean-k-mooney: that *IS* the problem in our case | |
| 13:53:22 | dansmith | bauzas: yeah, I think there are ways we could handle that, but yeah it's not ideal for sure | |
| 13:53:32 | dansmith | unless we make people annotate the config in that way | |
| 13:53:45 | sean-k-mooney | well we agreed in the ptg to make them do that ^ | |
| 13:53:45 | bauzas | if I was able to get a clue of how many VFs are available for a type, I could prevent nova to create RPs for the 16 of them | |
| 13:54:02 | sean-k-mooney | we agreed to make device_adress required | |
| 13:54:10 | bauzas | sean-k-mooney: no we said at the PTG that allocating a mdev is cheap | |
| 13:54:14 | bauzas | so we could continue to do this | |