| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-11-02 | |||
| 13:56:25 | dansmith | sean-k-mooney: meaning in [mdev_nvidia-35]/this_takes_n_vfs=10 right? | |
| 13:56:29 | dansmith | that sounds good to me | |
| 13:56:30 | bauzas | the only way for an operator to know what a nvidia type is doing is by getting the /name value and match it with docs | |
| 13:56:37 | dansmith | default to 1 and if they don't fill it in, then the math will be wrong | |
| 13:56:38 | sean-k-mooney | dansmith: not quite | |
| 13:57:07 | bauzas | dansmith: no, sean-k-mooney is mentioning to explicitely define in nova.conf all the pci addresses of the VFs | |
| 13:57:09 | sean-k-mooney | dansmith: you would look at the mdev type and list n vfs | |
| 13:57:23 | bauzas | no new option | |
| 13:57:29 | dansmith | sean-k-mooney: isn't that what I just said? | |
| 13:57:30 | sean-k-mooney | we could do it diffently however | |
| 13:57:38 | bauzas | [devices] enabled_mdev_types = nvidia-35, nvidia-36 [mdev_nvidia-35] device_addresses = 0000:84:00.0,0000:85:00.0 [vgpu_nvidia-36] device_addresses = 0000:86:00.0 | |
| 13:57:53 | bauzas | you would specify then on nova.conf all the VFs | |
| 13:58:01 | bauzas | at least the ones you gonna allocate | |
| 13:58:31 | sean-k-mooney | dansmith: so we could do something better | |
| 13:58:51 | bauzas | sean-k-mooney: so sean, you gonna prefer to -1 my fix and say "please rather statically allocate all VFs" ? | |
| 13:59:04 | bauzas | https://review.opendev.org/c/openstack/nova/+/899625 | |
| 13:59:13 | dansmith | -1 to static, I can't imagine that's really going to make people happy | |
| 13:59:19 | bauzas | not sure our operators will appreciate | |
| 13:59:23 | bauzas | yeah | |
| 13:59:39 | sean-k-mooney | dansmith: what we could do is per mdev type declare how many that can have | |
| 13:59:49 | sean-k-mooney | and then specify that device_address shoudl be the PF | |
| 14:00:00 | bauzas | that's a risky math | |
| 14:00:08 | sean-k-mooney | so then we would have one device address (and placment inventoy) per phsyical gpu | |
| 14:00:17 | dansmith | I still don't understand how that's not what I said above and why that's not a good solution | |
| 14:00:21 | bauzas | because nothing in the sysfs structure is telling us which parent PF is the VF | |
| 14:00:31 | sean-k-mooney | and we would just not use aviable_isntance at all for the placment total | |
| 14:00:42 | sean-k-mooney | bauzas: it really should | |
| 14:00:49 | dansmith | suuuurely it does | |
| 14:00:55 | sean-k-mooney | that might be missing in libvirt but it will be in sysfs | |
| 14:00:55 | bauzas | sean-k-mooney: if you don't use available_instances, then you're asking people to precreate the mdevs | |
| 14:01:04 | sean-k-mooney | bauzas: no im not | |
| 14:01:11 | sean-k-mooney | im asking them to read the nvidia docs | |
| 14:01:14 | dansmith | I have to do another meeting for a bit | |
| 14:01:19 | sean-k-mooney | and tell us how many it can allcoate | |
| 14:01:44 | dansmith | surely we could also have a generic listing for the common types | |
| 14:01:53 | bauzas | (15:00:31) sean-k-mooney: and we would just not use aviable_isntance at all for the placment total | |
| 14:01:53 | bauzas | sean-k-mooney: I don't get then what you said | |
| 14:02:00 | bauzas | dansmith: that's like boiling the ocean | |
| 14:02:13 | bauzas | types are hardware-dependent | |
| 14:02:27 | sean-k-mooney | bauzas: im saying for the placment inventory we read total form the config option not form aviable_instnace | |
| 14:02:31 | dansmith | nvidia-472 differs in size on different cards? | |
| 14:02:37 | bauzas | https://docs.nvidia.com/grid/latest/grid-vgpu-user-guide/index.html#supported-gpus-grid-vgpu | |
| 14:02:49 | sean-k-mooney | dansmith: nvidia are creating a mdev type per card | |
| 14:02:59 | bauzas | dansmith: count every line in each tab, you'll get the number of mdev types | |
| 14:03:36 | dansmith | ack I understand | |
| 14:03:49 | bauzas | dansmith: no, the other way, nvidia-472 only stands for A100-20C which is only a type supported by A100 cards | |
| 14:03:59 | dansmith | yup, I got it | |
| 14:04:26 | sean-k-mooney | bauzas: lets park this for a bit and create an etherpad or something to collaberate on | |
| 14:04:51 | bauzas | formerly, before the magic world of SRIOV GPUs, available_instances was giving us the number of VGPUs a PCI device was able to create | |
| 14:04:54 | sean-k-mooney | its simpler to discuss when we have some of the constriat written down liek a spec | |
| 14:05:20 | bauzas | sean-k-mooney: there is already the bug report | |
| 14:05:28 | sean-k-mooney | i would like somethign we can edit | |
| 14:05:39 | bauzas | I'm not a big fan of writing a spec for fixing this | |
| 14:05:50 | sean-k-mooney | bauzas: i think you have too | |
| 14:05:55 | sean-k-mooney | depending on the fix | |
| 14:06:05 | sean-k-mooney | there are ways to do the minimal fix as a bug | |
| 14:06:20 | dansmith | I think it's clear we need to document this more than just some sysfs dumps in a bug :) | |
| 14:06:21 | bauzas | sean-k-mooney: have you seen my patch already ? | |
| 14:06:24 | sean-k-mooney | but if we want to fix this properly we need to change how we are doing mdev managment | |
| 14:06:42 | bauzas | as a reminder, mdev inventory isn't a problem with non-SRIOV GPUs | |
| 14:06:52 | bauzas | so whatever we decide will necessarly be opt-in | |
| 14:06:56 | sean-k-mooney | its not a problem for all sriov devices | |
| 14:07:06 | bauzas | and keeping the legacy behaviour too | |
| 14:07:08 | sean-k-mooney | its litrally aproblem just for how nvidia did it | |
| 14:07:34 | bauzas | "its litrally aproblem just for how nvidia did it" I should quote it, print it and put it on my front wall | |
| 14:08:17 | bauzas | the way that available_instances isn't a static number is also a terrible driver implementation, if you want MHO | |
| 14:08:18 | sean-k-mooney | before i left intel, i used a protoytp intel QAT impelemtnation wtch had sriov VF for each enging type and they supprot mdevs per VF with indepenent aviableity | |
| 14:08:31 | dansmith | is it really a problem though? I mean it kinda seems like this is the point of mdevs, no? to be allocatable in more dynamic ways than just static slices of a card? | |
| 14:09:09 | bauzas | true | |
| 14:09:10 | sean-k-mooney | dansmith: mevs from diffent pci endpoitn are intented to not affect each other | |
| 14:09:34 | sean-k-mooney | that is what nvida started breaking | |
| 14:09:50 | sean-k-mooney | the framework allows them to be dynmaic | |
| 14:09:58 | dansmith | there are multiple pci devices per card? or you mean the vfs? | |
| 14:10:14 | bauzas | oh wait | |
| 14:10:16 | bauzas | [stack@lenovo-sr655-01 ~]$ cat /sys/class/mdev_bus/0000\:41\:01.0/mdev_supported_types/nvidia-472/description | |
| 14:10:19 | bauzas | num_heads=1, frl_config=60, framebuffer=20480M, max_resolution=4096x2400, max_instance=2 | |
| 14:10:34 | bauzas | we can get the number of VGPUs in the description | |
| 14:10:38 | sean-k-mooney | bauzas: the desciptoin is not a standard format | |
| 14:10:43 | bauzas | this is horribly driver-specific | |
| 14:11:03 | sean-k-mooney | yes it is its litrally just a text filed | |
| 14:11:03 | bauzas | yeah | |
| 14:11:32 | dansmith | but it's something | |
| 14:11:43 | sean-k-mooney | its not somehting nova should read | |
| 14:12:00 | dansmith | it's something an operator could read in order to put the math in the config :) | |
| 14:12:24 | sean-k-mooney | grep . /sys/class/mdev_bus/0000\:41\:01.0/mdev_supported_types/nvidia-472/* | |
| 14:12:27 | bauzas | operators aldray read : | |
| 14:12:28 | bauzas | [stack@lenovo-sr655-01 ~]$ cat /sys/class/mdev_bus/0000\:41\:01.0/mdev_supported_types/nvidia-472/name | |
| 14:12:31 | bauzas | GRID A100-20C | |
| 14:12:32 | sean-k-mooney | bauzas: can you run ^ | |
| 14:12:41 | dansmith | I also wouldn't say nova *shouldn't* read it.. I think we could read and parse it as long as we don't fail if it has changed | |
| 14:12:42 | bauzas | and then Ctrl-F that string into https://docs.nvidia.com/grid/latest/grid-vgpu-user-guide/index.html#supported-gpus-grid-vgpu | |
| 14:13:22 | sean-k-mooney | dansmith: yes the operator can read it | |
| 14:13:39 | sean-k-mooney | dansmith: so would you be ok with adding max_instances to the mdev_type config section | |
| 14:14:01 | sean-k-mooney | or is that still too static | |
| 14:14:02 | bauzas | sean-k-mooney: if we go this way, this wouldn't be backportable, right? | |
| 14:14:10 | dansmith | well, I'm not sure that's a solution entirely, but I think making the operator tell us some of the math so we can do it better on the backend makes sense | |
| 14:14:30 | bauzas | I really tried to propose https://review.opendev.org/c/openstack/nova/+/899625 as a backportable mechanism | |
| 14:14:31 | dansmith | I'd need to hear the whole story before "being okay" with it, but that seems like a good direction to me | |
| 14:15:03 | bauzas | dansmith: finish your call and then we could do a gmeet | |
| 14:15:09 | dansmith | sure | |