| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-01-23 | |||
| 18:11:25 | sean-k-mooney | normally we just generate a hostdev element like this | |
| 18:11:30 | sean-k-mooney | <hostdev mode='subsystem' type='pci' managed='yes'> | |
| 18:11:32 | sean-k-mooney | <source> | |
| 18:11:34 | sean-k-mooney | <address domain='0x0000' bus='0x06' slot='0x02' function='0x0'/> | |
| 18:11:36 | sean-k-mooney | </source> | |
| 18:11:38 | sean-k-mooney | <boot order='1'/> | |
| 18:11:40 | sean-k-mooney | <rom bar='on' file='/etc/fake/boot.bin'/> | |
| 18:11:42 | sean-k-mooney | </hostdev> | |
| 18:11:55 | sean-k-mooney | so we specify the source element and the host pci adress and libvirt chooes a target adress | |
| 18:12:07 | sean-k-mooney | that i assume is what you were changing with your manual edit | |
| 18:12:14 | deke997 | Yes | |
| 18:12:38 | deke997 | but I could only start that instance manually through libvirt after the edit, because nova just tries to overwrite it | |
| 18:13:39 | sean-k-mooney | https://libvirt.org/formatdomain.html#elementsAddress this is the docs for the device addressing | |
| 18:13:46 | deke997 | Also, adding multifunction = 'on' to the source address may be necessary. Libvirt docs say to use it, but I was able to have success either way | |
| 18:14:37 | deke997 | which would have to be done in nova | |
| 18:15:27 | deke997 | would be easy to just add multifunction = yes iff alias_type = multifunction or something | |
| 18:16:45 | sean-k-mooney | ya the multifunction bit is not what im worried about | |
| 18:17:14 | sean-k-mooney | i think the only way to make this work si to have nova set the target Adress on both devices | |
| 18:17:31 | sean-k-mooney | but since nova does not track the target adress today that is a prbalem | |
| 18:17:56 | sean-k-mooney | also you would have to take into account both q35 and the i440x machine types | |
| 18:18:14 | sean-k-mooney | the old i440x or pc machine type is pci only | |
| 18:18:32 | sean-k-mooney | and q35 is pcie by default but you can add pci busses too | |
| 18:18:47 | sean-k-mooney | so that makes it more complicated | |
| 18:19:47 | deke997 | definitely | |
| 18:20:41 | sean-k-mooney | with q35 if we wanted to set <address domain='0x0000' bus='0x06' slot='0x02' function='0x0'/> | |
| 18:20:59 | sean-k-mooney | for the target we would need to first create a pcie root port for it | |
| 18:21:16 | sean-k-mooney | if we leave out the target element libvirt does it for us | |
| 18:21:35 | sean-k-mooney | so baseicly when you specify it you also have to do the stuff libvirt does for you automatically | |
| 18:22:09 | sean-k-mooney | basically | |
| 18:22:13 | sean-k-mooney | devices> | |
| 18:22:15 | sean-k-mooney | <controller type='pci' index='0' model='pci-root'/> | |
| 18:22:17 | sean-k-mooney | <controller type='pci' index='1' model='pci-bridge'> | |
| 18:22:19 | sean-k-mooney | <address type='pci' domain='0' bus='0' slot='5' function='0' multifunction='off'/> | |
| 18:22:21 | sean-k-mooney | </controller> | |
| 18:22:23 | sean-k-mooney | </devices> | |
| 18:22:37 | sean-k-mooney | well it would be multifucniton=one in this case | |
| 18:22:52 | sean-k-mooney | but once we have the multi funtion bridge device you can add the enpoint beneth it | |
| 18:24:11 | sean-k-mooney | i guess the only real difference that mater for q35 vs pc is pcie-bridge vs pci-bridge in the contoler element | |
| 18:24:19 | deke997 | Okay so, haha, what if we let libvirt handle the target elements, but then before we boot, we have nova edit the slot and function numbers so that pairs match | |
| 18:24:31 | deke997 | :') | |
| 18:25:04 | sean-k-mooney | am first reaction i dont like that but thinking it though. | |
| 18:25:14 | sean-k-mooney | we could defien the domain without starting it | |
| 18:25:25 | deke997 | yes | |
| 18:25:27 | sean-k-mooney | that would cause libvirt to fill in the details i think | |
| 18:25:49 | sean-k-mooney | then we could read it back and ed the pair. maybe | |
| 18:26:00 | deke997 | all we'd have to do is check for two source lines that match and then make the targets match too | |
| 18:26:09 | sean-k-mooney | but it kind fo feels like a hack and it like wont scale well if there are multile deivecs | |
| 18:26:16 | deke997 | it is definitely a hack | |
| 18:26:31 | deke997 | why won't it scale well? | |
| 18:27:07 | deke997 | never gonna have more than ~8 devices in a server | |
| 18:27:11 | sean-k-mooney | if we had multile host deves we woudl get weired gaps in the pci adress space and if we had more then 8 virtual funciton it woudl get complcated | |
| 18:27:48 | sean-k-mooney | well the bit after the . is in octal so we can on ly have 8 fucntion per slot | |
| 18:28:01 | sean-k-mooney | so if you needed more then 8 virtual funtion it would not work right | |
| 18:28:17 | deke997 | I don't think such a device exists | |
| 18:28:30 | deke997 | haven't ever seen more than like 4 | |
| 18:28:34 | deke997 | my use case is only 2 | |
| 18:28:54 | sean-k-mooney | ya i have seen up to 8 well no i have seen way more | |
| 18:29:10 | sean-k-mooney | intel nics when using sriov can have 64 - 128 vf per pf | |
| 18:29:17 | sean-k-mooney | but they do the adressing slightly differently | |
| 18:29:20 | deke997 | sriov is totally different tho | |
| 18:29:37 | sean-k-mooney | ya for there case teh assing a full bus per card | |
| 18:29:58 | deke997 | this would be specifically for non-sriov multi-function devices | |
| 18:30:00 | efried | stephenfin: https://zuul.opendev.org/t/openstack/build/cc8e185b95cf4d5cbe973c4b81493084 is on rackcdn so I can't get to it. Can you see what the fail is? (ec2) | |
| 18:30:06 | sean-k-mooney | so you have mulitple slots and funtion to support up to 256 vf totall | |
| 18:30:25 | sean-k-mooney | deke997: ack | |
| 18:30:37 | sean-k-mooney | deke997: so you can see why i think this would need a spec | |
| 18:30:46 | sean-k-mooney | to design correctly | |
| 18:31:22 | deke997 | yes, I'm just hoping for an intermediate band aid while we work on a spec | |
| 18:31:26 | sean-k-mooney | i do see a use case for both multi funtion deivce passthough and the more generif i have a pair of device tha tmus be allocated together | |
| 18:31:57 | sean-k-mooney | e.g. the gpu and the audio contole on the gpu | |
| 18:32:49 | sean-k-mooney | nviad might actully be exposing it as a multi function device so that might actully be the same usecase | |
| 18:35:08 | sean-k-mooney | ya they are | |
| 18:35:13 | sean-k-mooney | 81:00.0 VGA compatible controller: NVIDIA Corporation GK104GL [Quadro K5000] (rev a1) | |
| 18:35:15 | sean-k-mooney | 81:00.1 Audio device: NVIDIA Corporation GK104 HDMI Audio Controller (rev a1) | |
| 18:35:44 | sean-k-mooney | so if we we made ^ work it would work for you too right | |
| 18:38:08 | deke997 | yes | |
| 18:38:27 | deke997 | That looks pretty much identical to my use case | |
| 18:42:09 | sean-k-mooney | ok what i would basicaly suggest is as follows. extent the alias with a child tag that can reference other alisas. | |
| 18:42:48 | sean-k-mooney | when you add an alias to a flaovr which has a child tag we woudl claim the deivce and all its childern | |
| 18:43:19 | sean-k-mooney | and expose them as a multi funcitn device | |
| 18:44:16 | sean-k-mooney | the pci pasthogh through filter would have to be enhanced to ensure it only passes host where the parent and child alias can be allocated form the same slot | |
| 18:44:51 | sean-k-mooney | the pci manger would also need to updated so that when we do the pci claim it claims the pair form the same slot | |
| 18:44:54 | deke997 | wouldn't that always be the case? | |
| 18:45:09 | deke997 | I don't see why the passthrough filter needs to be updated | |
| 18:45:21 | sean-k-mooney | well if you put two of those card in the same server it would break | |
| 18:46:07 | sean-k-mooney | as we coudl get 1 fucntion of type a form one card and one fucntion of type b from another | |
| 18:46:21 | sean-k-mooney | so the filter and assignment code would have to be hardened to prevent that | |
| 18:48:27 | deke997 | The assignment code, yes, but the filter, I don't think so | |
| 18:48:59 | deke997 | because before assignment, it would always be true that both the parent and child could be allocated together | |
| 18:49:31 | sean-k-mooney | its true if you alwasy need to request both | |
| 18:49:38 | deke997 | right | |
| 18:49:38 | sean-k-mooney | which is true in your case | |
| 18:49:42 | sean-k-mooney | not in the gpu case | |
| 18:49:47 | deke997 | I see | |
| 18:50:12 | deke997 | in that case, then yea the filter too | |
| 18:50:16 | sean-k-mooney | anyway the assignment code and filter share the same code | |
| 18:50:41 | sean-k-mooney | the filter basicaly passes if it would be able to assign | |
| 18:51:44 | sean-k-mooney | deke997: https://github.com/openstack/nova/blob/master/nova/scheduler/filters/pci_passthrough_filter.py#L51 | |
| 18:52:01 | sean-k-mooney | see we just loop over the request and ask can it suppor the request | |
| 18:52:20 | sean-k-mooney | well the loopin is don internally in the suport_request funtion | |