| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-01-23 | |||
| 18:03:06 | sean-k-mooney | no | |
| 18:03:17 | openstackgerrit | Merged openstack/nova master: libvirt: Add a default VirtIO-RNG device to guests https://review.opendev.org/616603 | |
| 18:04:12 | sean-k-mooney | so there are three parts. the pci mager woudl have to be enhanced to pars and track paired deivces, the pci pasthough filter would have to be update and the libvirt driver xml generateion would have to be updated | |
| 18:05:22 | sean-k-mooney | the alias schema coudl certenly be extended to referen a child device https://github.com/openstack/nova/blob/master/nova/pci/request.py#L66-L99 | |
| 18:05:29 | deke997 | How would we go about updating the xml generation? | |
| 18:06:27 | sean-k-mooney | am ill see if i can find it. but likely i would see if we can still leave libvirt choose the target pci adders in the guest but mark a specicifc pci requiest as multi funcion | |
| 18:08:53 | deke997 | I need to know a bit more about how libvirt chooses those addresses | |
| 18:09:19 | sean-k-mooney | so you would have to modify https://github.com/openstack/nova/blob/6db486e9fd4f6b8dd02371b043e66808cdd1e0cd/nova/virt/libvirt/driver.py#L5871 | |
| 18:10:10 | sean-k-mooney | the general docs for thi in libvirt are https://libvirt.org/formatdomain.html#elementsHostDevSubsys | |
| 18:11:25 | sean-k-mooney | normally we just generate a hostdev element like this | |
| 18:11:30 | sean-k-mooney | <hostdev mode='subsystem' type='pci' managed='yes'> | |
| 18:11:32 | sean-k-mooney | <source> | |
| 18:11:34 | sean-k-mooney | <address domain='0x0000' bus='0x06' slot='0x02' function='0x0'/> | |
| 18:11:36 | sean-k-mooney | </source> | |
| 18:11:38 | sean-k-mooney | <boot order='1'/> | |
| 18:11:40 | sean-k-mooney | <rom bar='on' file='/etc/fake/boot.bin'/> | |
| 18:11:42 | sean-k-mooney | </hostdev> | |
| 18:11:55 | sean-k-mooney | so we specify the source element and the host pci adress and libvirt chooes a target adress | |
| 18:12:07 | sean-k-mooney | that i assume is what you were changing with your manual edit | |
| 18:12:14 | deke997 | Yes | |
| 18:12:38 | deke997 | but I could only start that instance manually through libvirt after the edit, because nova just tries to overwrite it | |
| 18:13:39 | sean-k-mooney | https://libvirt.org/formatdomain.html#elementsAddress this is the docs for the device addressing | |
| 18:13:46 | deke997 | Also, adding multifunction = 'on' to the source address may be necessary. Libvirt docs say to use it, but I was able to have success either way | |
| 18:14:37 | deke997 | which would have to be done in nova | |
| 18:15:27 | deke997 | would be easy to just add multifunction = yes iff alias_type = multifunction or something | |
| 18:16:45 | sean-k-mooney | ya the multifunction bit is not what im worried about | |
| 18:17:14 | sean-k-mooney | i think the only way to make this work si to have nova set the target Adress on both devices | |
| 18:17:31 | sean-k-mooney | but since nova does not track the target adress today that is a prbalem | |
| 18:17:56 | sean-k-mooney | also you would have to take into account both q35 and the i440x machine types | |
| 18:18:14 | sean-k-mooney | the old i440x or pc machine type is pci only | |
| 18:18:32 | sean-k-mooney | and q35 is pcie by default but you can add pci busses too | |
| 18:18:47 | sean-k-mooney | so that makes it more complicated | |
| 18:19:47 | deke997 | definitely | |
| 18:20:41 | sean-k-mooney | with q35 if we wanted to set <address domain='0x0000' bus='0x06' slot='0x02' function='0x0'/> | |
| 18:20:59 | sean-k-mooney | for the target we would need to first create a pcie root port for it | |
| 18:21:16 | sean-k-mooney | if we leave out the target element libvirt does it for us | |
| 18:21:35 | sean-k-mooney | so baseicly when you specify it you also have to do the stuff libvirt does for you automatically | |
| 18:22:09 | sean-k-mooney | basically | |
| 18:22:13 | sean-k-mooney | devices> | |
| 18:22:15 | sean-k-mooney | <controller type='pci' index='0' model='pci-root'/> | |
| 18:22:17 | sean-k-mooney | <controller type='pci' index='1' model='pci-bridge'> | |
| 18:22:19 | sean-k-mooney | <address type='pci' domain='0' bus='0' slot='5' function='0' multifunction='off'/> | |
| 18:22:21 | sean-k-mooney | </controller> | |
| 18:22:23 | sean-k-mooney | </devices> | |
| 18:22:37 | sean-k-mooney | well it would be multifucniton=one in this case | |
| 18:22:52 | sean-k-mooney | but once we have the multi funtion bridge device you can add the enpoint beneth it | |
| 18:24:11 | sean-k-mooney | i guess the only real difference that mater for q35 vs pc is pcie-bridge vs pci-bridge in the contoler element | |
| 18:24:19 | deke997 | Okay so, haha, what if we let libvirt handle the target elements, but then before we boot, we have nova edit the slot and function numbers so that pairs match | |
| 18:24:31 | deke997 | :') | |
| 18:25:04 | sean-k-mooney | am first reaction i dont like that but thinking it though. | |
| 18:25:14 | sean-k-mooney | we could defien the domain without starting it | |
| 18:25:25 | deke997 | yes | |
| 18:25:27 | sean-k-mooney | that would cause libvirt to fill in the details i think | |
| 18:25:49 | sean-k-mooney | then we could read it back and ed the pair. maybe | |
| 18:26:00 | deke997 | all we'd have to do is check for two source lines that match and then make the targets match too | |
| 18:26:09 | sean-k-mooney | but it kind fo feels like a hack and it like wont scale well if there are multile deivecs | |
| 18:26:16 | deke997 | it is definitely a hack | |
| 18:26:31 | deke997 | why won't it scale well? | |
| 18:27:07 | deke997 | never gonna have more than ~8 devices in a server | |
| 18:27:11 | sean-k-mooney | if we had multile host deves we woudl get weired gaps in the pci adress space and if we had more then 8 virtual funciton it woudl get complcated | |
| 18:27:48 | sean-k-mooney | well the bit after the . is in octal so we can on ly have 8 fucntion per slot | |
| 18:28:01 | sean-k-mooney | so if you needed more then 8 virtual funtion it would not work right | |
| 18:28:17 | deke997 | I don't think such a device exists | |
| 18:28:30 | deke997 | haven't ever seen more than like 4 | |
| 18:28:34 | deke997 | my use case is only 2 | |
| 18:28:54 | sean-k-mooney | ya i have seen up to 8 well no i have seen way more | |
| 18:29:10 | sean-k-mooney | intel nics when using sriov can have 64 - 128 vf per pf | |
| 18:29:17 | sean-k-mooney | but they do the adressing slightly differently | |
| 18:29:20 | deke997 | sriov is totally different tho | |
| 18:29:37 | sean-k-mooney | ya for there case teh assing a full bus per card | |
| 18:29:58 | deke997 | this would be specifically for non-sriov multi-function devices | |
| 18:30:00 | efried | stephenfin: https://zuul.opendev.org/t/openstack/build/cc8e185b95cf4d5cbe973c4b81493084 is on rackcdn so I can't get to it. Can you see what the fail is? (ec2) | |
| 18:30:06 | sean-k-mooney | so you have mulitple slots and funtion to support up to 256 vf totall | |
| 18:30:25 | sean-k-mooney | deke997: ack | |
| 18:30:37 | sean-k-mooney | deke997: so you can see why i think this would need a spec | |
| 18:30:46 | sean-k-mooney | to design correctly | |
| 18:31:22 | deke997 | yes, I'm just hoping for an intermediate band aid while we work on a spec | |
| 18:31:26 | sean-k-mooney | i do see a use case for both multi funtion deivce passthough and the more generif i have a pair of device tha tmus be allocated together | |
| 18:31:57 | sean-k-mooney | e.g. the gpu and the audio contole on the gpu | |
| 18:32:49 | sean-k-mooney | nviad might actully be exposing it as a multi function device so that might actully be the same usecase | |
| 18:35:08 | sean-k-mooney | ya they are | |
| 18:35:13 | sean-k-mooney | 81:00.0 VGA compatible controller: NVIDIA Corporation GK104GL [Quadro K5000] (rev a1) | |
| 18:35:15 | sean-k-mooney | 81:00.1 Audio device: NVIDIA Corporation GK104 HDMI Audio Controller (rev a1) | |
| 18:35:44 | sean-k-mooney | so if we we made ^ work it would work for you too right | |
| 18:38:08 | deke997 | yes | |
| 18:38:27 | deke997 | That looks pretty much identical to my use case | |
| 18:42:09 | sean-k-mooney | ok what i would basicaly suggest is as follows. extent the alias with a child tag that can reference other alisas. | |
| 18:42:48 | sean-k-mooney | when you add an alias to a flaovr which has a child tag we woudl claim the deivce and all its childern | |
| 18:43:19 | sean-k-mooney | and expose them as a multi funcitn device | |
| 18:44:16 | sean-k-mooney | the pci pasthogh through filter would have to be enhanced to ensure it only passes host where the parent and child alias can be allocated form the same slot | |
| 18:44:51 | sean-k-mooney | the pci manger would also need to updated so that when we do the pci claim it claims the pair form the same slot | |
| 18:44:54 | deke997 | wouldn't that always be the case? | |
| 18:45:09 | deke997 | I don't see why the passthrough filter needs to be updated | |
| 18:45:21 | sean-k-mooney | well if you put two of those card in the same server it would break | |
| 18:46:07 | sean-k-mooney | as we coudl get 1 fucntion of type a form one card and one fucntion of type b from another | |
| 18:46:21 | sean-k-mooney | so the filter and assignment code would have to be hardened to prevent that | |
| 18:48:27 | deke997 | The assignment code, yes, but the filter, I don't think so | |
| 18:48:59 | deke997 | because before assignment, it would always be true that both the parent and child could be allocated together | |
| 18:49:31 | sean-k-mooney | its true if you alwasy need to request both | |
| 18:49:38 | sean-k-mooney | which is true in your case | |