| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-01-23 | |||
| 17:57:12 | sean-k-mooney | so you have only 1 device on the compute node and your booting 1 vm that use all the fucntion form that device | |
| 17:57:27 | deke997 | yes that's what I have set up for testing | |
| 17:57:42 | sean-k-mooney | ya in that specific case you whave worked around that issue | |
| 17:57:49 | deke997 | But in the libvirt XML it is very easy to see which devices from the host were passed through | |
| 17:57:58 | deke997 | and match them | |
| 17:58:02 | sean-k-mooney | this has come up in the context of gpus in the past | |
| 17:58:21 | sean-k-mooney | the GPU and audio contoler are usally 2 differnt pci endpoint on the same device | |
| 17:58:48 | sean-k-mooney | at pressent we dont have a way to passthough both form the same card and ensure they are form the same device | |
| 17:59:51 | sean-k-mooney | deke997: form a nova point of view you would basically want ot have an alsi that can refernce a second alias and request both as a pair | |
| 18:00:08 | deke997 | yea exactly | |
| 18:00:19 | deke997 | That doesn't seem super difficult | |
| 18:00:43 | sean-k-mooney | and then in nova/libvirt have it select the pair form the same device an make them a multifunction device in the xml if they are expose that way on the host | |
| 18:01:04 | sean-k-mooney | deke997: you say that now | |
| 18:01:08 | deke997 | haha | |
| 18:01:43 | deke997 | Is there a way to make nova play nice with edited libvirt.xmls? | |
| 18:01:44 | sean-k-mooney | so this all live in the pci manger code https://github.com/openstack/nova/tree/master/nova/pci | |
| 18:01:52 | sean-k-mooney | no | |
| 18:02:18 | sean-k-mooney | nova regenerates the xml based on seveal api actions | |
| 18:02:22 | sean-k-mooney | like hard reboot | |
| 18:02:36 | sean-k-mooney | so your eddit would last until someone stops and start the vm | |
| 18:02:56 | deke997 | Is there a way to disable that regeneration? | |
| 18:03:06 | sean-k-mooney | no | |
| 18:03:17 | openstackgerrit | Merged openstack/nova master: libvirt: Add a default VirtIO-RNG device to guests https://review.opendev.org/616603 | |
| 18:04:12 | sean-k-mooney | so there are three parts. the pci mager woudl have to be enhanced to pars and track paired deivces, the pci pasthough filter would have to be update and the libvirt driver xml generateion would have to be updated | |
| 18:05:22 | sean-k-mooney | the alias schema coudl certenly be extended to referen a child device https://github.com/openstack/nova/blob/master/nova/pci/request.py#L66-L99 | |
| 18:05:29 | deke997 | How would we go about updating the xml generation? | |
| 18:06:27 | sean-k-mooney | am ill see if i can find it. but likely i would see if we can still leave libvirt choose the target pci adders in the guest but mark a specicifc pci requiest as multi funcion | |
| 18:08:53 | deke997 | I need to know a bit more about how libvirt chooses those addresses | |
| 18:09:19 | sean-k-mooney | so you would have to modify https://github.com/openstack/nova/blob/6db486e9fd4f6b8dd02371b043e66808cdd1e0cd/nova/virt/libvirt/driver.py#L5871 | |
| 18:10:10 | sean-k-mooney | the general docs for thi in libvirt are https://libvirt.org/formatdomain.html#elementsHostDevSubsys | |
| 18:11:25 | sean-k-mooney | normally we just generate a hostdev element like this | |
| 18:11:30 | sean-k-mooney | <hostdev mode='subsystem' type='pci' managed='yes'> | |
| 18:11:32 | sean-k-mooney | <source> | |
| 18:11:34 | sean-k-mooney | <address domain='0x0000' bus='0x06' slot='0x02' function='0x0'/> | |
| 18:11:36 | sean-k-mooney | </source> | |
| 18:11:38 | sean-k-mooney | <boot order='1'/> | |
| 18:11:40 | sean-k-mooney | <rom bar='on' file='/etc/fake/boot.bin'/> | |
| 18:11:42 | sean-k-mooney | </hostdev> | |
| 18:11:55 | sean-k-mooney | so we specify the source element and the host pci adress and libvirt chooes a target adress | |
| 18:12:07 | sean-k-mooney | that i assume is what you were changing with your manual edit | |
| 18:12:14 | deke997 | Yes | |
| 18:12:38 | deke997 | but I could only start that instance manually through libvirt after the edit, because nova just tries to overwrite it | |
| 18:13:39 | sean-k-mooney | https://libvirt.org/formatdomain.html#elementsAddress this is the docs for the device addressing | |
| 18:13:46 | deke997 | Also, adding multifunction = 'on' to the source address may be necessary. Libvirt docs say to use it, but I was able to have success either way | |
| 18:14:37 | deke997 | which would have to be done in nova | |
| 18:15:27 | deke997 | would be easy to just add multifunction = yes iff alias_type = multifunction or something | |
| 18:16:45 | sean-k-mooney | ya the multifunction bit is not what im worried about | |
| 18:17:14 | sean-k-mooney | i think the only way to make this work si to have nova set the target Adress on both devices | |
| 18:17:31 | sean-k-mooney | but since nova does not track the target adress today that is a prbalem | |
| 18:17:56 | sean-k-mooney | also you would have to take into account both q35 and the i440x machine types | |
| 18:18:14 | sean-k-mooney | the old i440x or pc machine type is pci only | |
| 18:18:32 | sean-k-mooney | and q35 is pcie by default but you can add pci busses too | |
| 18:18:47 | sean-k-mooney | so that makes it more complicated | |
| 18:19:47 | deke997 | definitely | |
| 18:20:41 | sean-k-mooney | with q35 if we wanted to set <address domain='0x0000' bus='0x06' slot='0x02' function='0x0'/> | |
| 18:20:59 | sean-k-mooney | for the target we would need to first create a pcie root port for it | |
| 18:21:16 | sean-k-mooney | if we leave out the target element libvirt does it for us | |
| 18:21:35 | sean-k-mooney | so baseicly when you specify it you also have to do the stuff libvirt does for you automatically | |
| 18:22:09 | sean-k-mooney | basically | |
| 18:22:13 | sean-k-mooney | devices> | |
| 18:22:15 | sean-k-mooney | <controller type='pci' index='0' model='pci-root'/> | |
| 18:22:17 | sean-k-mooney | <controller type='pci' index='1' model='pci-bridge'> | |
| 18:22:19 | sean-k-mooney | <address type='pci' domain='0' bus='0' slot='5' function='0' multifunction='off'/> | |
| 18:22:21 | sean-k-mooney | </controller> | |
| 18:22:23 | sean-k-mooney | </devices> | |
| 18:22:37 | sean-k-mooney | well it would be multifucniton=one in this case | |
| 18:22:52 | sean-k-mooney | but once we have the multi funtion bridge device you can add the enpoint beneth it | |
| 18:24:11 | sean-k-mooney | i guess the only real difference that mater for q35 vs pc is pcie-bridge vs pci-bridge in the contoler element | |
| 18:24:19 | deke997 | Okay so, haha, what if we let libvirt handle the target elements, but then before we boot, we have nova edit the slot and function numbers so that pairs match | |
| 18:24:31 | deke997 | :') | |
| 18:25:04 | sean-k-mooney | am first reaction i dont like that but thinking it though. | |
| 18:25:14 | sean-k-mooney | we could defien the domain without starting it | |
| 18:25:25 | deke997 | yes | |
| 18:25:27 | sean-k-mooney | that would cause libvirt to fill in the details i think | |
| 18:25:49 | sean-k-mooney | then we could read it back and ed the pair. maybe | |
| 18:26:00 | deke997 | all we'd have to do is check for two source lines that match and then make the targets match too | |
| 18:26:09 | sean-k-mooney | but it kind fo feels like a hack and it like wont scale well if there are multile deivecs | |
| 18:26:16 | deke997 | it is definitely a hack | |
| 18:26:31 | deke997 | why won't it scale well? | |
| 18:27:07 | deke997 | never gonna have more than ~8 devices in a server | |
| 18:27:11 | sean-k-mooney | if we had multile host deves we woudl get weired gaps in the pci adress space and if we had more then 8 virtual funciton it woudl get complcated | |
| 18:27:48 | sean-k-mooney | well the bit after the . is in octal so we can on ly have 8 fucntion per slot | |
| 18:28:01 | sean-k-mooney | so if you needed more then 8 virtual funtion it would not work right | |
| 18:28:17 | deke997 | I don't think such a device exists | |
| 18:28:30 | deke997 | haven't ever seen more than like 4 | |
| 18:28:34 | deke997 | my use case is only 2 | |
| 18:28:54 | sean-k-mooney | ya i have seen up to 8 well no i have seen way more | |
| 18:29:10 | sean-k-mooney | intel nics when using sriov can have 64 - 128 vf per pf | |
| 18:29:17 | sean-k-mooney | but they do the adressing slightly differently | |
| 18:29:20 | deke997 | sriov is totally different tho | |
| 18:29:37 | sean-k-mooney | ya for there case teh assing a full bus per card | |
| 18:29:58 | deke997 | this would be specifically for non-sriov multi-function devices | |
| 18:30:00 | efried | stephenfin: https://zuul.opendev.org/t/openstack/build/cc8e185b95cf4d5cbe973c4b81493084 is on rackcdn so I can't get to it. Can you see what the fail is? (ec2) | |
| 18:30:06 | sean-k-mooney | so you have mulitple slots and funtion to support up to 256 vf totall | |
| 18:30:25 | sean-k-mooney | deke997: ack | |
| 18:30:37 | sean-k-mooney | deke997: so you can see why i think this would need a spec | |
| 18:30:46 | sean-k-mooney | to design correctly | |
| 18:31:22 | deke997 | yes, I'm just hoping for an intermediate band aid while we work on a spec | |
| 18:31:26 | sean-k-mooney | i do see a use case for both multi funtion deivce passthough and the more generif i have a pair of device tha tmus be allocated together | |
| 18:31:57 | sean-k-mooney | e.g. the gpu and the audio contole on the gpu | |