| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-01-23 | |||
| 17:38:36 | deke997 | by manually editing the libvirt.xml file, I verified that it works properly in the VM when set up like that | |
| 17:39:17 | deke997 | However, the libvirt.xml file is now generated automatically by nova, and by default it puts the devices in separate slots like: 0000:00:05.0 and 0000:00:06.0 | |
| 17:40:49 | deke997 | So I want to change that behavior so that when I have a pci device that is passed through, it will pay attention to whether 2 devices were in the same slot with different functions on the host machine, and if so, it will repeat that behavior in the VM | |
| 17:47:19 | deke997 | stephenfin my understanding is that the xml file generation is handled by driver.py and config.py, but I cannot find the specific areas that deal with generating the guest pci address | |
| 17:47:29 | stephenfin | I'm trying to root that out at the moment | |
| 17:48:22 | yoctozepto | deke997: btw, is the device in question a secret or can you shed some light on it? | |
| 17:49:53 | sean-k-mooney | deke997: we dont do all the generation in nova | |
| 17:50:16 | sean-k-mooney | we actully leave it up to libvirt to assign the guesst pci adress and as a result define the toplogy | |
| 17:51:38 | sean-k-mooney | deke997: i have a backloag item to modify how that works for other reasons in the futre as i want o model 1 pci root complex per numa node to optimize how devices are exposed to vms | |
| 17:52:35 | sean-k-mooney | but it sould like you need to detect that the device is a multi fucntion device and then we would need to explcitly stat generating the toplogy and assing pci address in the guest to enable use to expose the device as a multi function device to the guest | |
| 17:52:58 | sean-k-mooney | deke997: this would be well into the catagory of a spec not a bugfix in my view | |
| 17:53:51 | deke997 | sean-k-mooney I see. This sounds like a fun project to work on. In the meantime, however, I have to find a solution that will let me get my VMs up | |
| 17:53:51 | sean-k-mooney | we might be able to skip the adress assignmentand still delegate that to libvirt but indicate that the device are both part of a multi function device but we would have to explore that to confirm | |
| 17:54:38 | deke997 | libvirt has a multifunction=on setting too | |
| 17:55:04 | sean-k-mooney | how to you request and whitelist the device at present | |
| 17:55:17 | sean-k-mooney | i assume you are using the pci whitelist and a flavor alias | |
| 17:55:21 | deke997 | yes | |
| 17:55:30 | deke997 | but I have to use a separate alias for each function | |
| 17:55:31 | sean-k-mooney | and you have 2 aliase right | |
| 17:55:35 | sean-k-mooney | one for each funciton | |
| 17:55:41 | sean-k-mooney | ya | |
| 17:55:57 | deke997 | We could add a multifunction alias type | |
| 17:56:07 | sean-k-mooney | so how are you ensuring the two VF/PF come form the same device on the hsot | |
| 17:56:29 | sean-k-mooney | deke997: we could add a tage but unless we can adress ^ | |
| 17:56:37 | sean-k-mooney | we have bigger problems | |
| 17:56:44 | deke997 | right now, I am just using all of them, so it's not an issue haha | |
| 17:57:12 | sean-k-mooney | so you have only 1 device on the compute node and your booting 1 vm that use all the fucntion form that device | |
| 17:57:27 | deke997 | yes that's what I have set up for testing | |
| 17:57:42 | sean-k-mooney | ya in that specific case you whave worked around that issue | |
| 17:57:49 | deke997 | But in the libvirt XML it is very easy to see which devices from the host were passed through | |
| 17:57:58 | deke997 | and match them | |
| 17:58:02 | sean-k-mooney | this has come up in the context of gpus in the past | |
| 17:58:21 | sean-k-mooney | the GPU and audio contoler are usally 2 differnt pci endpoint on the same device | |
| 17:58:48 | sean-k-mooney | at pressent we dont have a way to passthough both form the same card and ensure they are form the same device | |
| 17:59:51 | sean-k-mooney | deke997: form a nova point of view you would basically want ot have an alsi that can refernce a second alias and request both as a pair | |
| 18:00:08 | deke997 | yea exactly | |
| 18:00:19 | deke997 | That doesn't seem super difficult | |
| 18:00:43 | sean-k-mooney | and then in nova/libvirt have it select the pair form the same device an make them a multifunction device in the xml if they are expose that way on the host | |
| 18:01:04 | sean-k-mooney | deke997: you say that now | |
| 18:01:08 | deke997 | haha | |
| 18:01:43 | deke997 | Is there a way to make nova play nice with edited libvirt.xmls? | |
| 18:01:44 | sean-k-mooney | so this all live in the pci manger code https://github.com/openstack/nova/tree/master/nova/pci | |
| 18:01:52 | sean-k-mooney | no | |
| 18:02:18 | sean-k-mooney | nova regenerates the xml based on seveal api actions | |
| 18:02:22 | sean-k-mooney | like hard reboot | |
| 18:02:36 | sean-k-mooney | so your eddit would last until someone stops and start the vm | |
| 18:02:56 | deke997 | Is there a way to disable that regeneration? | |
| 18:03:06 | sean-k-mooney | no | |
| 18:03:17 | openstackgerrit | Merged openstack/nova master: libvirt: Add a default VirtIO-RNG device to guests https://review.opendev.org/616603 | |
| 18:04:12 | sean-k-mooney | so there are three parts. the pci mager woudl have to be enhanced to pars and track paired deivces, the pci pasthough filter would have to be update and the libvirt driver xml generateion would have to be updated | |
| 18:05:22 | sean-k-mooney | the alias schema coudl certenly be extended to referen a child device https://github.com/openstack/nova/blob/master/nova/pci/request.py#L66-L99 | |
| 18:05:29 | deke997 | How would we go about updating the xml generation? | |
| 18:06:27 | sean-k-mooney | am ill see if i can find it. but likely i would see if we can still leave libvirt choose the target pci adders in the guest but mark a specicifc pci requiest as multi funcion | |
| 18:08:53 | deke997 | I need to know a bit more about how libvirt chooses those addresses | |
| 18:09:19 | sean-k-mooney | so you would have to modify https://github.com/openstack/nova/blob/6db486e9fd4f6b8dd02371b043e66808cdd1e0cd/nova/virt/libvirt/driver.py#L5871 | |
| 18:10:10 | sean-k-mooney | the general docs for thi in libvirt are https://libvirt.org/formatdomain.html#elementsHostDevSubsys | |
| 18:11:25 | sean-k-mooney | normally we just generate a hostdev element like this | |
| 18:11:30 | sean-k-mooney | <hostdev mode='subsystem' type='pci' managed='yes'> | |
| 18:11:32 | sean-k-mooney | <source> | |
| 18:11:34 | sean-k-mooney | <address domain='0x0000' bus='0x06' slot='0x02' function='0x0'/> | |
| 18:11:36 | sean-k-mooney | </source> | |
| 18:11:38 | sean-k-mooney | <boot order='1'/> | |
| 18:11:40 | sean-k-mooney | <rom bar='on' file='/etc/fake/boot.bin'/> | |
| 18:11:42 | sean-k-mooney | </hostdev> | |
| 18:11:55 | sean-k-mooney | so we specify the source element and the host pci adress and libvirt chooes a target adress | |
| 18:12:07 | sean-k-mooney | that i assume is what you were changing with your manual edit | |
| 18:12:14 | deke997 | Yes | |
| 18:12:38 | deke997 | but I could only start that instance manually through libvirt after the edit, because nova just tries to overwrite it | |
| 18:13:39 | sean-k-mooney | https://libvirt.org/formatdomain.html#elementsAddress this is the docs for the device addressing | |
| 18:13:46 | deke997 | Also, adding multifunction = 'on' to the source address may be necessary. Libvirt docs say to use it, but I was able to have success either way | |
| 18:14:37 | deke997 | which would have to be done in nova | |
| 18:15:27 | deke997 | would be easy to just add multifunction = yes iff alias_type = multifunction or something | |
| 18:16:45 | sean-k-mooney | ya the multifunction bit is not what im worried about | |
| 18:17:14 | sean-k-mooney | i think the only way to make this work si to have nova set the target Adress on both devices | |
| 18:17:31 | sean-k-mooney | but since nova does not track the target adress today that is a prbalem | |
| 18:17:56 | sean-k-mooney | also you would have to take into account both q35 and the i440x machine types | |
| 18:18:14 | sean-k-mooney | the old i440x or pc machine type is pci only | |
| 18:18:32 | sean-k-mooney | and q35 is pcie by default but you can add pci busses too | |
| 18:18:47 | sean-k-mooney | so that makes it more complicated | |
| 18:19:47 | deke997 | definitely | |
| 18:20:41 | sean-k-mooney | with q35 if we wanted to set <address domain='0x0000' bus='0x06' slot='0x02' function='0x0'/> | |
| 18:20:59 | sean-k-mooney | for the target we would need to first create a pcie root port for it | |
| 18:21:16 | sean-k-mooney | if we leave out the target element libvirt does it for us | |
| 18:21:35 | sean-k-mooney | so baseicly when you specify it you also have to do the stuff libvirt does for you automatically | |
| 18:22:09 | sean-k-mooney | basically | |
| 18:22:13 | sean-k-mooney | devices> | |
| 18:22:15 | sean-k-mooney | <controller type='pci' index='0' model='pci-root'/> | |
| 18:22:17 | sean-k-mooney | <controller type='pci' index='1' model='pci-bridge'> | |
| 18:22:19 | sean-k-mooney | <address type='pci' domain='0' bus='0' slot='5' function='0' multifunction='off'/> | |
| 18:22:21 | sean-k-mooney | </controller> | |
| 18:22:23 | sean-k-mooney | </devices> | |
| 18:22:37 | sean-k-mooney | well it would be multifucniton=one in this case | |
| 18:22:52 | sean-k-mooney | but once we have the multi funtion bridge device you can add the enpoint beneth it | |
| 18:24:11 | sean-k-mooney | i guess the only real difference that mater for q35 vs pc is pcie-bridge vs pci-bridge in the contoler element | |
| 18:24:19 | deke997 | Okay so, haha, what if we let libvirt handle the target elements, but then before we boot, we have nova edit the slot and function numbers so that pairs match | |
| 18:24:31 | deke997 | :') | |
| 18:25:04 | sean-k-mooney | am first reaction i dont like that but thinking it though. | |
| 18:25:14 | sean-k-mooney | we could defien the domain without starting it | |
| 18:25:25 | deke997 | yes | |
| 18:25:27 | sean-k-mooney | that would cause libvirt to fill in the details i think | |