| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-01-23 | |||
| 18:15:27 | deke997 | would be easy to just add multifunction = yes iff alias_type = multifunction or something | |
| 18:16:45 | sean-k-mooney | ya the multifunction bit is not what im worried about | |
| 18:17:14 | sean-k-mooney | i think the only way to make this work si to have nova set the target Adress on both devices | |
| 18:17:31 | sean-k-mooney | but since nova does not track the target adress today that is a prbalem | |
| 18:17:56 | sean-k-mooney | also you would have to take into account both q35 and the i440x machine types | |
| 18:18:14 | sean-k-mooney | the old i440x or pc machine type is pci only | |
| 18:18:32 | sean-k-mooney | and q35 is pcie by default but you can add pci busses too | |
| 18:18:47 | sean-k-mooney | so that makes it more complicated | |
| 18:19:47 | deke997 | definitely | |
| 18:20:41 | sean-k-mooney | with q35 if we wanted to set <address domain='0x0000' bus='0x06' slot='0x02' function='0x0'/> | |
| 18:20:59 | sean-k-mooney | for the target we would need to first create a pcie root port for it | |
| 18:21:16 | sean-k-mooney | if we leave out the target element libvirt does it for us | |
| 18:21:35 | sean-k-mooney | so baseicly when you specify it you also have to do the stuff libvirt does for you automatically | |
| 18:22:09 | sean-k-mooney | basically | |
| 18:22:13 | sean-k-mooney | devices> | |
| 18:22:15 | sean-k-mooney | <controller type='pci' index='0' model='pci-root'/> | |
| 18:22:17 | sean-k-mooney | <controller type='pci' index='1' model='pci-bridge'> | |
| 18:22:19 | sean-k-mooney | <address type='pci' domain='0' bus='0' slot='5' function='0' multifunction='off'/> | |
| 18:22:21 | sean-k-mooney | </controller> | |
| 18:22:23 | sean-k-mooney | </devices> | |
| 18:22:37 | sean-k-mooney | well it would be multifucniton=one in this case | |
| 18:22:52 | sean-k-mooney | but once we have the multi funtion bridge device you can add the enpoint beneth it | |
| 18:24:11 | sean-k-mooney | i guess the only real difference that mater for q35 vs pc is pcie-bridge vs pci-bridge in the contoler element | |
| 18:24:19 | deke997 | Okay so, haha, what if we let libvirt handle the target elements, but then before we boot, we have nova edit the slot and function numbers so that pairs match | |
| 18:24:31 | deke997 | :') | |
| 18:25:04 | sean-k-mooney | am first reaction i dont like that but thinking it though. | |
| 18:25:14 | sean-k-mooney | we could defien the domain without starting it | |
| 18:25:25 | deke997 | yes | |
| 18:25:27 | sean-k-mooney | that would cause libvirt to fill in the details i think | |
| 18:25:49 | sean-k-mooney | then we could read it back and ed the pair. maybe | |
| 18:26:00 | deke997 | all we'd have to do is check for two source lines that match and then make the targets match too | |
| 18:26:09 | sean-k-mooney | but it kind fo feels like a hack and it like wont scale well if there are multile deivecs | |
| 18:26:16 | deke997 | it is definitely a hack | |
| 18:26:31 | deke997 | why won't it scale well? | |
| 18:27:07 | deke997 | never gonna have more than ~8 devices in a server | |
| 18:27:11 | sean-k-mooney | if we had multile host deves we woudl get weired gaps in the pci adress space and if we had more then 8 virtual funciton it woudl get complcated | |
| 18:27:48 | sean-k-mooney | well the bit after the . is in octal so we can on ly have 8 fucntion per slot | |
| 18:28:01 | sean-k-mooney | so if you needed more then 8 virtual funtion it would not work right | |
| 18:28:17 | deke997 | I don't think such a device exists | |
| 18:28:30 | deke997 | haven't ever seen more than like 4 | |
| 18:28:34 | deke997 | my use case is only 2 | |
| 18:28:54 | sean-k-mooney | ya i have seen up to 8 well no i have seen way more | |
| 18:29:10 | sean-k-mooney | intel nics when using sriov can have 64 - 128 vf per pf | |
| 18:29:17 | sean-k-mooney | but they do the adressing slightly differently | |
| 18:29:20 | deke997 | sriov is totally different tho | |
| 18:29:37 | sean-k-mooney | ya for there case teh assing a full bus per card | |
| 18:29:58 | deke997 | this would be specifically for non-sriov multi-function devices | |
| 18:30:00 | efried | stephenfin: https://zuul.opendev.org/t/openstack/build/cc8e185b95cf4d5cbe973c4b81493084 is on rackcdn so I can't get to it. Can you see what the fail is? (ec2) | |
| 18:30:06 | sean-k-mooney | so you have mulitple slots and funtion to support up to 256 vf totall | |
| 18:30:25 | sean-k-mooney | deke997: ack | |
| 18:30:37 | sean-k-mooney | deke997: so you can see why i think this would need a spec | |
| 18:30:46 | sean-k-mooney | to design correctly | |
| 18:31:22 | deke997 | yes, I'm just hoping for an intermediate band aid while we work on a spec | |
| 18:31:26 | sean-k-mooney | i do see a use case for both multi funtion deivce passthough and the more generif i have a pair of device tha tmus be allocated together | |
| 18:31:57 | sean-k-mooney | e.g. the gpu and the audio contole on the gpu | |
| 18:32:49 | sean-k-mooney | nviad might actully be exposing it as a multi function device so that might actully be the same usecase | |
| 18:35:08 | sean-k-mooney | ya they are | |
| 18:35:13 | sean-k-mooney | 81:00.0 VGA compatible controller: NVIDIA Corporation GK104GL [Quadro K5000] (rev a1) | |
| 18:35:15 | sean-k-mooney | 81:00.1 Audio device: NVIDIA Corporation GK104 HDMI Audio Controller (rev a1) | |
| 18:35:44 | sean-k-mooney | so if we we made ^ work it would work for you too right | |
| 18:38:08 | deke997 | yes | |
| 18:38:27 | deke997 | That looks pretty much identical to my use case | |
| 18:42:09 | sean-k-mooney | ok what i would basicaly suggest is as follows. extent the alias with a child tag that can reference other alisas. | |
| 18:42:48 | sean-k-mooney | when you add an alias to a flaovr which has a child tag we woudl claim the deivce and all its childern | |
| 18:43:19 | sean-k-mooney | and expose them as a multi funcitn device | |
| 18:44:16 | sean-k-mooney | the pci pasthogh through filter would have to be enhanced to ensure it only passes host where the parent and child alias can be allocated form the same slot | |
| 18:44:51 | sean-k-mooney | the pci manger would also need to updated so that when we do the pci claim it claims the pair form the same slot | |
| 18:44:54 | deke997 | wouldn't that always be the case? | |
| 18:45:09 | deke997 | I don't see why the passthrough filter needs to be updated | |
| 18:45:21 | sean-k-mooney | well if you put two of those card in the same server it would break | |
| 18:46:07 | sean-k-mooney | as we coudl get 1 fucntion of type a form one card and one fucntion of type b from another | |
| 18:46:21 | sean-k-mooney | so the filter and assignment code would have to be hardened to prevent that | |
| 18:48:27 | deke997 | The assignment code, yes, but the filter, I don't think so | |
| 18:48:59 | deke997 | because before assignment, it would always be true that both the parent and child could be allocated together | |
| 18:49:31 | sean-k-mooney | its true if you alwasy need to request both | |
| 18:49:38 | sean-k-mooney | which is true in your case | |
| 18:49:38 | deke997 | right | |
| 18:49:42 | sean-k-mooney | not in the gpu case | |
| 18:49:47 | deke997 | I see | |
| 18:50:12 | deke997 | in that case, then yea the filter too | |
| 18:50:16 | sean-k-mooney | anyway the assignment code and filter share the same code | |
| 18:50:41 | sean-k-mooney | the filter basicaly passes if it would be able to assign | |
| 18:51:44 | sean-k-mooney | deke997: https://github.com/openstack/nova/blob/master/nova/scheduler/filters/pci_passthrough_filter.py#L51 | |
| 18:52:01 | sean-k-mooney | see we just loop over the request and ask can it suppor the request | |
| 18:52:20 | sean-k-mooney | well the loopin is don internally in the suport_request funtion | |
| 18:53:06 | sean-k-mooney | we just make a copy of the avliable devices and try to assigntem on hte copy https://github.com/openstack/nova/blob/6db486e9fd4f6b8dd02371b043e66808cdd1e0cd/nova/pci/stats.py#L374-L375 | |
| 18:54:40 | sean-k-mooney | so if you update _apply_request to handel the assignment constriatt the filter gets updated for free | |
| 18:55:13 | deke997 | got it | |
| 18:56:10 | deke997 | are we proposing to let libvirt handle initial target device config and then have nova take over, or are we proposing to have nova handle target device config? | |
| 18:56:30 | deke997 | These are the only two options so far, yea? | |
| 18:56:52 | sean-k-mooney | the only two i am aware of. if we were to do this updatream i would prefer to do it properly | |
| 18:57:15 | sean-k-mooney | and have nova do it. if you need to do it downstream quick well that is up to you | |
| 18:57:42 | sean-k-mooney | deke997: what is the eta you had in mind for this | |
| 18:57:52 | deke997 | 2 days ago haha | |
| 18:58:05 | deke997 | I won't be sleeping much till I get this working | |
| 18:58:19 | sean-k-mooney | ya fair warning to land this upstream its proably going to be next cycle | |
| 18:58:35 | deke997 | yea so I think we'll do both | |
| 18:58:44 | deke997 | downstream quick, and then work on upstream | |
| 18:58:51 | sean-k-mooney | i could be ussuri if it was really pusshed for but this code is complex and a pain to test | |
| 18:58:51 | deke997 | when is the next cycle? | |