| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-01-23 | |||
| 18:25:25 | deke997 | yes | |
| 18:25:27 | sean-k-mooney | that would cause libvirt to fill in the details i think | |
| 18:25:49 | sean-k-mooney | then we could read it back and ed the pair. maybe | |
| 18:26:00 | deke997 | all we'd have to do is check for two source lines that match and then make the targets match too | |
| 18:26:09 | sean-k-mooney | but it kind fo feels like a hack and it like wont scale well if there are multile deivecs | |
| 18:26:16 | deke997 | it is definitely a hack | |
| 18:26:31 | deke997 | why won't it scale well? | |
| 18:27:07 | deke997 | never gonna have more than ~8 devices in a server | |
| 18:27:11 | sean-k-mooney | if we had multile host deves we woudl get weired gaps in the pci adress space and if we had more then 8 virtual funciton it woudl get complcated | |
| 18:27:48 | sean-k-mooney | well the bit after the . is in octal so we can on ly have 8 fucntion per slot | |
| 18:28:01 | sean-k-mooney | so if you needed more then 8 virtual funtion it would not work right | |
| 18:28:17 | deke997 | I don't think such a device exists | |
| 18:28:30 | deke997 | haven't ever seen more than like 4 | |
| 18:28:34 | deke997 | my use case is only 2 | |
| 18:28:54 | sean-k-mooney | ya i have seen up to 8 well no i have seen way more | |
| 18:29:10 | sean-k-mooney | intel nics when using sriov can have 64 - 128 vf per pf | |
| 18:29:17 | sean-k-mooney | but they do the adressing slightly differently | |
| 18:29:20 | deke997 | sriov is totally different tho | |
| 18:29:37 | sean-k-mooney | ya for there case teh assing a full bus per card | |
| 18:29:58 | deke997 | this would be specifically for non-sriov multi-function devices | |
| 18:30:00 | efried | stephenfin: https://zuul.opendev.org/t/openstack/build/cc8e185b95cf4d5cbe973c4b81493084 is on rackcdn so I can't get to it. Can you see what the fail is? (ec2) | |
| 18:30:06 | sean-k-mooney | so you have mulitple slots and funtion to support up to 256 vf totall | |
| 18:30:25 | sean-k-mooney | deke997: ack | |
| 18:30:37 | sean-k-mooney | deke997: so you can see why i think this would need a spec | |
| 18:30:46 | sean-k-mooney | to design correctly | |
| 18:31:22 | deke997 | yes, I'm just hoping for an intermediate band aid while we work on a spec | |
| 18:31:26 | sean-k-mooney | i do see a use case for both multi funtion deivce passthough and the more generif i have a pair of device tha tmus be allocated together | |
| 18:31:57 | sean-k-mooney | e.g. the gpu and the audio contole on the gpu | |
| 18:32:49 | sean-k-mooney | nviad might actully be exposing it as a multi function device so that might actully be the same usecase | |
| 18:35:08 | sean-k-mooney | ya they are | |
| 18:35:13 | sean-k-mooney | 81:00.0 VGA compatible controller: NVIDIA Corporation GK104GL [Quadro K5000] (rev a1) | |
| 18:35:15 | sean-k-mooney | 81:00.1 Audio device: NVIDIA Corporation GK104 HDMI Audio Controller (rev a1) | |
| 18:35:44 | sean-k-mooney | so if we we made ^ work it would work for you too right | |
| 18:38:08 | deke997 | yes | |
| 18:38:27 | deke997 | That looks pretty much identical to my use case | |
| 18:42:09 | sean-k-mooney | ok what i would basicaly suggest is as follows. extent the alias with a child tag that can reference other alisas. | |
| 18:42:48 | sean-k-mooney | when you add an alias to a flaovr which has a child tag we woudl claim the deivce and all its childern | |
| 18:43:19 | sean-k-mooney | and expose them as a multi funcitn device | |
| 18:44:16 | sean-k-mooney | the pci pasthogh through filter would have to be enhanced to ensure it only passes host where the parent and child alias can be allocated form the same slot | |
| 18:44:51 | sean-k-mooney | the pci manger would also need to updated so that when we do the pci claim it claims the pair form the same slot | |
| 18:44:54 | deke997 | wouldn't that always be the case? | |
| 18:45:09 | deke997 | I don't see why the passthrough filter needs to be updated | |
| 18:45:21 | sean-k-mooney | well if you put two of those card in the same server it would break | |
| 18:46:07 | sean-k-mooney | as we coudl get 1 fucntion of type a form one card and one fucntion of type b from another | |
| 18:46:21 | sean-k-mooney | so the filter and assignment code would have to be hardened to prevent that | |
| 18:48:27 | deke997 | The assignment code, yes, but the filter, I don't think so | |
| 18:48:59 | deke997 | because before assignment, it would always be true that both the parent and child could be allocated together | |
| 18:49:31 | sean-k-mooney | its true if you alwasy need to request both | |
| 18:49:38 | sean-k-mooney | which is true in your case | |
| 18:49:38 | deke997 | right | |
| 18:49:42 | sean-k-mooney | not in the gpu case | |
| 18:49:47 | deke997 | I see | |
| 18:50:12 | deke997 | in that case, then yea the filter too | |
| 18:50:16 | sean-k-mooney | anyway the assignment code and filter share the same code | |
| 18:50:41 | sean-k-mooney | the filter basicaly passes if it would be able to assign | |
| 18:51:44 | sean-k-mooney | deke997: https://github.com/openstack/nova/blob/master/nova/scheduler/filters/pci_passthrough_filter.py#L51 | |
| 18:52:01 | sean-k-mooney | see we just loop over the request and ask can it suppor the request | |
| 18:52:20 | sean-k-mooney | well the loopin is don internally in the suport_request funtion | |
| 18:53:06 | sean-k-mooney | we just make a copy of the avliable devices and try to assigntem on hte copy https://github.com/openstack/nova/blob/6db486e9fd4f6b8dd02371b043e66808cdd1e0cd/nova/pci/stats.py#L374-L375 | |
| 18:54:40 | sean-k-mooney | so if you update _apply_request to handel the assignment constriatt the filter gets updated for free | |
| 18:55:13 | deke997 | got it | |
| 18:56:10 | deke997 | are we proposing to let libvirt handle initial target device config and then have nova take over, or are we proposing to have nova handle target device config? | |
| 18:56:30 | deke997 | These are the only two options so far, yea? | |
| 18:56:52 | sean-k-mooney | the only two i am aware of. if we were to do this updatream i would prefer to do it properly | |
| 18:57:15 | sean-k-mooney | and have nova do it. if you need to do it downstream quick well that is up to you | |
| 18:57:42 | sean-k-mooney | deke997: what is the eta you had in mind for this | |
| 18:57:52 | deke997 | 2 days ago haha | |
| 18:58:05 | deke997 | I won't be sleeping much till I get this working | |
| 18:58:19 | sean-k-mooney | ya fair warning to land this upstream its proably going to be next cycle | |
| 18:58:35 | deke997 | yea so I think we'll do both | |
| 18:58:44 | deke997 | downstream quick, and then work on upstream | |
| 18:58:51 | sean-k-mooney | i could be ussuri if it was really pusshed for but this code is complex and a pain to test | |
| 18:58:51 | deke997 | when is the next cycle? | |
| 18:59:16 | sean-k-mooney | it starts in 3 monts os like september/october | |
| 18:59:30 | sean-k-mooney | there is still about 2-3 weeks to propose specs for this cycle | |
| 18:59:57 | sean-k-mooney | you would need to figure out what need to chagne and write up a propals and get it review by then | |
| 19:00:17 | sean-k-mooney | then there is able 2 months left to land the feature | |
| 19:00:30 | sean-k-mooney | but beign realisting if you need this quick | |
| 19:00:52 | sean-k-mooney | then likely createing a poc and upstreaming it next cycle is what your going to have to do | |
| 19:01:32 | deke997 | One thing to check on: | |
| 19:01:32 | sean-k-mooney | for the poc you could do the hack where you fix up the adresses | |
| 19:01:52 | deke997 | I think cyborg is doing some stuff with multi function too | |
| 19:01:59 | deke997 | I saw it while doing research | |
| 19:02:04 | sean-k-mooney | ya so cyborg would be another option | |
| 19:02:15 | sean-k-mooney | if the integration is done this cycle | |
| 19:02:34 | sean-k-mooney | then you could write a device dirver for you custom multifunion device | |
| 19:02:50 | sean-k-mooney | but the xml change would still need to be done | |
| 19:03:03 | sean-k-mooney | you would not have to touch the filer or pci manager in that case | |
| 19:03:14 | sean-k-mooney | but you would need to do all the cyborg work | |
| 19:04:51 | sean-k-mooney | deke997: cyborg support in nova is planned for this cycle but we said that last cycle too | |
| 19:05:02 | sean-k-mooney | deke997: its much closer to being read this time however | |
| 19:06:35 | deke997 | Good to know | |
| 19:07:48 | deke997 | I need to look into how they're implementing multi function a bit more | |
| 19:08:13 | deke997 | but even so, I can't wait till the next cycle to get a beta working here | |
| 19:09:05 | sean-k-mooney | at present without the libvirt support for multifuntion devices in nova cybporg cant support what you need | |
| 19:09:06 | deke997 | I think the options in order of increasing time, complexity, and correctness would probably be | |
| 19:09:17 | deke997 | 1. hack | |
| 19:09:22 | deke997 | 2. nova | |
| 19:09:24 | deke997 | 3. cyborg | |
| 19:09:58 | sean-k-mooney | 0 heiring an intern to manually update every vm as it spawns | |