| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-cyborg - 2018-03-21 | |||
| 14:15:13 | kosamara | It looks like something that could be done in cyborg? | |
| 14:15:42 | zhipeng | i think so, i remember we had similar convo regarding clean up on FPGA in the PTG | |
| 14:15:43 | Li_Liu | are you try to force a "reset" after usage of the device? | |
| 14:16:02 | kosamara | Reset of the device? | |
| 14:16:32 | kosamara | According to a research article, in nvidia's case performing a device reset through nvidia-smi is not enough for the data leaks. | |
| 14:16:42 | Li_Liu | something like that. like Zhipeng said, "clean up" | |
| 14:16:49 | kosamara | - article: https://www.semanticscholar.org/paper/Confidentiality-Issues-on-a-GPU-in-a-Virtualized-E-Maurice-Neumann/693a8b56a9e961052702ff088131eb553e88d9ae | |
| 14:17:19 | kosamara | The additional complexity in pci passthrough is that the host can't access the GPUs (no drivers) | |
| 14:17:39 | shaohe_feng_ | so once the GPU devices are detached, cyborg should do clean up at once. | |
| 14:17:40 | Sundar | What kind of clean up do you have in mind? Zero out the RAM? | |
| 14:18:15 | kosamara | My thinking is to put the relevant resources in a special state after deallocation from their previous VM and use a service VM to perform the cleanup ops themselves. | |
| 14:18:21 | Li_Liu | memset() to zero for all ? | |
| 14:18:27 | kosamara | Yes, zero the ram | |
| 14:19:00 | kosamara | But ideally we would like to ensure the firmware's state is valid too. | |
| 14:19:13 | shaohe_feng_ | only this one method to clear up? | |
| 14:19:47 | kosamara | Sorry, I'm out of context. This method is from where? | |
| 14:19:58 | shaohe_feng_ | zero the ram | |
| 14:20:15 | kosamara | Yes for the ram. | |
| 14:20:30 | shaohe_feng_ | OK, got it. | |
| 14:20:35 | kosamara | But we would also like to check the firmware's state. | |
| 14:20:54 | kosamara | And at least ensure that it hasn't been tampered with. | |
| 14:21:22 | shaohe_feng_ | so no ways to power off the GPU device, and re-power it? | |
| 14:21:23 | zhipeng | i think this is doable with cyborg | |
| 14:21:43 | kosamara | Not that I know of, apart from rebooting the host. | |
| 14:21:44 | zhipeng | shaohe_feng_ that will mess up the state ? | |
| 14:21:59 | Sundar | The service VM needs to run in every compute node, and it needs to have all the right drivers for various GPU devices. We need to see how practical that is. | |
| 14:22:14 | Li_Liu | we might need to get some help from Nvidia | |
| 14:22:48 | zhipeng | let's say we have the NVIDIA driver support | |
| 14:22:50 | Sundar | In a vGPU scenario, how do you power off the device without affecting other users? | |
| 14:22:52 | zhipeng | for cyborg | |
| 14:22:59 | kosamara | Sundar: unless vfio is capable of doing these basic operations? | |
| 14:23:39 | zhipeng | do we still need the service VM ? | |
| 14:24:11 | kosamara | I haven't explored practically nvidia's vGPU scenario. The host is supposed to be able to operate on the GPU in that case. | |
| 14:24:32 | Sundar | kosamara: Not sure I understand. vfio is a generic module for allowing device access. How would it know about specific nvidia devices? | |
| 14:24:44 | shaohe_feng_ | kosamara: do you use vfio pci passthrough or pci-stub passthrough? | |
| 14:25:08 | kosamara | zhipeng: vfio-pci is the host stub driver when the gpu is passed through. If we could do something through that, then we wouldn't need the service VM. | |
| 14:25:32 | kosamara | shaohe_feng_: vfio-pci | |
| 14:25:37 | zhipeng | kosamara good to know | |
| 14:26:16 | kosamara | Sundar: yes, I'm just making a hypothesis. I expect it can't, but perhaps zeroing-out the ram is a general enough operation and it can do it...? | |
| 14:27:40 | Vipparthy | hi | |
| 14:27:49 | shaohe_feng_ | kosamara: is zeroing-out the ram time consuming？ | |
| 14:28:20 | Sundar | kosamara: I am not a Nvidia expert. But presumably, to access the RAM, one would need to look at the PCI BAR space and find the base address or something? If so, that would be device-specific | |
| 14:29:49 | kosamara | Sundar: thanks for the input. I can research the feasibility of this option this week. | |
| 14:30:15 | zhipeng | kosamara I will also talk to the NVIDIA OpenStack team about it | |
| 14:30:26 | zhipeng | see if we could come out with something for Rocky :) | |
| 14:30:45 | kosamara | shaohe_feng_: I don't have a number right now. It should depend on the manner: if it happens on the device without pci transports it should be quite fast. | |
| 14:31:08 | kosamara | zhipeng thanks :) | |
| 14:32:01 | zhipeng | kosamara is there anything like an architecture diagram for the use case ? | |
| 14:32:20 | kosamara | Not really. | |
| 14:32:32 | zhipeng | okey :) | |
| 14:33:00 | Sundar | kosamara: you also mentioned quotas on GPUs? | |
| 14:33:03 | kosamara | What exactly do you mean by architecture diagram? | |
| 14:33:09 | Li_Liu | this rings a bell to me about the driver apis | |
| 14:33:20 | zhipeng | kosarama like the overall setup | |
| 14:33:33 | Li_Liu | currently we have report() and program() for the vendor driver api | |
| 14:33:50 | kosamara | Yes. We currently implement quotas indirectly. We only allow GPU flavors on specific projects and quota them by cpu. | |
| 14:33:58 | Li_Liu | we might want to consider a reset() api for the drivers | |
| 14:34:09 | zhipeng | Li_Liu makes sense | |
| 14:34:21 | zhipeng | kosamara we will also have quota support for Rocky | |
| 14:34:22 | shaohe_feng_ | +1 | |
| 14:35:13 | kosamara | Good to know. | |
| 14:35:28 | zhipeng | okey folks let's move on to the next topic | |
| 14:35:36 | zhipeng | thx again kosamara | |
| 14:35:40 | kosamara | thanks! | |
| 14:35:47 | shaohe_feng_ | kosamara: so do you not support GPU quota for specific projects？ | |
| 14:35:56 | zhipeng | #topic subteam lead report | |
| 14:36:04 | kosamara | No, we do it indirectly, in the way I mentioned. | |
| 14:36:24 | zhipeng | shaohe_feng_ scrow up :) | |
| 14:36:39 | zhipeng | scroll | |
| 14:37:02 | zhipeng | okey yumeng could you introduce the progress on your side ? | |
| 14:37:43 | zhipeng | Yumeng__ | |
| 14:39:24 | Yumeng__ | Last week I set up a repository for cyborg-specs | |
| 14:40:24 | Yumeng__ | some patches are still waiting for review from the infra team | |
| 14:40:49 | Yumeng__ | Hope it could be merged ASAP | |
| 14:40:55 | Sundar | Are we not using the git repo's doc/specs/ area anymore? | |
| 14:41:14 | zhipeng | Sundar we are migrating it out :) | |
| 14:41:22 | Sundar | Maybe I am missing the context. What is this repository for specs? | |
| 14:41:27 | zhipeng | but it is fine now to keep submit to that folder | |
| 14:41:46 | zhipeng | we will migrate all the approved specs after MS1 to cyborg-specs | |
| 14:42:11 | Sundar | Howard, could you explain why we are doing that? | |
| 14:42:36 | zhipeng | It would be better for the documentation when we do release | |
| 14:42:47 | zhipeng | all the core projects are doing it | |
| 14:43:33 | zhipeng | i think yumeng also add the gate check on docs for cyborg-specs | |
| 14:44:13 | crushil | Sundar All the core projects split out the specs and the main project. So, it makes sense to follow suite | |
| 14:44:31 | zhipeng | yes exactly | |
| 14:44:35 | Sundar | IIUC, in the git repo, approved specs will be doc/specs/<release>/approved, but in the new repo, all release specs will be in one place. Is that right? | |
| 14:45:03 | zhipeng | it will still follows the similar directory structure | |
| 14:45:36 | Sundar | OK, so we are just separating code from docs | |
| 14:45:41 | zhipeng | yes | |
| 14:45:48 | zhipeng | from specs to be preceise | |
| 14:45:50 | zhipeng | precise | |
| 14:45:57 | Sundar | Got it, thanks :) | |
| 14:46:08 | zhipeng | since general documentation is still in cyborg repo, if I understand correctly | |
| 14:46:09 | Li_Liu | for now we still check in the docs to the code repo right? | |
| 14:46:19 | zhipeng | Li_Liu yes, nothing changes | |
| 14:46:24 | Li_Liu | ok | |
| 14:46:40 | zhipeng | thx to Yumeng__ for the quick progress | |
| 14:46:56 | Sundar | In future releases, would we check specs into code repo, and have it be migrated after approval? | |
| 14:47:04 | Yumeng__ | zhipeng: :) | |
| 14:47:21 | zhipeng | in the future we will just submit the spec patch to cyborg-specs | |