| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-11-15 | |||
| 08:45:16 | stephenfin | awalende: So this is interesting. The Tesla T4 supports SR-IOV https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/tesla-t4/t4-tensor-core-product-brief.pdf | |
| 08:46:21 | stephenfin | and because we're detecting that, we're flagging it as a PF https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L6562-L6565 | |
| 08:46:44 | awalende | So there is probably a difference passing them through as other regular gpus? | |
| 08:47:45 | awalende | Can we pass them even at all? | |
| 08:47:54 | stephenfin | Possibly. I could be wrong, but I do recall that type-PF devices were disabled by default because you typically wouldn't want to use them when you talk about NICs | |
| 08:47:59 | stephenfin | but it can be enabled | |
| 08:48:08 | stephenfin | do you want to pass through the PF or the VF? | |
| 08:48:23 | awalende | the whole hardware | |
| 08:48:53 | awalende | I guess probably the PF....so the usecase is, we want to pass the whole card to the guest. so that they have to install nvidia drivers by themselfs on the guest machine | |
| 08:49:09 | awalende | like we did with our older Tesla V100 cards | |
| 08:50:06 | stephenfin | Seems like you could get better resource utilization if you passed through the VFs though (since apparently that would give you up to 16 devices) | |
| 08:50:11 | stephenfin | That doesn't matter for now though | |
| 08:51:03 | stephenfin | If you want to use type-PF devices, I _think_ you need to either (a) set the 'device_type' flag in the 'alias' config option or (b) make sure the VFs are not whitelisted | |
| 08:51:46 | awalende | ye we already tried to set 'device_type' but it didn't make a difference :( | |
| 08:52:06 | stephenfin | have you tried (b)? | |
| 08:52:16 | stephenfin | to do that, you'll need to also set the 'product_id' in the 'passthrough_whitelist' | |
| 08:52:44 | stephenfin | to whatever the product_id of the PF is, assuming the PF and VF have different product_id's (they should) | |
| 08:52:54 | awalende | my colleague means he "maybe" tried it as well. We try it again and let you know | |
| 08:53:23 | stephenfin | If they don't, you're going to need to use the 'address' field, I'm afraid | |
| 08:53:31 | stephenfin | okay, do please | |
| 08:53:59 | stephenfin | sean-k-mooney might be able to help too but they're definitely fast asleep atm :) | |
| 08:54:13 | awalende | How do you make sure that "VFs are not whitelisted"? | |
| 08:55:08 | bauzas | the T4 ? | |
| 08:55:19 | bauzas | awalende: which grid driver are you using ? | |
| 08:55:25 | bauzas | GRID8 ? | |
| 08:55:53 | awalende | no driver at all, we plugged the T4 into our hypervisor nodes and want to pass them through to guests | |
| 08:56:05 | stephenfin | awalende: If SR-IOV is enabled on the device, you will have additional PCI devices appearing at e.g. 'lspci' | |
| 08:56:19 | awalende | one moment | |
| 08:57:06 | stephenfin | 'lspci | grep -i nvidia' should flag them, I'd imagine | |
| 08:57:07 | bauzas | oh right, then that's normal | |
| 08:57:26 | bauzas | Nvidia plans to deliver virtual GPUs as VFs later | |
| 08:57:36 | bauzas | I guess they already started to expose the GPU as a PF | |
| 08:57:44 | awalende | stephenfin, it only shows me one card as af:00.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) | |
| 08:57:45 | stephenfin | I'm not sure how you'd turn them on/off. Probably by echoing stuff to sysfs like you do for SR-IOV NICs, I guess | |
| 08:57:48 | bauzas | but I thought the Tesla models wouldn't get it | |
| 08:57:55 | bauzas | just the Volta ones | |
| 08:58:20 | bauzas | I'm not a nVidia salesperson, (un?)fornutately :p | |
| 08:58:23 | stephenfin | okay, it's probably disabled so | |
| 08:59:02 | bauzas | awalende: just do something like 'lspci -vv | grep 10de | |
| 08:59:06 | stephenfin | awalende: So there is definitely _something_ specific you need to do in order to use type-PF devices. I'm just trying to figure out what it is | |
| 08:59:53 | awalende | bauzas, returns empty | |
| 08:59:59 | bauzas | mmm | |
| 09:00:04 | stephenfin | awalende: This is what I'm thinking of https://github.com/openstack/nova/blob/master/nova/pci/stats.py#L289-L297 | |
| 09:00:19 | bauzas | awalende: are you OK with pasting your whole lspci output ? | |
| 09:00:49 | stephenfin | so if I'm reading that correctly, you *must* set 'dev_type': 'type-PF' in the alias | |
| 09:01:22 | awalende | which lspci command to you want pasted? 'lspci -vv'? | |
| 09:01:22 | bauzas | I agree | |
| 09:01:33 | bauzas | awalende: yes please | |
| 09:02:09 | bauzas | stephenfin: looks like you need to expose your type-PF this way | |
| 09:02:22 | stephenfin | wdym? | |
| 09:02:27 | bauzas | but I'm amazed that nvidia provides a PF and not a straight PCI ID | |
| 09:03:02 | bauzas | stephenfin: you have to expose the PCI devices in nova.conf in order to allow Nova to use them, right? | |
| 09:03:22 | stephenfin | yeah, awalende has done that mostly correctly I think | |
| 09:03:51 | awalende | https://paste.ubuntu.com/p/xXkRfFMRYh/ | |
| 09:04:12 | stephenfin | the '[pci] passthrough_whitelist' on the compute node is correct, but the '[pci] alias' on the controller and compute node doesn't appear to be (it needs 'dev_type') | |
| 09:04:43 | awalende | stephenfin, we will try soon! | |
| 09:05:09 | awalende | does dev_type needs to be specified on controller and compute? | |
| 09:05:48 | stephenfin | sorry, 'device_type' | |
| 09:06:01 | stephenfin | in the 'alias' option on both | |
| 09:06:22 | awalende | ok, we'll try it | |
| 09:06:38 | stephenfin | alias={"name":"T4","vendor_id":"10de","product_id":"1eb8","device_type":"type-PF"} | |
| 09:06:40 | stephenfin | or something like that | |
| 09:07:56 | bauzas | very interesting lspci | |
| 09:08:11 | bauzas | https://paste.ubuntu.com/p/xXkRfFMRYh/ L3241 | |
| 09:08:28 | bauzas | it shows a SR-IOV capability | |
| 09:08:35 | bauzas | but no VGA capability (!) | |
| 09:08:56 | bauzas | that's beyond my comfort zone | |
| 09:09:02 | stephenfin | yeah, it's an accelerator, not a GPU | |
| 09:09:10 | bauzas | my bad then | |
| 09:09:18 | bauzas | I'm not familiar with the product line | |
| 09:09:45 | bauzas | so we're just talking of PCI passthrugh, not VGA passthru, my bad | |
| 09:10:08 | bauzas | anyway, it doesn't expose a standard PCI address | |
| 09:12:49 | bauzas | I just feel that stephenfin is absolutely right, nova punts the address because of the internal conditional he said | |
| 09:15:21 | awalende | someone rebooted the server =.= one more minute | |
| 09:17:19 | awalende | ok, its spawning :D fingers crossed | |
| 09:20:53 | awalende | Ok, the device shows up in the guest | |
| 09:20:54 | awalende | hooray | |
| 09:21:29 | awalende | But why do we explicitly have to specify device_type. In all of our other devices it wasn't really needed.....is it because pci is default? | |
| 09:21:52 | stephenfin | awalende: https://github.com/openstack/nova/blob/master/nova/pci/stats.py#L289-L297 | |
| 09:22:43 | stephenfin | If you have a device that supports SR-IOV, the assumption we've made is that you'll generally want to use that capability because of the scalability it offers | |
| 09:23:33 | awalende | Uhhh just a minor complaint....this isn't documented :D :D. But nice, the T4 is now there in the guest. Thanks all for helping us! | |
| 09:23:44 | stephenfin | So by default we don't allow you to use the parent/root/PF device because if you do, you won't be able to use the child/VF device(s) | |
| 09:24:11 | stephenfin | Agreed. I'm working on a patch to do that now. I'll stick you on the review once it's up, if you like | |
| 09:24:24 | awalende | sure thing :D | |
| 09:27:09 | awalende | irc name is launchpad name | |
| 09:33:30 | huaqiang | sean-k-mooney: about the per-vm-pci-NUMA-policy spec, I want to confirm that it defines a policy applied to all PCI devices for whole VM, right? | |
| 09:43:51 | huaqiang | sean-k-mooney: seems I have sent my questions we have drawn a conclusion agian by mistake, pls ignore it. | |
| 10:01:53 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP block_device: Use original volume_type when creating volumes from snapshots https://review.opendev.org/694497 | |
| 10:21:26 | openstackgerrit | Liang Fang proposed openstack/nova-specs master: Support volume local cache https://review.opendev.org/689070 | |
| 11:01:22 | openstackgerrit | Lee Yarwood proposed openstack/nova-specs master: Virtual instance rescue with stable disk devices https://review.opendev.org/693849 | |
| 11:01:23 | openstackgerrit | Lee Yarwood proposed openstack/nova-specs master: Boot from volume instance rescue https://review.opendev.org/694063 | |
| 11:46:01 | openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Change order of PCI configuration steps https://review.opendev.org/694521 | |
| 11:46:01 | openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Clarify configuration steps for PF devices https://review.opendev.org/694522 | |
| 11:46:04 | stephenfin | awalende: ^ | |
| 11:46:14 | stephenfin | Also sean-k-mooney, adrianc: ^ | |
| 11:46:30 | stephenfin | I'd like to backport that as far as we can go | |
| 11:48:18 | openstackgerrit | Merged openstack/nova master: Reset vm_state to original value if rebuild claim fails https://review.opendev.org/692185 | |
| 11:48:51 | openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Clarify configuration steps for PF devices https://review.opendev.org/694522 | |
| 11:53:52 | openstackgerrit | Merged openstack/nova master: FUP for Ib62ac0b692eb92a2ed364ec9f486ded05def39ad https://review.opendev.org/693556 | |
| 12:19:02 | sean-k-mooney | stephenfin: im not sure if that is correct | |