| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-11-15 | |||
| 08:52:06 | stephenfin | have you tried (b)? | |
| 08:52:16 | stephenfin | to do that, you'll need to also set the 'product_id' in the 'passthrough_whitelist' | |
| 08:52:44 | stephenfin | to whatever the product_id of the PF is, assuming the PF and VF have different product_id's (they should) | |
| 08:52:54 | awalende | my colleague means he "maybe" tried it as well. We try it again and let you know | |
| 08:53:23 | stephenfin | If they don't, you're going to need to use the 'address' field, I'm afraid | |
| 08:53:31 | stephenfin | okay, do please | |
| 08:53:59 | stephenfin | sean-k-mooney might be able to help too but they're definitely fast asleep atm :) | |
| 08:54:13 | awalende | How do you make sure that "VFs are not whitelisted"? | |
| 08:55:08 | bauzas | the T4 ? | |
| 08:55:19 | bauzas | awalende: which grid driver are you using ? | |
| 08:55:25 | bauzas | GRID8 ? | |
| 08:55:53 | awalende | no driver at all, we plugged the T4 into our hypervisor nodes and want to pass them through to guests | |
| 08:56:05 | stephenfin | awalende: If SR-IOV is enabled on the device, you will have additional PCI devices appearing at e.g. 'lspci' | |
| 08:56:19 | awalende | one moment | |
| 08:57:06 | stephenfin | 'lspci | grep -i nvidia' should flag them, I'd imagine | |
| 08:57:07 | bauzas | oh right, then that's normal | |
| 08:57:26 | bauzas | Nvidia plans to deliver virtual GPUs as VFs later | |
| 08:57:36 | bauzas | I guess they already started to expose the GPU as a PF | |
| 08:57:44 | awalende | stephenfin, it only shows me one card as af:00.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) | |
| 08:57:45 | stephenfin | I'm not sure how you'd turn them on/off. Probably by echoing stuff to sysfs like you do for SR-IOV NICs, I guess | |
| 08:57:48 | bauzas | but I thought the Tesla models wouldn't get it | |
| 08:57:55 | bauzas | just the Volta ones | |
| 08:58:20 | bauzas | I'm not a nVidia salesperson, (un?)fornutately :p | |
| 08:58:23 | stephenfin | okay, it's probably disabled so | |
| 08:59:02 | bauzas | awalende: just do something like 'lspci -vv | grep 10de | |
| 08:59:06 | stephenfin | awalende: So there is definitely _something_ specific you need to do in order to use type-PF devices. I'm just trying to figure out what it is | |
| 08:59:53 | awalende | bauzas, returns empty | |
| 08:59:59 | bauzas | mmm | |
| 09:00:04 | stephenfin | awalende: This is what I'm thinking of https://github.com/openstack/nova/blob/master/nova/pci/stats.py#L289-L297 | |
| 09:00:19 | bauzas | awalende: are you OK with pasting your whole lspci output ? | |
| 09:00:49 | stephenfin | so if I'm reading that correctly, you *must* set 'dev_type': 'type-PF' in the alias | |
| 09:01:22 | awalende | which lspci command to you want pasted? 'lspci -vv'? | |
| 09:01:22 | bauzas | I agree | |
| 09:01:33 | bauzas | awalende: yes please | |
| 09:02:09 | bauzas | stephenfin: looks like you need to expose your type-PF this way | |
| 09:02:22 | stephenfin | wdym? | |
| 09:02:27 | bauzas | but I'm amazed that nvidia provides a PF and not a straight PCI ID | |
| 09:03:02 | bauzas | stephenfin: you have to expose the PCI devices in nova.conf in order to allow Nova to use them, right? | |
| 09:03:22 | stephenfin | yeah, awalende has done that mostly correctly I think | |
| 09:03:51 | awalende | https://paste.ubuntu.com/p/xXkRfFMRYh/ | |
| 09:04:12 | stephenfin | the '[pci] passthrough_whitelist' on the compute node is correct, but the '[pci] alias' on the controller and compute node doesn't appear to be (it needs 'dev_type') | |
| 09:04:43 | awalende | stephenfin, we will try soon! | |
| 09:05:09 | awalende | does dev_type needs to be specified on controller and compute? | |
| 09:05:48 | stephenfin | sorry, 'device_type' | |
| 09:06:01 | stephenfin | in the 'alias' option on both | |
| 09:06:22 | awalende | ok, we'll try it | |
| 09:06:38 | stephenfin | alias={"name":"T4","vendor_id":"10de","product_id":"1eb8","device_type":"type-PF"} | |
| 09:06:40 | stephenfin | or something like that | |
| 09:07:56 | bauzas | very interesting lspci | |
| 09:08:11 | bauzas | https://paste.ubuntu.com/p/xXkRfFMRYh/ L3241 | |
| 09:08:28 | bauzas | it shows a SR-IOV capability | |
| 09:08:35 | bauzas | but no VGA capability (!) | |
| 09:08:56 | bauzas | that's beyond my comfort zone | |
| 09:09:02 | stephenfin | yeah, it's an accelerator, not a GPU | |
| 09:09:10 | bauzas | my bad then | |
| 09:09:18 | bauzas | I'm not familiar with the product line | |
| 09:09:45 | bauzas | so we're just talking of PCI passthrugh, not VGA passthru, my bad | |
| 09:10:08 | bauzas | anyway, it doesn't expose a standard PCI address | |
| 09:12:49 | bauzas | I just feel that stephenfin is absolutely right, nova punts the address because of the internal conditional he said | |
| 09:15:21 | awalende | someone rebooted the server =.= one more minute | |
| 09:17:19 | awalende | ok, its spawning :D fingers crossed | |
| 09:20:53 | awalende | Ok, the device shows up in the guest | |
| 09:20:54 | awalende | hooray | |
| 09:21:29 | awalende | But why do we explicitly have to specify device_type. In all of our other devices it wasn't really needed.....is it because pci is default? | |
| 09:21:52 | stephenfin | awalende: https://github.com/openstack/nova/blob/master/nova/pci/stats.py#L289-L297 | |
| 09:22:43 | stephenfin | If you have a device that supports SR-IOV, the assumption we've made is that you'll generally want to use that capability because of the scalability it offers | |
| 09:23:33 | awalende | Uhhh just a minor complaint....this isn't documented :D :D. But nice, the T4 is now there in the guest. Thanks all for helping us! | |
| 09:23:44 | stephenfin | So by default we don't allow you to use the parent/root/PF device because if you do, you won't be able to use the child/VF device(s) | |
| 09:24:11 | stephenfin | Agreed. I'm working on a patch to do that now. I'll stick you on the review once it's up, if you like | |
| 09:24:24 | awalende | sure thing :D | |
| 09:27:09 | awalende | irc name is launchpad name | |
| 09:33:30 | huaqiang | sean-k-mooney: about the per-vm-pci-NUMA-policy spec, I want to confirm that it defines a policy applied to all PCI devices for whole VM, right? | |
| 09:43:51 | huaqiang | sean-k-mooney: seems I have sent my questions we have drawn a conclusion agian by mistake, pls ignore it. | |
| 10:01:53 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP block_device: Use original volume_type when creating volumes from snapshots https://review.opendev.org/694497 | |
| 10:21:26 | openstackgerrit | Liang Fang proposed openstack/nova-specs master: Support volume local cache https://review.opendev.org/689070 | |
| 11:01:22 | openstackgerrit | Lee Yarwood proposed openstack/nova-specs master: Virtual instance rescue with stable disk devices https://review.opendev.org/693849 | |
| 11:01:23 | openstackgerrit | Lee Yarwood proposed openstack/nova-specs master: Boot from volume instance rescue https://review.opendev.org/694063 | |
| 11:46:01 | openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Change order of PCI configuration steps https://review.opendev.org/694521 | |
| 11:46:01 | openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Clarify configuration steps for PF devices https://review.opendev.org/694522 | |
| 11:46:04 | stephenfin | awalende: ^ | |
| 11:46:14 | stephenfin | Also sean-k-mooney, adrianc: ^ | |
| 11:46:30 | stephenfin | I'd like to backport that as far as we can go | |
| 11:48:18 | openstackgerrit | Merged openstack/nova master: Reset vm_state to original value if rebuild claim fails https://review.opendev.org/692185 | |
| 11:48:51 | openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Clarify configuration steps for PF devices https://review.opendev.org/694522 | |
| 11:53:52 | openstackgerrit | Merged openstack/nova master: FUP for Ib62ac0b692eb92a2ed364ec9f486ded05def39ad https://review.opendev.org/693556 | |
| 12:19:02 | sean-k-mooney | stephenfin: im not sure if that is correct | |
| 12:19:23 | sean-k-mooney | if you dont set the device_type in the alias at all i think it will work with type-PF | |
| 12:19:44 | sean-k-mooney | the device_type is not required in the alias as far as i remember | |
| 12:19:56 | sean-k-mooney | if it is present it definetly needs to match | |
| 12:20:24 | sean-k-mooney | i need to test some sriov stuff later today so ill try this if i have time otherwise ill do it on monday | |
| 12:21:25 | shilpasd | gibi: hi, can you please elaborate on point 'add compute nodes to the aggregates of shared RP aggregates' regarding Shared NFS, so that will think on designing the same | |
| 12:22:56 | sean-k-mooney | the product id will be differente for the PF vs VF so if you have the correct product id and no VF are allocated i think it just works but again that is just going off memory i have not tried this in a year or two | |
| 12:25:28 | shilpasd | gibi: at driver level, how ww will get aggragtes associated to shared RP and will map those to compute node? And if compute node already has aggregates (not the same as of shared RP) are we really bother of them? | |
| 12:37:59 | openstackgerrit | Lee Yarwood proposed openstack/nova master: docs: Extract rescue from reboot https://review.opendev.org/694529 | |
| 12:39:23 | gibi | shilpasd: there are two types of aggregates: there are placement aggregates https://docs.openstack.org/api-ref/placement/#resource-provider-aggregates and there are nova host aggregates https://docs.openstack.org/api-ref/compute/#host-aggregates-os-aggregates | |
| 12:39:50 | gibi | the disk sharing RP needs to be in the same placement aggregate as the compute RP it shares disk with | |
| 12:40:34 | gibi | we agreed that a new nova config option will tell the nova compute the uuid of the placement aggregate | |
| 12:40:49 | gibi | in which the admin added the sharing disk RP | |
| 12:40:58 | openstackgerrit | Lee Yarwood proposed openstack/nova master: docs: Extract rescue from reboot https://review.opendev.org/694529 | |
| 12:41:26 | gibi | so when the compute start up it can change if its own RP (the compute PR) is already part of the placement aggregate configured in it's nova.conf | |