| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-23 | |||
| 10:16:57 | pvc_ | bauzas hi | |
| 10:22:34 | pvc_ | can help me regarding vgpu bauzas? | |
| 10:22:59 | bauzas | pvc_: I'm pretty busy today with specs reviews and writing, can we discuss other days ? | |
| 10:23:34 | bauzas | pvc_: anyway, looking at your paste, doesn't seem related to nova at all | |
| 10:23:38 | pvc_ | noted on this i cant install the driver | |
| 10:23:41 | pvc_ | thank you | |
| 10:32:53 | stephenfin | Oh, today is spec review day. Oops | |
| 10:36:01 | pvc_ | can i ask if i need the vfio-pci or not? | |
| 10:38:17 | pvc_ | this one NVIDIA-Linux-x86_64-390.72-vgpu-kvm.run? | |
| 10:50:35 | pvc_ | how can i add now a vgpu to use by my instance? | |
| 10:54:36 | sean-k-mooney | pvc_: you need to add a resouce class request to your instance flavor | |
| 10:54:55 | sean-k-mooney | if you do then the libvirt dirver will automatically add it to your vm | |
| 10:55:42 | sean-k-mooney | that said that assumes your host has an nvida vgpus capable graphics card and you have the correct drivers on the host to enable it | |
| 10:56:13 | sean-k-mooney | pvc_: e.g. you card needs to be on this list https://docs.nvidia.com/grid/gpus-supported-by-vgpu.html | |
| 10:57:20 | sean-k-mooney | pvc_: if your card is not on that list you cannot use the vgpu support and can only use pci passthough instead to dedicate the entire gpu to the guest | |
| 10:57:58 | pvc_ | i have this mdev | |
| 10:58:34 | pvc_ | sean-k-mooney http://paste.openstack.org/show/732798/ | |
| 10:59:35 | sean-k-mooney | those are the mdev typs not mdevs but you need to choose one to enable in teh nova config | |
| 11:00:44 | pvc_ | i already have this wait | |
| 11:01:38 | pvc_ | enabled_vgpu_types = nvidia-160 | |
| 11:01:48 | pvc_ | i add it on overcloud-compute | |
| 11:02:44 | pvc_ | Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 4c6de80f-cc4d-47c9-a63d-f72a5584ecb6. | |
| 11:03:38 | openstackgerrit | Sylvain Bauza proposed openstack/nova-specs master: Proposes NUMA topology with RPs https://review.openstack.org/552924 | |
| 11:03:39 | pvc_ | do i need to do something sean-k-mooney? | |
| 11:03:40 | sean-k-mooney | pvc_: overcloud compute are you doing a triple-o deployment? if so im assumeing the compute node is an ironic baremental node | |
| 11:03:53 | pvc_ | yes it is a tripleo-deployment | |
| 11:05:15 | sean-k-mooney | looking at the docs https://docs.openstack.org/nova/queens/admin/virtual-gpu.html i dont see anything specifically | |
| 11:06:57 | pvc_ | i already set my flavor http://paste.openstack.org/show/732799/ | |
| 11:08:26 | openstackgerrit | Merged openstack/nova master: Rename tempest-nova job to follow conventions https://review.openstack.org/612230 | |
| 11:09:27 | pvc_ | libvirtError: internal error: qemu unexpectedly closed the monitor: 2018-10-23T11:07:33.700541Z qemu-kvm: -device vfio-pci,id=hostdev0,sysfsdev=/sys/bus/mdev/devices/9d0df47b-1fc5-4868-bd39-3cf301918e78,bus=pci.0,addr=0x5: vfio error: 9d0df47b-1fc5-4868-bd39-3cf301918e78: error getting device from group 58: Input/output error | |
| 11:10:03 | pvc_ | sean-k-mooney http://paste.openstack.org/show/732800/ | |
| 11:10:54 | sean-k-mooney | pvc_: that look like you have having issue with your iommu config | |
| 11:11:15 | openstackgerrit | Merged openstack/nova master: Use assertRegex instead of assertRegexpMatches https://review.openstack.org/611608 | |
| 11:11:22 | sean-k-mooney | just to check a few things you are useing kvm as the hyptervior correct | |
| 11:11:29 | sean-k-mooney | can you show me your kernel commandline | |
| 11:11:52 | sean-k-mooney | you should have intel_iommu=on and iommu=pt set | |
| 11:12:22 | sean-k-mooney | similarly you should have vt-d enabled in the bios asd you would for sriov | |
| 11:12:28 | pvc_ | http://paste.openstack.org/show/732802/ sean-k-mooney | |
| 11:13:04 | sean-k-mooney | but my guess is that the slot the gpu is in is shareing an iommu group with another device | |
| 11:13:48 | sean-k-mooney | pvc_: add iommu=pt to the host as a starting point. | |
| 11:13:52 | pvc_ | wait | |
| 11:13:55 | pvc_ | i add it then reboot | |
| 11:14:15 | sean-k-mooney | after you regenerate your grub file yes | |
| 11:14:18 | pvc_ | i dont need the vfio_pci right? | |
| 11:14:30 | sean-k-mooney | you do | |
| 11:14:41 | pvc_ | vfio-pci module? | |
| 11:14:46 | sean-k-mooney | mdev stands for vfio mediated device | |
| 11:15:09 | pvc_ | i disable it since i cannot install the NVIDIA-kvm driver | |
| 11:16:04 | sean-k-mooney | if you disable it then kvm cannot pass through any device that depend on the kernel vfio framework which included mdevs | |
| 11:16:24 | pvc_ | okay i'll set it back | |
| 11:16:25 | sean-k-mooney | did you follow instruction that said you should disable it from nvidia? | |
| 11:16:33 | pvc_ | but my compute node already have the nvidia driver | |
| 11:16:52 | pvc_ | because earlier i cant install the nvidia-driver | |
| 11:17:00 | pvc_ | after disabled the vfio_pci, i successfully installed it | |
| 11:17:04 | pvc_ | i will get it back | |
| 11:18:15 | sean-k-mooney | so looking at https://images.nvidia.com/content/grid/pdf/GRID-vGPU-User-Guide.pdf in section 3.1 it does not say anything about disableing vfio so i would guess that is the issue | |
| 11:18:46 | pvc_ | i will get it back sean-k-mooney wait | |
| 11:19:16 | pvc_ | so install the driver then enable it. I enable it first after installing the driver | |
| 11:22:37 | pvc_ | sean-k-mooney can i add all the available vgpu on the nova.conf? | |
| 11:23:06 | pvc_ | then create a flavor that have 1 VGPU, 2 VGPU, 3 VGPU? | |
| 11:25:23 | sean-k-mooney | pvc_: you can only have 1 vgpu per vm currently and you can only enable 1 vgpu mdev type per phyical host | |
| 11:26:40 | sean-k-mooney | part of the limitation comes form libvirt and part form nvidia. multiple mdevs can be attached to the same instance but its a rather new addtion to the kernel and things have not really mautred yet in libvirt/qemu | |
| 11:26:43 | pvc_ | so i cannot use the 24gpus on my physical host? | |
| 11:27:24 | jangutter | sean-k-mooney: minor correction, intel_iommu=on is required, and "iommu=pt" is discouraged. | |
| 11:28:16 | jangutter | sean-k-mooney: iommu=pt used to be required when DPDK hadn't set vfio-pci as it's default transport yet, and people used uio. | |
| 11:29:56 | sean-k-mooney | jangutter: yes but i still recommend iommu=pt as the iommu is picky somethimes and i find you hit less corner cases if you limit its scope to pasthrough devices | |
| 11:30:28 | pvc_ | ERROR nova.compute.manager [instance: 014fcc5f-660b-40df-92ed-9f4587993fa7] Verify all devices in group 58 are bound to vfio-<bus> or pci-stub and not already in use | |
| 11:31:00 | sean-k-mooney | pvc_: yes so as i seaid previously i think your error is the gpus is sharingin an iommu group with anothe rdevice | |
| 11:31:18 | sean-k-mooney | pvc_: all devices in a iommu group must use the same kernel driver | |
| 11:31:22 | jangutter | sean-k-mooney: heh, iommu=pt is one of the worst-named options. The "passthrough" mapping it enables is a way to 'bypass' the IOMMU by creating a 1:1 memory map for the PCI space. | |
| 11:31:50 | pvc_ | how can i check that sean-k-mooney? | |
| 11:31:59 | sean-k-mooney | jangutter: yes and that 1:1 mapping fixes so manny things :P | |
| 11:32:01 | jangutter | pvc_: yeah, you have to hand off _all_ the devices in an iommu group, and some platforms can't divide between them. | |
| 11:32:21 | sean-k-mooney | pvc_: am you can find this in sysfs | |
| 11:32:28 | pvc_ | sysfs? | |
| 11:32:29 | sean-k-mooney | let me see if i can remember | |
| 11:32:33 | pvc_ | thank you | |
| 11:32:53 | pvc_ | jangutter i cannot use the 12vgpus of my 1 tesla? | |
| 11:32:58 | pvc_ | only 1 vgpus? | |
| 11:33:43 | jangutter | pvc_: it depends, the chipset may allow you to pass the entire card at once, but not a portion of it. | |
| 11:34:01 | sean-k-mooney | pvc_: you can but if your tesla share an iommu group with a nic then they both need to be bound to vfio-pci or pci-stub | |
| 11:34:18 | jangutter | pvc_: the kernel documentation (low level warning) is at: https://www.kernel.org/doc/Documentation/vfio.txt | |
| 11:34:38 | pvc_ | thankyou jangutter, how can i do that sean-k-mooney? | |
| 11:34:53 | pvc_ | sorry this is my first time using vgpu, im using just the pci-passthroigh | |
| 11:35:05 | sean-k-mooney | first we need to see what is in iommu group 58 in /sys/class/iommu/ | |
| 11:35:17 | jangutter | pvc_: you can do something like: ls -l /sys/bus/pci/devices/0000:06:0d.0/iommu_group/devices | |
| 11:35:29 | jangutter | pvc_: where the pci address is obviously yours. | |
| 11:35:45 | jangutter | pvc_: that's a list of PCI devices in one group | |
| 11:36:19 | jangutter | pvc: they _all_ have to be passed through together, or it will fail. | |
| 11:36:37 | pvc_ | lrwxrwxrwx. 1 root root 0 Oct 23 11:36 0000:06:00.0 -> ../../../../devices/pci0000:00/0000:00:02.0/0000:06:00.0 | |
| 11:37:27 | jangutter | Also check /sys/kernel/iommu_groups/58 ? | |
| 11:37:38 | openstackgerrit | Elod Illes proposed openstack/nova master: Transform scheduler.select_destinations notification https://review.openstack.org/508506 | |
| 11:38:01 | pvc_ | 979d010e-17a3-4ac9-987a-565f9ba4b4a6 | |
| 11:38:07 | pvc_ | [root@overcloud-novacompute-0 iommu]# ls /sys/kernel/iommu_groups/58/devices/ 979d010e-17a3-4ac9-987a-565f9ba4b4a6 | |
| 11:39:07 | jangutter | pvc_: interesting... I haven't seen a UUID there yet. can you ls -l it? | |
| 11:39:27 | sean-k-mooney | jangutter: my guess is the uuid is a mdev uuid | |
| 11:39:43 | pvc_ | http://paste.openstack.org/show/732803/ | |
| 11:41:30 | pvc_ | mdev bus types plus driver http://paste.openstack.org/show/732805/ | |
| 11:42:58 | claudiub | heyo. since it's spec review day, could you take a look again at the live-resize one? https://review.openstack.org/#/c/141219/ | |
| 11:43:05 | jangutter | pvc_, sean-k-mooney: should the PCI passthrough libxml element set "managed=true"? | |