Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-10-23
11:35:29 jangutter pvc_: where the pci address is obviously yours.
11:35:45 jangutter pvc_: that's a list of PCI devices in one group
11:36:19 jangutter pvc: they _all_ have to be passed through together, or it will fail.
11:36:37 pvc_ lrwxrwxrwx. 1 root root 0 Oct 23 11:36 0000:06:00.0 -> ../../../../devices/pci0000:00/0000:00:02.0/0000:06:00.0
11:37:27 jangutter Also check /sys/kernel/iommu_groups/58 ?
11:37:38 openstackgerrit Elod Illes proposed openstack/nova master: Transform scheduler.select_destinations notification https://review.openstack.org/508506
11:38:01 pvc_ 979d010e-17a3-4ac9-987a-565f9ba4b4a6
11:38:07 pvc_ [root@overcloud-novacompute-0 iommu]# ls /sys/kernel/iommu_groups/58/devices/ 979d010e-17a3-4ac9-987a-565f9ba4b4a6
11:39:07 jangutter pvc_: interesting... I haven't seen a UUID there yet. can you ls -l it?
11:39:27 sean-k-mooney jangutter: my guess is the uuid is a mdev uuid
11:39:43 pvc_ http://paste.openstack.org/show/732803/
11:41:30 pvc_ mdev bus types plus driver http://paste.openstack.org/show/732805/
11:42:58 claudiub heyo. since it's spec review day, could you take a look again at the live-resize one? https://review.openstack.org/#/c/141219/
11:43:05 jangutter pvc_, sean-k-mooney: should the PCI passthrough libxml element set "managed=true"?
11:43:30 jangutter sean-k-mooney: managed=true generally means that it will auto-bind vfio-pci to the device before attempting passthrough?
11:43:42 bauzas jangutter, sean-k-mooney: context ?
11:43:45 sean-k-mooney jangutter: sorry where was the libvirt xml i missed that
11:43:58 sean-k-mooney bauzas: trying to help pvc_ with there vgpu issue
11:44:15 bauzas and?
11:44:16 sean-k-mooney bauzas: pvc_ is seeing a weird iommu error
11:44:22 jangutter hang on, looking up the libxml doc.
11:44:45 bauzas we don't manage the iommu group
11:45:05 jangutter (rofl) s/libxml/libvirt/
11:45:32 sean-k-mooney bauzas: yes that is managed by the uefi and kernel
11:46:12 bauzas oh wait
11:46:13 jangutter sean-k-mooney: when doing https://libvirt.org/formatdomain.html#elementsHostDevSubsys <--- there's a 'managed=yes' element in the xml. if that's set it will do the vfio-pci binding for you.
11:46:24 bauzas is pvc_ doing PXI
11:46:26 sean-k-mooney bauzas: pvc_ is getting Verify all devices in group 58 are bound to vfio-<bus> or pci-stub and not already in use
11:46:36 bauzas pci passthrough?
11:47:02 pvc_ im using vgpu bauzas
11:47:03 sean-k-mooney bauzas: no pvc_ is trying to do vgpu passthrouhg not pci
11:47:29 sean-k-mooney bauzas: this is there flavor http://paste.openstack.org/show/732799/
11:47:58 bauzas sec, GPU passthrough?
11:47:59 sean-k-mooney and pvc_ has set the gpu type in the config to nvidia-160
11:48:08 bauzas I'm confused
11:48:25 jangutter pvc_: what does "readlink /sys/bus/pci/devices/0000:06:00.0" say?
11:48:32 bauzas virtual GPU or GPU passthrough?
11:48:45 sean-k-mooney bauzas: pvc_ has a tesla gp100 and is trying to ues the mdev based virtual gpu
11:49:36 bauzas then don't do vfio bus
11:49:44 bauzas or pci stub
11:50:06 bauzas just use the nvidia kernel module
11:50:10 jangutter aaah, the penny drops.
11:50:11 sean-k-mooney pvc_: can you provide the libvirt xml that nova generated so we can see what its doing
11:50:37 pvc_ i add an option of options vfio-pci ids=10de:15f8 should i disable this?
11:50:44 pvc_ then new error is occured
11:50:49 bauzas sean-k-mooney : I think pvc_ is mixing two different things
11:50:55 pvc_ 2018-10-23 11:49:46.754 7 WARNING nova.virt.libvirt.driver [req-a3570604-eed3-4f8f-a244-202ac2b92b7d - - - - -] Error from libvirt while getting description of instance-00000001: [Error Code 42] Domain not found: no domain with matching uuid '2ac6c395-5f92-4e9e-a52a-cf90b9d551c5' (instance-00000001): libvirtError: Domain not found: no domain with matching uuid '2ac6c395-5f92-4e9e-a52a-cf90b9d551c5' (instance-00000001)
11:50:57 openstackgerrit Jim Rollenhagen proposed openstack/nova-specs master: Use conductor groups to partition nova-compute services for Ironic https://review.openstack.org/609709
11:51:34 sean-k-mooney pvc_: you should not be disableing or foceing vfio-pci
11:52:11 sean-k-mooney you should allow it to be loaded if need but oterwise do not set any options for the vfio kernel module at all
11:53:32 pvc_ http://paste.openstack.org/show/732806/
11:54:06 bauzas yup this
11:54:08 sean-k-mooney pvc_: the phyical gpu needs to be bound to the nvidia grid driver and the mdevs will be created via the vfio framwork in the kenel but you should not force the pgpu to use vfio-pci
11:54:22 pvc_ i need to remove that?
11:54:24 bauzas yup it's only for GPU passthrough
11:54:27 pvc_ okay wait
11:54:30 pvc_ i'll remove
11:54:30 bauzas hence my confusion
11:54:56 sean-k-mooney bauzas: im guessing the driver in use should be nvidia_vgpu_vfio or nvida correct
11:55:10 sean-k-mooney praobly nvidia_vgpu_vfio
11:55:15 pvc_ i reboot again the hypervisor wait
11:56:52 bauzas sean-k-mooney, I don't remember the module name but yeah something like that
11:57:30 pvc_ bauzas is it possible to use the 12vgpus of my gpu?
11:57:42 bauzas FWIW, I'll have connection issues this afternoon due to some planned outage in my street
11:58:13 sean-k-mooney well of the 3 nouveau, nvidia_vgpu_vfio, nvidia. nouveau is the opensouce driver for the pgpu, nvidia is the binary driver from nvida for the same so that just leaves nvidia_vgpu_vfio
11:59:51 sean-k-mooney pvc_: that depends on the mdev type you selected. but you should be able to however you can only request 1 vgpu per guest currently
12:00:03 pvc_ 06:00.0 3D controller [0302]: NVIDIA Corporation GP100GL [Tesla P100 PCIe 16GB] [10de:15f8] (rev a1) Subsystem: NVIDIA Corporation Device [10de:118f] Kernel driver in use: nvidia
12:00:23 pvc_ it's already nvidia
12:00:37 pvc_ http://paste.openstack.org/show/732807/
12:01:10 sean-k-mooney pvc_: yes but it may need to be nvidia_vgpu_vfio. nvidia is gust the normal binary driver for using the gpu on the host
12:01:31 sean-k-mooney best thing to do is try and boot a vm and see what happens
12:02:10 pvc_ same error :( http://paste.openstack.org/show/732808/
12:02:54 pvc_ http://paste.openstack.org/show/732808/ bauzas and sean-k-mooney
12:04:33 sean-k-mooney pvc_: in this case try unbinding the card from nvdia driver and bind it to nvidia_vgpu_vfio
12:04:51 pvc_ noted on this
12:04:54 pvc_ i add this on module
12:05:02 pvc_ options nvidia_vgpu_vfio ids=10de:15f8
12:05:39 sean-k-mooney can yo bind it by hand instad of via the moduel file to test it
12:05:57 pvc_ how can i bind it? i'm sorry im not done it before
12:05:59 bauzas that's super weird
12:06:33 pvc_ http://paste.openstack.org/show/732809/
12:06:33 pvc_ this is my conf
12:07:16 bauzas pvc_ did you remove the nouveau driver ?
12:07:32 sean-k-mooney echo 06:00.0 | sudo tee /sys/bus/pci/drivers/nvidia/unbind
12:07:50 sean-k-mooney echo 06:00.0 | sudo tee /sys/bus/pci/drivers/nvidia_vgpu_vfio/bind
12:07:53 pvc_ [root@overcloud-novacompute-0 nova]# lsmod | grep nou [root@overcloud-novacompute-0 nova]#
12:07:56 pvc_ yes bauzas
12:08:48 pvc_ tee: /sys/bus/pci/drivers/nvidia/unbind: No such device
12:09:00 pvc_ I install this driver
12:09:09 pvc_ NVIDIA-Linux-x86_64-390.72-vgpu-kvm.run
12:09:15 bauzas I need to drop, planned outage here
12:10:01 sean-k-mooney pvc_: from http://paste.openstack.org/show/732807/ that should have been the driver in use
12:10:10 pvc_ i use this 0000:06:00.0
12:10:31 pvc_ tee: /sys/bus/pci/drivers/nvidia_vgpu_vfio/bind: No such file or directory
12:10:43 pvc_ it is already unbind
12:11:04 pvc_ no nvidia_vgpu_vfio on drivers
12:11:08 pvc_ just nvidia
12:14:16 sean-k-mooney pvc_: this is the latest verion of the nvdia vgpu user guide https://docs.nvidia.com/grid/5.0/pdf/grid-vgpu-user-guide.pdf i think you need to back through it and section 4.2 specifically
12:14:33 pvc_ sean-k-mooney im using a ubuntu image with img_hide_hypervisor_id='true'
12:15:13 sean-k-mooney pvc_: i asked thi earliar but is the compute node a phyical server or a vm

Earlier   Later