| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-23 | |||
| 11:48:08 | bauzas | I'm confused | |
| 11:48:25 | jangutter | pvc_: what does "readlink /sys/bus/pci/devices/0000:06:00.0" say? | |
| 11:48:32 | bauzas | virtual GPU or GPU passthrough? | |
| 11:48:45 | sean-k-mooney | bauzas: pvc_ has a tesla gp100 and is trying to ues the mdev based virtual gpu | |
| 11:49:36 | bauzas | then don't do vfio bus | |
| 11:49:44 | bauzas | or pci stub | |
| 11:50:06 | bauzas | just use the nvidia kernel module | |
| 11:50:10 | jangutter | aaah, the penny drops. | |
| 11:50:11 | sean-k-mooney | pvc_: can you provide the libvirt xml that nova generated so we can see what its doing | |
| 11:50:37 | pvc_ | i add an option of options vfio-pci ids=10de:15f8 should i disable this? | |
| 11:50:44 | pvc_ | then new error is occured | |
| 11:50:49 | bauzas | sean-k-mooney : I think pvc_ is mixing two different things | |
| 11:50:55 | pvc_ | 2018-10-23 11:49:46.754 7 WARNING nova.virt.libvirt.driver [req-a3570604-eed3-4f8f-a244-202ac2b92b7d - - - - -] Error from libvirt while getting description of instance-00000001: [Error Code 42] Domain not found: no domain with matching uuid '2ac6c395-5f92-4e9e-a52a-cf90b9d551c5' (instance-00000001): libvirtError: Domain not found: no domain with matching uuid '2ac6c395-5f92-4e9e-a52a-cf90b9d551c5' (instance-00000001) | |
| 11:50:57 | openstackgerrit | Jim Rollenhagen proposed openstack/nova-specs master: Use conductor groups to partition nova-compute services for Ironic https://review.openstack.org/609709 | |
| 11:51:34 | sean-k-mooney | pvc_: you should not be disableing or foceing vfio-pci | |
| 11:52:11 | sean-k-mooney | you should allow it to be loaded if need but oterwise do not set any options for the vfio kernel module at all | |
| 11:53:32 | pvc_ | http://paste.openstack.org/show/732806/ | |
| 11:54:06 | bauzas | yup this | |
| 11:54:08 | sean-k-mooney | pvc_: the phyical gpu needs to be bound to the nvidia grid driver and the mdevs will be created via the vfio framwork in the kenel but you should not force the pgpu to use vfio-pci | |
| 11:54:22 | pvc_ | i need to remove that? | |
| 11:54:24 | bauzas | yup it's only for GPU passthrough | |
| 11:54:27 | pvc_ | okay wait | |
| 11:54:30 | bauzas | hence my confusion | |
| 11:54:30 | pvc_ | i'll remove | |
| 11:54:56 | sean-k-mooney | bauzas: im guessing the driver in use should be nvidia_vgpu_vfio or nvida correct | |
| 11:55:10 | sean-k-mooney | praobly nvidia_vgpu_vfio | |
| 11:55:15 | pvc_ | i reboot again the hypervisor wait | |
| 11:56:52 | bauzas | sean-k-mooney, I don't remember the module name but yeah something like that | |
| 11:57:30 | pvc_ | bauzas is it possible to use the 12vgpus of my gpu? | |
| 11:57:42 | bauzas | FWIW, I'll have connection issues this afternoon due to some planned outage in my street | |
| 11:58:13 | sean-k-mooney | well of the 3 nouveau, nvidia_vgpu_vfio, nvidia. nouveau is the opensouce driver for the pgpu, nvidia is the binary driver from nvida for the same so that just leaves nvidia_vgpu_vfio | |
| 11:59:51 | sean-k-mooney | pvc_: that depends on the mdev type you selected. but you should be able to however you can only request 1 vgpu per guest currently | |
| 12:00:03 | pvc_ | 06:00.0 3D controller [0302]: NVIDIA Corporation GP100GL [Tesla P100 PCIe 16GB] [10de:15f8] (rev a1) Subsystem: NVIDIA Corporation Device [10de:118f] Kernel driver in use: nvidia | |
| 12:00:23 | pvc_ | it's already nvidia | |
| 12:00:37 | pvc_ | http://paste.openstack.org/show/732807/ | |
| 12:01:10 | sean-k-mooney | pvc_: yes but it may need to be nvidia_vgpu_vfio. nvidia is gust the normal binary driver for using the gpu on the host | |
| 12:01:31 | sean-k-mooney | best thing to do is try and boot a vm and see what happens | |
| 12:02:10 | pvc_ | same error :( http://paste.openstack.org/show/732808/ | |
| 12:02:54 | pvc_ | http://paste.openstack.org/show/732808/ bauzas and sean-k-mooney | |
| 12:04:33 | sean-k-mooney | pvc_: in this case try unbinding the card from nvdia driver and bind it to nvidia_vgpu_vfio | |
| 12:04:51 | pvc_ | noted on this | |
| 12:04:54 | pvc_ | i add this on module | |
| 12:05:02 | pvc_ | options nvidia_vgpu_vfio ids=10de:15f8 | |
| 12:05:39 | sean-k-mooney | can yo bind it by hand instad of via the moduel file to test it | |
| 12:05:57 | pvc_ | how can i bind it? i'm sorry im not done it before | |
| 12:05:59 | bauzas | that's super weird | |
| 12:06:33 | pvc_ | this is my conf | |
| 12:06:33 | pvc_ | http://paste.openstack.org/show/732809/ | |
| 12:07:16 | bauzas | pvc_ did you remove the nouveau driver ? | |
| 12:07:32 | sean-k-mooney | echo 06:00.0 | sudo tee /sys/bus/pci/drivers/nvidia/unbind | |
| 12:07:50 | sean-k-mooney | echo 06:00.0 | sudo tee /sys/bus/pci/drivers/nvidia_vgpu_vfio/bind | |
| 12:07:53 | pvc_ | [root@overcloud-novacompute-0 nova]# lsmod | grep nou [root@overcloud-novacompute-0 nova]# | |
| 12:07:56 | pvc_ | yes bauzas | |
| 12:08:48 | pvc_ | tee: /sys/bus/pci/drivers/nvidia/unbind: No such device | |
| 12:09:00 | pvc_ | I install this driver | |
| 12:09:09 | pvc_ | NVIDIA-Linux-x86_64-390.72-vgpu-kvm.run | |
| 12:09:15 | bauzas | I need to drop, planned outage here | |
| 12:10:01 | sean-k-mooney | pvc_: from http://paste.openstack.org/show/732807/ that should have been the driver in use | |
| 12:10:10 | pvc_ | i use this 0000:06:00.0 | |
| 12:10:31 | pvc_ | tee: /sys/bus/pci/drivers/nvidia_vgpu_vfio/bind: No such file or directory | |
| 12:10:43 | pvc_ | it is already unbind | |
| 12:11:04 | pvc_ | no nvidia_vgpu_vfio on drivers | |
| 12:11:08 | pvc_ | just nvidia | |
| 12:14:16 | sean-k-mooney | pvc_: this is the latest verion of the nvdia vgpu user guide https://docs.nvidia.com/grid/5.0/pdf/grid-vgpu-user-guide.pdf i think you need to back through it and section 4.2 specifically | |
| 12:14:33 | pvc_ | sean-k-mooney im using a ubuntu image with img_hide_hypervisor_id='true' | |
| 12:15:13 | sean-k-mooney | pvc_: i asked thi earliar but is the compute node a phyical server or a vm | |
| 12:15:34 | pvc_ | compute node is a physical server | |
| 12:15:42 | pvc_ | on docs it said the grid driver | |
| 12:15:48 | pvc_ | but this is the driver i installed | |
| 12:16:01 | pvc_ | installed NVIDIA-Linux-x86_64-390.72-vgpu-kvm.run | |
| 12:16:10 | pvc_ | i have this grid driver NVIDIA-Linux-x86_64-390.75-grid.run | |
| 12:17:03 | sean-k-mooney | pvc_: oh ok i think i understand the issue then | |
| 12:17:12 | sean-k-mooney | you install the guest driver on the host | |
| 12:17:25 | pvc_ | yes sean-k-mooney | |
| 12:17:25 | pvc_ | to enable this | |
| 12:17:31 | pvc_ | /sys/class/mdev_bus/*/mdev_supported_types | |
| 12:18:04 | pvc_ | i install the driver on my compute node ( baremetal server ) im using a tripleo-deployment | |
| 12:22:26 | aperevalov | hello, do nova or neutron has functional test for direct (SR-IOV) port (something like tempest test)? | |
| 12:22:26 | pvc_ | sean-k-mooney what will i do then? | |
| 12:23:37 | sean-k-mooney | aperevalov: i dont belive so in upstream tempest. neutron may have fullstack test but ingerneral our sriov testing is limited | |
| 12:24:58 | sean-k-mooney | pvc_: this seams to be a driver issue not a nova one. there is little more advice i can give other then they to follow the nvida docs form start to finish exactly | |
| 12:25:01 | aperevalov | sean-k-mooney: I assume, if such functional test exists it uses real HW with SR-IOV, but not simulators or emulators. | |
| 12:25:24 | pvc_ | so you think i installed the grid one? | |
| 12:25:31 | pvc_ | my baremetal is centos 7 | |
| 12:25:34 | sean-k-mooney | aperevalov: so the fact we can use simulator/emulators for sriov is the main reason we have such limited testing | |
| 12:25:58 | sean-k-mooney | pvc_: yes you need to install the grid one | |
| 12:26:13 | sean-k-mooney | at least i think so. | |
| 12:26:26 | pvc_ | okay wait | |
| 12:27:09 | sean-k-mooney | aperevalov: i recently looked at using the netdevsim kernel module for sriov testing but it only emulates the kernel netdevs it does not emulate the pci devices or virtual function so we cant use it for testing | |
| 12:28:40 | aperevalov | sean-k-mooney: I checked netdevsim on the latest kernel too, there are no device to put it into docker container. Docker container it's because I'm doing such research for kuryr-kubernetes. | |
| 12:29:49 | aperevalov | sean-k-mooney: yes netdevsim is based on its own bus, but not pci. I also found attemp to submit to QEMU's pci emulation the SR-IOV support. | |
| 12:30:36 | sean-k-mooney | aperevalov: yes so there is a netdev but no pci device on the virtual pci device. so you can use ip link and allocate vfs via sysfs but they dont show up on the virtual pci bus | |
| 12:31:20 | sean-k-mooney | aperevalov: yes i have see that in the past but that wont help us with testing as we would need the hosting vms provided by the could providers to emulate pci device that support sriov | |
| 12:31:30 | aperevalov | sean-k-mooney: but it was postponed due to lack of existing working qemu drivers, initial author suggested copied e1000, but it wasn't in working condition. | |
| 12:32:21 | sean-k-mooney | aperevalov: so first qemu would have to be extended then libvirt and then nova. once that is done we would need cloud providers to proved devices with virt sriov capable nics then we could start using it in the upstream gate | |
| 12:32:41 | sean-k-mooney | *provided vms with ... | |
| 12:33:29 | sean-k-mooney | aperevalov: effectivly we rely on thirdpart cis to test sriov. either intels or melonox's ci | |
| 12:33:54 | sean-k-mooney | intels ci was intented to have signifcatily more sriov testing then it currently has but that never happened | |
| 12:34:31 | aperevalov | sean-k-mooney, I see it's a long way, and seems netdevsim (just kernel module) looks like easiest (if kernel community will be agreed to bind it with pci bus). | |
| 12:35:50 | sean-k-mooney | aperevalov: most people dont know that sr-iov is a specificaiton from the PCI-SIG i dont think the current netdevsim moudle is technically a confroming sriov implentaiton without the pci emulation | |