| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-23 | |||
| 02:59:56 | openstackgerrit | Seyeong Kim proposed openstack/nova master: Enable connection_info refresh for new-style attachments https://review.openstack.org/579004 | |
| 03:03:09 | openstackgerrit | Merged openstack/nova stable/rocky: Use nova-consoleauth only if workaround enabled https://review.openstack.org/610673 | |
| 03:10:49 | openstackgerrit | Merged openstack/nova master: conductor: Recreate volume attachments during a reschedule https://review.openstack.org/587071 | |
| 04:02:50 | openstackgerrit | Sam Morrison proposed openstack/nova stable/rocky: Fix up compute rpcapi version for pike release https://review.openstack.org/612561 | |
| 04:03:41 | openstackgerrit | Sam Morrison proposed openstack/nova stable/queens: Fix up compute rpcapi version for pike release https://review.openstack.org/612562 | |
| 04:51:34 | openstackgerrit | Seyeong Kim proposed openstack/nova master: Enable connection_info refresh for new-style attachments https://review.openstack.org/579004 | |
| 07:01:29 | aperevalov | Hello, community! Do you have functional test for Direct ports. Does this test requires HW with SR-IOV support? I can't find something similar in the neutron-tempest-test. | |
| 07:02:01 | bauzas | good morning Nova | |
| 07:02:16 | openstackgerrit | Seyeong Kim proposed openstack/nova master: Enable connection_info refresh for new-style attachments https://review.openstack.org/579004 | |
| 07:40:48 | openstackgerrit | Jan Gutter proposed openstack/nova-specs master: Spec to implement generic HW offloads for os-vif https://review.openstack.org/607610 | |
| 07:44:41 | openstackgerrit | Adrian Chiris proposed openstack/nova master: WIP - SRIOV live migration https://review.openstack.org/612620 | |
| 08:02:40 | openstackgerrit | Adrian Chiris proposed openstack/nova master: WIP - SRIOV live migration https://review.openstack.org/612620 | |
| 08:15:33 | openstackgerrit | Pooja Jadhav proposed openstack/nova master: Ignore root_gb for BFV in simple tenant usage API https://review.openstack.org/612626 | |
| 08:29:40 | openstackgerrit | Merged openstack/nova master: Convert legacy-tempest-dsvm-neutron-src-oslo.versionedobjects job https://review.openstack.org/610271 | |
| 08:37:27 | openstackgerrit | Elod Illes proposed openstack/nova master: Transform compute_task notifications https://review.openstack.org/482629 | |
| 08:43:28 | openstackgerrit | Elod Illes proposed openstack/nova master: Transform compute_task notifications https://review.openstack.org/482629 | |
| 09:33:06 | bauzas | stephenfin: good morning | |
| 09:33:21 | bauzas | stephenfin: do we have some documentation about PCI NUMA policies in https://docs.openstack.org/nova/queens/admin/adv-config.html ? | |
| 09:33:36 | bauzas | stephenfin: or shall I refer to the implemented spec https://specs.openstack.org/openstack/nova-specs/specs/queens/implemented/share-pci-between-numa-nodes.html ? | |
| 09:33:50 | bauzas | stephenfin: context is me trying to update https://review.openstack.org/#/c/552924/10/specs/stein/approved/numa-topology-with-rps.rst | |
| 09:51:47 | stephenfin | bauzas: We should have. Sec | |
| 09:55:59 | stephenfin | bauzas: This is all we have https://docs.openstack.org/nova/rocky/configuration/config.html#pci | |
| 09:56:36 | stephenfin | We should probably expand that out. As such though, the spec looks like the best option | |
| 09:58:17 | abhi89 | Hi.. whenever nova service runs a command using sudo, does this make a call to unix_chkpwd program which is part of pam_unix module? | |
| 10:01:59 | openstackgerrit | Elod Illes proposed openstack/nova master: Transform scheduler.select_destinations notification https://review.openstack.org/508506 | |
| 10:15:35 | pvc_ | hi bauzas can you help me regarding vgpu? | |
| 10:15:50 | pvc_ | http://paste.openstack.org/show/732763/ | |
| 10:16:07 | bauzas | stephenfin: ack, thanks | |
| 10:16:36 | bauzas | stephenfin: anway, it's just for explainint that it won't be modified by my spec :) | |
| 10:16:57 | pvc_ | bauzas hi | |
| 10:22:34 | pvc_ | can help me regarding vgpu bauzas? | |
| 10:22:59 | bauzas | pvc_: I'm pretty busy today with specs reviews and writing, can we discuss other days ? | |
| 10:23:34 | bauzas | pvc_: anyway, looking at your paste, doesn't seem related to nova at all | |
| 10:23:38 | pvc_ | noted on this i cant install the driver | |
| 10:23:41 | pvc_ | thank you | |
| 10:32:53 | stephenfin | Oh, today is spec review day. Oops | |
| 10:36:01 | pvc_ | can i ask if i need the vfio-pci or not? | |
| 10:38:17 | pvc_ | this one NVIDIA-Linux-x86_64-390.72-vgpu-kvm.run? | |
| 10:50:35 | pvc_ | how can i add now a vgpu to use by my instance? | |
| 10:54:36 | sean-k-mooney | pvc_: you need to add a resouce class request to your instance flavor | |
| 10:54:55 | sean-k-mooney | if you do then the libvirt dirver will automatically add it to your vm | |
| 10:55:42 | sean-k-mooney | that said that assumes your host has an nvida vgpus capable graphics card and you have the correct drivers on the host to enable it | |
| 10:56:13 | sean-k-mooney | pvc_: e.g. you card needs to be on this list https://docs.nvidia.com/grid/gpus-supported-by-vgpu.html | |
| 10:57:20 | sean-k-mooney | pvc_: if your card is not on that list you cannot use the vgpu support and can only use pci passthough instead to dedicate the entire gpu to the guest | |
| 10:57:58 | pvc_ | i have this mdev | |
| 10:58:34 | pvc_ | sean-k-mooney http://paste.openstack.org/show/732798/ | |
| 10:59:35 | sean-k-mooney | those are the mdev typs not mdevs but you need to choose one to enable in teh nova config | |
| 11:00:44 | pvc_ | i already have this wait | |
| 11:01:38 | pvc_ | enabled_vgpu_types = nvidia-160 | |
| 11:01:48 | pvc_ | i add it on overcloud-compute | |
| 11:02:44 | pvc_ | Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 4c6de80f-cc4d-47c9-a63d-f72a5584ecb6. | |
| 11:03:38 | openstackgerrit | Sylvain Bauza proposed openstack/nova-specs master: Proposes NUMA topology with RPs https://review.openstack.org/552924 | |
| 11:03:39 | pvc_ | do i need to do something sean-k-mooney? | |
| 11:03:40 | sean-k-mooney | pvc_: overcloud compute are you doing a triple-o deployment? if so im assumeing the compute node is an ironic baremental node | |
| 11:03:53 | pvc_ | yes it is a tripleo-deployment | |
| 11:05:15 | sean-k-mooney | looking at the docs https://docs.openstack.org/nova/queens/admin/virtual-gpu.html i dont see anything specifically | |
| 11:06:57 | pvc_ | i already set my flavor http://paste.openstack.org/show/732799/ | |
| 11:08:26 | openstackgerrit | Merged openstack/nova master: Rename tempest-nova job to follow conventions https://review.openstack.org/612230 | |
| 11:09:27 | pvc_ | libvirtError: internal error: qemu unexpectedly closed the monitor: 2018-10-23T11:07:33.700541Z qemu-kvm: -device vfio-pci,id=hostdev0,sysfsdev=/sys/bus/mdev/devices/9d0df47b-1fc5-4868-bd39-3cf301918e78,bus=pci.0,addr=0x5: vfio error: 9d0df47b-1fc5-4868-bd39-3cf301918e78: error getting device from group 58: Input/output error | |
| 11:10:03 | pvc_ | sean-k-mooney http://paste.openstack.org/show/732800/ | |
| 11:10:54 | sean-k-mooney | pvc_: that look like you have having issue with your iommu config | |
| 11:11:15 | openstackgerrit | Merged openstack/nova master: Use assertRegex instead of assertRegexpMatches https://review.openstack.org/611608 | |
| 11:11:22 | sean-k-mooney | just to check a few things you are useing kvm as the hyptervior correct | |
| 11:11:29 | sean-k-mooney | can you show me your kernel commandline | |
| 11:11:52 | sean-k-mooney | you should have intel_iommu=on and iommu=pt set | |
| 11:12:22 | sean-k-mooney | similarly you should have vt-d enabled in the bios asd you would for sriov | |
| 11:12:28 | pvc_ | http://paste.openstack.org/show/732802/ sean-k-mooney | |
| 11:13:04 | sean-k-mooney | but my guess is that the slot the gpu is in is shareing an iommu group with another device | |
| 11:13:48 | sean-k-mooney | pvc_: add iommu=pt to the host as a starting point. | |
| 11:13:52 | pvc_ | wait | |
| 11:13:55 | pvc_ | i add it then reboot | |
| 11:14:15 | sean-k-mooney | after you regenerate your grub file yes | |
| 11:14:18 | pvc_ | i dont need the vfio_pci right? | |
| 11:14:30 | sean-k-mooney | you do | |
| 11:14:41 | pvc_ | vfio-pci module? | |
| 11:14:46 | sean-k-mooney | mdev stands for vfio mediated device | |
| 11:15:09 | pvc_ | i disable it since i cannot install the NVIDIA-kvm driver | |
| 11:16:04 | sean-k-mooney | if you disable it then kvm cannot pass through any device that depend on the kernel vfio framework which included mdevs | |
| 11:16:24 | pvc_ | okay i'll set it back | |
| 11:16:25 | sean-k-mooney | did you follow instruction that said you should disable it from nvidia? | |
| 11:16:33 | pvc_ | but my compute node already have the nvidia driver | |
| 11:16:52 | pvc_ | because earlier i cant install the nvidia-driver | |
| 11:17:00 | pvc_ | after disabled the vfio_pci, i successfully installed it | |
| 11:17:04 | pvc_ | i will get it back | |
| 11:18:15 | sean-k-mooney | so looking at https://images.nvidia.com/content/grid/pdf/GRID-vGPU-User-Guide.pdf in section 3.1 it does not say anything about disableing vfio so i would guess that is the issue | |
| 11:18:46 | pvc_ | i will get it back sean-k-mooney wait | |
| 11:19:16 | pvc_ | so install the driver then enable it. I enable it first after installing the driver | |
| 11:22:37 | pvc_ | sean-k-mooney can i add all the available vgpu on the nova.conf? | |
| 11:23:06 | pvc_ | then create a flavor that have 1 VGPU, 2 VGPU, 3 VGPU? | |
| 11:25:23 | sean-k-mooney | pvc_: you can only have 1 vgpu per vm currently and you can only enable 1 vgpu mdev type per phyical host | |
| 11:26:40 | sean-k-mooney | part of the limitation comes form libvirt and part form nvidia. multiple mdevs can be attached to the same instance but its a rather new addtion to the kernel and things have not really mautred yet in libvirt/qemu | |
| 11:26:43 | pvc_ | so i cannot use the 24gpus on my physical host? | |
| 11:27:24 | jangutter | sean-k-mooney: minor correction, intel_iommu=on is required, and "iommu=pt" is discouraged. | |
| 11:28:16 | jangutter | sean-k-mooney: iommu=pt used to be required when DPDK hadn't set vfio-pci as it's default transport yet, and people used uio. | |
| 11:29:56 | sean-k-mooney | jangutter: yes but i still recommend iommu=pt as the iommu is picky somethimes and i find you hit less corner cases if you limit its scope to pasthrough devices | |
| 11:30:28 | pvc_ | ERROR nova.compute.manager [instance: 014fcc5f-660b-40df-92ed-9f4587993fa7] Verify all devices in group 58 are bound to vfio-<bus> or pci-stub and not already in use | |
| 11:31:00 | sean-k-mooney | pvc_: yes so as i seaid previously i think your error is the gpus is sharingin an iommu group with anothe rdevice | |
| 11:31:18 | sean-k-mooney | pvc_: all devices in a iommu group must use the same kernel driver | |
| 11:31:22 | jangutter | sean-k-mooney: heh, iommu=pt is one of the worst-named options. The "passthrough" mapping it enables is a way to 'bypass' the IOMMU by creating a 1:1 memory map for the PCI space. | |
| 11:31:50 | pvc_ | how can i check that sean-k-mooney? | |