| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-23 | |||
| 12:29:49 | aperevalov | sean-k-mooney: yes netdevsim is based on its own bus, but not pci. I also found attemp to submit to QEMU's pci emulation the SR-IOV support. | |
| 12:30:36 | sean-k-mooney | aperevalov: yes so there is a netdev but no pci device on the virtual pci device. so you can use ip link and allocate vfs via sysfs but they dont show up on the virtual pci bus | |
| 12:31:20 | sean-k-mooney | aperevalov: yes i have see that in the past but that wont help us with testing as we would need the hosting vms provided by the could providers to emulate pci device that support sriov | |
| 12:31:30 | aperevalov | sean-k-mooney: but it was postponed due to lack of existing working qemu drivers, initial author suggested copied e1000, but it wasn't in working condition. | |
| 12:32:21 | sean-k-mooney | aperevalov: so first qemu would have to be extended then libvirt and then nova. once that is done we would need cloud providers to proved devices with virt sriov capable nics then we could start using it in the upstream gate | |
| 12:32:41 | sean-k-mooney | *provided vms with ... | |
| 12:33:29 | sean-k-mooney | aperevalov: effectivly we rely on thirdpart cis to test sriov. either intels or melonox's ci | |
| 12:33:54 | sean-k-mooney | intels ci was intented to have signifcatily more sriov testing then it currently has but that never happened | |
| 12:34:31 | aperevalov | sean-k-mooney, I see it's a long way, and seems netdevsim (just kernel module) looks like easiest (if kernel community will be agreed to bind it with pci bus). | |
| 12:35:50 | sean-k-mooney | aperevalov: most people dont know that sr-iov is a specificaiton from the PCI-SIG i dont think the current netdevsim moudle is technically a confroming sriov implentaiton without the pci emulation | |
| 12:36:19 | pvc_ | sean-k-mooney ls: cannot access /sys/class/mdev_bus/*/mdev_supported_types: No such file or directory | |
| 12:36:28 | sean-k-mooney | aperevalov: that said its a netdev simulator not a sriov simulator so they skipped the bits they did not need for there own testing | |
| 12:36:32 | pvc_ | i reboot again to reflect the new driver | |
| 12:38:35 | aperevalov | sean-k-mooney: ok, I'll talk with authors of netdevsim. Thank you for information. BTW is intel or mellanox ci is publicly available. Or that work is going through their teems involved into openstack community? | |
| 12:40:40 | aperevalov | sean-k-mooney: I think moshele knows it. | |
| 12:41:18 | pvc_ | hi sean-k-mooney | |
| 12:41:20 | moshele | aperevalov: I know what? | |
| 12:41:30 | pvc_ | after installign the grid driver the mdev is gone | |
| 12:41:33 | pvc_ | ls: cannot access /sys/class/mdev_bus/*/mdev_supported_types: No such file or directory | |
| 12:41:33 | sean-k-mooney | so i used to work at intel and one of my roles there was product owner of the intel nfv ci. it used to fall to the upstream teams to identify which features needed ci testing and either add it or request that the team maintaining it add it to there backlog | |
| 12:41:59 | pvc_ | sean-k-mooney any ideas? ls: cannot access /sys/class/mdev_bus/*/mdev_supported_types: No such file or directory | |
| 12:42:13 | sean-k-mooney | im not sure how the intel nfv ci currently works but if you reach out to the new maintianer im sure they will respond | |
| 12:42:13 | pvc_ | after installing the grid driver only not the vgpu | |
| 12:45:30 | moshele | aperevalov: The Mellanox CI is public but not part of the openstack community. and I think intel CI is the same, because it depend on nic vendor | |
| 12:45:33 | sean-k-mooney | pvc_: sorry not really. as i said this seams to be an nvidia driver issue. i dod not have acess to the hardware to test it my self so beyond reading there docs there is not much more advice i can give | |
| 12:46:06 | pvc_ | but sean-k-mooney when i install the kvm-vgpu i can list the mdev | |
| 12:47:03 | aperevalov | moshele: does it use in gerrit integration tests by zuul? | |
| 12:48:32 | moshele | aperevalov: we run tempest scenarios and configure the tempest to use vnic_type=direct | |
| 12:49:35 | moshele | aperevalov: see https://github.com/openstack/tempest/blob/master/tempest/config.py#L628 | |
| 12:50:10 | sean-k-mooney | moshele: oh when was that option added? | |
| 12:50:15 | openstackgerrit | Stephen Finucane proposed openstack/nova stable/rocky: fixtures: Track volume attachments within CinderFixtureNewAttachFlow https://review.openstack.org/612485 | |
| 12:50:16 | openstackgerrit | Stephen Finucane proposed openstack/nova stable/rocky: conductor: Recreate volume attachments during a reschedule https://review.openstack.org/612487 | |
| 12:50:16 | openstackgerrit | Stephen Finucane proposed openstack/nova stable/rocky: Add regression test for bug#1784353 https://review.openstack.org/612486 | |
| 12:50:16 | sean-k-mooney | that is useful to know about | |
| 12:50:25 | moshele | aperevalov: long time ago | |
| 12:50:39 | moshele | sean-k-mooney: long time ago | |
| 12:51:02 | pvc_ | vfio_iommu_type1.allow_unsafe_interrupts=1 | |
| 12:51:07 | pvc_ | sean-k-mooney vfio_iommu_type1.allow_unsafe_interrupts=1? | |
| 12:51:19 | sean-k-mooney | moshele: cool aperevalov the intel ci does somthing similar | |
| 12:52:07 | sean-k-mooney | aperevalov: the intel ci uses the standard senario test but addes extraflaovr extraspecs for cpu pinning hugepages numa toplogy exctra | |
| 12:52:44 | aperevalov | sean-k-mooney: If I trully understood, kuryr-kubernetes (tempest test) can also be running there? | |
| 12:52:47 | sean-k-mooney | pvc_: are you getting a message in dmesg? vfio_iommu_type1.allow_unsafe_interrupts=1 is specificaly for working around old buggy hardware | |
| 12:53:28 | sean-k-mooney | aperevalov: the intel nfv ci does not load the kuryr-kuberntese tempest module or deploy tempetst | |
| 12:53:40 | sean-k-mooney | at least it didnt when i was invovled with it | |
| 12:53:52 | pvc_ | Oct 23 12:53:40 overcloud-novacompute-0 journal: 2018-10-23 12:53:40.708+0000: 3128: warning : virDomainAuditHostdev:424 : Unexpected hostdev type while en | |
| 12:53:56 | pvc_ | Oct 23 12:53:40 overcloud-novacompute-0 journal: libvirt: QEMU Driver error : Requested operation is not valid: domain is not running | |
| 12:53:58 | sean-k-mooney | * or deply kuryr-kubernetes | |
| 12:54:59 | sean-k-mooney | pvc_: i dont really have time to contiue debugging sorry. i need to update some review and catch up on spec review today | |
| 12:56:41 | pvc_ | 2018-10-23 12:41:10.331+0000: 3216: error : virPCIDeviceNew:1787 : Device 0003:01:05.1 not found: could not access /sys/bus/pci/devices/0003:01:05.1/config | |
| 12:57:57 | pvc_ | virPidFileAcquirePath:422 : Failed to acquire pid file '/var/run/libvirtd.pid': Resource temporarily unavailable | |
| 13:02:22 | pvc_ | sean-k-mooney there is an issue on my nova_libvirt | |
| 13:05:38 | sean-k-mooney | pvc_: ok but that is not an nova issue. its a either a libvirt or a docker/triplo issue assuming you can acess /sys/bus/pci/devices/0003:01:05.1/config from the host. | |
| 13:06:56 | pvc_ | there is no 0003:01:01.1 sean | |
| 13:15:10 | pvc_ | sean-k-mooney is libvirtd not running is not an issue? | |
| 13:17:42 | efried | bauzas: https://review.openstack.org/#/c/612497/ <== provider config yaml file, split out from the device passthrough spec (with some of jaypipes' Rocky provider config file mixed in) | |
| 13:19:17 | bauzas | efried: ack | |
| 13:19:53 | bauzas | I have some planned outage this EU afternoon hence me being a bit afk but will look later tonight | |
| 13:20:38 | pvc_ | bauzas Failed to acquire pid file '/var/run/libvirtd.pid': Resource temporarily unavailable :( | |
| 13:20:52 | pvc_ | bauzas fio error: cad68f60-930c-4d9b-b954-3e0cd855651e: error getting device from group 58: Input/output error | |
| 13:33:09 | pvc_ | anyone can help? | |
| 13:53:03 | pvc_ | hi sean-k-moone do i need to hide the hypervisor of the image? | |
| 13:53:12 | pvc_ | hi sean-k-mooney do i need to hide the hypervisor of the image? | |
| 13:53:13 | alex_xu | jaypipes: for https://review.openstack.org/#/c/555081, are you saying that the user must specify guest numa topology when using resources:PCPU=1 or resources:VCPU=1 | |
| 13:54:42 | sean-k-mooney | alex_xu: im not suer if cpu pinning auto creates a numa toplogy today but it is does not its one of the few numa specifc things that does not | |
| 13:56:16 | pvc_ | sean-k-mooney i have an error on my XML | |
| 13:56:53 | pvc_ | 2bf12bf5 - default default] Error launching a defined domain with XML: <domain type='kvm'> | |
| 13:57:29 | alex_xu | sean-k-mooney: yes, I also think that. If the flavor doesn't include any guest numa topo, then we will get a None value for the InstanceTopologyObj. But jaypipes still want to use InstnaceTopology to store the cpu pinning. that is my confuse. | |
| 13:58:49 | sean-k-mooney | alex_xu: well cpus have numa affintiy so i would be fine with saying if your request pinning you now have a numa toploy of 1 numa node for the vm unless you set a numa toploygy explcitly | |
| 13:58:57 | sean-k-mooney | alex_xu: we do this for hugepages | |
| 13:59:25 | sean-k-mooney | personly i have normally argued against that but we have too much presdent to change it at this point | |
| 13:59:26 | openstackgerrit | Dan Smith proposed openstack/nova master: Make CellDatabases fixture reentrant https://review.openstack.org/611665 | |
| 13:59:27 | openstackgerrit | Dan Smith proposed openstack/nova master: Modify get_by_cell_and_project() to get_not_deleted_by_cell_and_projects() https://review.openstack.org/607663 | |
| 13:59:28 | openstackgerrit | Dan Smith proposed openstack/nova master: Return a minimal construct for nova show when a cell is down https://review.openstack.org/591658 | |
| 13:59:28 | openstackgerrit | Dan Smith proposed openstack/nova master: Refactor scatter-gather utility to return exception objects https://review.openstack.org/607934 | |
| 13:59:28 | openstackgerrit | Dan Smith proposed openstack/nova master: Minimal construct plumbing for nova list when a cell is down https://review.openstack.org/567785 | |
| 13:59:29 | openstackgerrit | Dan Smith proposed openstack/nova master: Return a minimal construct for nova service-list when a cell is down https://review.openstack.org/584829 | |
| 13:59:37 | pvc_ | is that related sean-k-mooney you think? | |
| 14:00:13 | sean-k-mooney | pvc_: i dont know but im busy with 3 other things. i do not have time to help futher sorry. | |
| 14:01:09 | alex_xu | sean-k-mooney: yea, that should be ok, that is just a clarify I ask on the spec, since it isn't clear about that | |
| 14:02:41 | sean-k-mooney | alex_xu: for what its worth the free cpus are already tracked in the numa toployg blob in the nova db so i dont hink jay was proposing changing that | |
| 14:02:44 | alex_xu | jaypipes: ^ probably that is what I'm asking, are you plan to change the guest without numa topo to single numa cell topo | |
| 14:03:02 | jaypipes | alex_xu: *currently* there is no way for a user to get pinned CPUs without the instance_extra.numa fields containing a serialized blob of InstanceNUMATopology object. | |
| 14:03:47 | jaypipes | alex_xu: because, as you know, we couple the CPU pinning, memory page and NUMA topology stuff all together in the InstanceNUMATopology object :( | |
| 14:03:55 | sean-k-mooney | jaypipes: do we currently invent a singel numa node topology today. its been to long since i looked at the details of that code to rember that off the top of my head | |
| 14:04:04 | jaypipes | alex_xu: the cpu-resource-tracking spec proposes absolutely no changes to any of that. | |
| 14:04:43 | sean-k-mooney | jaypipes: i have you spec on my list to review but i assumed we would still contiue to do whatever we do today on that front | |
| 14:05:19 | jaypipes | sean-k-mooney: mriedem has basically shot down the possibility of cpu-resource-tracking happening in stein anyway, so I haven't been spending much time on it. :( | |
| 14:05:44 | alex_xu | jaypipes: yes, but your spec didn't say I must specify HW:NUMA_xxx stuff when I use resources:PCPU=1 | |
| 14:05:46 | mriedem | once again i am the killer of all hopes and dreams and kittens | |
| 14:06:02 | jroll | always and forever | |
| 14:06:12 | mriedem | if others think we can pull that change off in stein and are planning on devoting review time to it, then go nuts | |
| 14:06:17 | jaypipes | alex_xu: I was asked by bauzas to take all NUMA stuff out of the cpu-resource-tracking spec so he could address it in his numa spec. | |
| 14:06:58 | alex_xu | jaypipes: and what does mean for CONF.shared_cpu_set, it is for the VCPU will pinning to the CPU set of CONF.shared_cpu_set, and then I must specify HW:NUMA_xxx with resources:VCPU=1? | |
| 14:08:00 | jaypipes | alex_xu: I don't understand your question. could you rephrase? | |
| 14:08:07 | alex_xu | let me try :) | |
| 14:08:13 | jaypipes | alex_xu: CONF.cpu_shared_set already exists, btw | |
| 14:09:12 | sean-k-mooney | alex_xu: jaypipes so jsut looking at https://github.com/openstack/nova/blob/297de7fb9fbabe74b5305ef0aa82e196d5f48d5e/nova/virt/hardware.py#L1543-L1554 we create a singel node numa toplogy for the guest today if using pinning unless you say otherwise | |
| 14:09:49 | jaypipes | sean-k-mooney: right, because we've coupled CPU pinning and NUMA together into the InstanceNUMAToplogy object. | |
| 14:09:55 | sean-k-mooney | alex_xu: so if you just set resources:PCPU=1 then i would assume we would create a singel numa toplogy | |
| 14:10:20 | sean-k-mooney | jaypipes: ya. if you want to decouple them and fix it im happy with that idea too | |