| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-02-15 | |||
| 11:58:08 | dmitriis | https://git.qemu.org/?p=qemu.git;a=commitdiff;h=e37dac06dc4e85a2f46c24261c0dfdf2a30b50e3 | |
| 11:58:08 | dmitriis | Still there in recent versions of QEMU: https://github.com/qemu/qemu/blob/v6.2.0/hw/vfio/pci.c#L2094-L2097 | |
| 11:58:27 | dmitriis | this landed upstream https://lore.kernel.org/linux-pci/20200319184653.6c10638b.cohuck@redhat.com/T/ but the QEMU limitation is still there | |
| 11:58:28 | sean-k-mooney | oh.. | |
| 11:59:08 | opendevreview | Merged openstack/nova stable/xena: libvirt: Add announce-self post live-migration workaround https://review.opendev.org/c/openstack/nova/+/825064 | |
| 11:59:08 | dmitriis | (jumped into a meeting) | |
| 11:59:30 | sean-k-mooney | i wonder what happens if you jsut remove the sriov case there | |
| 12:01:37 | sean-k-mooney | /* Read-only VF BARs confuse OVMF */ implies its a uefi limitation | |
| 12:02:35 | sean-k-mooney | https://github.com/qemu/qemu/commit/e37dac06dc4e85a2f46c24261c0dfdf2a30b50e3 | |
| 13:03:01 | opendevreview | David Hill proposed openstack/nova master: Validate burst value is equal or smaller than 4194303 https://review.opendev.org/c/openstack/nova/+/829064 | |
| 13:14:47 | dmitriis | sean-k-mooney: yeah, the commit message says "OVMF tries to size the VF BARs and comes up with absurd results, ending with an assert". Haven't looked at the OVMF in-depth but I suspect it might be around this code which haven't seen much change besides automatic style fixes in 13 years: | |
| 13:14:47 | dmitriis | https://github.com/tianocore/edk2/blame/c095122d4b5f3152417cd97dabecfe31cc3b6508/MdeModulePkg/Bus/Pci/PciBusDxe/PciEnumeratorSupport.c#L494-L502 | |
| 13:14:47 | dmitriis | https://github.com/tianocore/edk2/blame/c095122d4b5f3152417cd97dabecfe31cc3b6508/MdeModulePkg/Bus/Pci/PciBusDxe/PciEnumeratorSupport.c#L1554-L1578 | |
| 13:35:31 | opendevreview | Jonathan Race proposed openstack/nova master: object/notification for Adds Pick guest CPU architecture based on host arch in libvirt driver support https://review.opendev.org/c/openstack/nova/+/828369 | |
| 13:35:31 | opendevreview | Jonathan Race proposed openstack/nova master: driver/secheduler/docs for Adds Pick guest CPU architecture based on host arch in libvirt driver support https://review.opendev.org/c/openstack/nova/+/822053 | |
| 13:35:32 | opendevreview | Jonathan Race proposed openstack/nova master: zuul-job for Adds Pick guest CPU architecture based on host arch in libvirt driver support https://review.opendev.org/c/openstack/nova/+/828372 | |
| 13:43:32 | opendevreview | Jonathan Race proposed openstack/nova master: object/notification for Adds Pick guest CPU architecture based on host arch in libvirt driver support https://review.opendev.org/c/openstack/nova/+/828369 | |
| 13:43:33 | opendevreview | Jonathan Race proposed openstack/nova master: driver/secheduler/docs for Adds Pick guest CPU architecture based on host arch in libvirt driver support https://review.opendev.org/c/openstack/nova/+/822053 | |
| 13:43:33 | opendevreview | Jonathan Race proposed openstack/nova master: zuul-job for Adds Pick guest CPU architecture based on host arch in libvirt driver support https://review.opendev.org/c/openstack/nova/+/828372 | |
| 13:46:46 | dmitriis | sean-k-mooney: One other way of working around this which might work is using two containers for Nova compute services (I've certainly ran Libvirt even in unprivileged containers before https://dshcherb.github.io/2017/12/04/qemu-kvm-virtual-machines-in-unprivileged-lxd.html). PCI device access isn't namespaced: we'd need privileged containers and | |
| 13:46:46 | dmitriis | have passthrough_whitelist to use "VFs of a PF" type of config in two different containers: | |
| 13:46:46 | dmitriis | `passthrough_whitelist = {"vendor_id":"*", "product_id":"15b3", "address": "0000:82:00.0", "physical_network": null, "remote_managed": "true"}` | |
| 13:46:46 | dmitriis | `passthrough_whitelist = {"vendor_id":"*", "product_id":"15b3", "address": "0000:82:00.1", "physical_network": null, "remote_managed": "true"}` | |
| 13:46:46 | dmitriis | where 0000:82:00.0 and 0000:82:00.1 are PF addresses. | |
| 13:49:35 | dmitriis | That should give us two "compute hosts" with different hostnames. We would be able to test the control plane path at least while faking the notion of a different host. In the live migration case a QEMU process would migrate locally into a different container and get a VF allocated from a different PF. | |
| 13:51:17 | sean-k-mooney | dmitriis: ya you can do that | |
| 13:51:33 | sean-k-mooney | but its not quite the same as a real live migration to differnt hosts | |
| 13:51:40 | sean-k-mooney | however it would work | |
| 13:52:51 | sean-k-mooney | that is proably more work then i want to do to do some testing but i have a 2 port connectx-6 so i could consider it | |
| 13:53:19 | sean-k-mooney | you woudl need two copies of libvirt too | |
| 13:53:30 | sean-k-mooney | with different hostnames in each contaienr | |
| 13:53:55 | sean-k-mooney | its doable but a lot of work. hopefully you have access to 2 hosts and can test this properly | |
| 13:53:56 | dmitriis | sean-k-mooney: yeah, a UTS namespace should give me that. IPC is namespaced too so shouldn't run into an issue with unix sockets. | |
| 13:54:08 | dmitriis | sean-k-mooney: agreed, just testing the control path in this case. I have a host with ConnectX5 as well that can be used for the live migration case (from BF2 to ConnectX 5). | |
| 13:54:22 | sean-k-mooney | dmitriis: by the way are you working on updating the melonox sriov ci to test this | |
| 13:54:34 | sean-k-mooney | i think it proably should be extended to test this | |
| 13:54:56 | sean-k-mooney | dmitriis: well in production that would not work | |
| 13:55:08 | sean-k-mooney | live migration form BF2 to connectx 5 | |
| 13:55:19 | sean-k-mooney | they woudl use differnt vnic types form a neutorn point of view | |
| 13:55:34 | sean-k-mooney | so that is not a capablity you will be able to leverage in an openstack env | |
| 13:55:41 | dmitriis | sean-k-mooney: I was hoping to get some clarity on the mellanox CI last week but that was postponed, I'm still pursuing it though | |
| 13:55:43 | sean-k-mooney | even though it could be made work | |
| 13:55:51 | sean-k-mooney | ack | |
| 13:56:28 | dmitriis | sean-k-mooney: I could configure the remote-managed feature to work with connectx as well since it's not mandatory to have networking agents on a different host | |
| 13:56:55 | dmitriis | i.e. I could tell Nova that a VF is remote-managed but run ovs-vswitchd and ovn-controller locally | |
| 13:56:59 | sean-k-mooney | technically however i dont think we shoudl really support that | |
| 13:57:08 | sean-k-mooney | we certenly should not document it in any upstream docs | |
| 13:57:25 | sean-k-mooney | except perhaps a developer docs | |
| 13:57:26 | dmitriis | yes, I don't intend to to avoid a clash with the existing approach | |
| 13:58:24 | sean-k-mooney | well its not so much that it would be a clash its more that its semanticaly incorrect. im also not sure what would add teh representor vf to ovs in that case | |
| 13:58:34 | sean-k-mooney | since os-vif wont do that nor will nova | |
| 13:58:46 | dmitriis | sean-k-mooney: that would be ovn-controller with ovn-vif | |
| 13:58:47 | sean-k-mooney | ovn would have to do that | |
| 13:58:49 | sean-k-mooney | ya | |
| 13:58:58 | dmitriis | yep, much like we do on BF2 | |
| 13:58:58 | sean-k-mooney | so you would have to also deploy that | |
| 13:59:03 | dmitriis | yes | |
| 13:59:17 | sean-k-mooney | so for a dev setup it coudl be used to fake a BF2 | |
| 13:59:34 | dmitriis | yes, exactly. ConnectX devices also have a VPD | |
| 13:59:35 | sean-k-mooney | but we dont want operators doing that really becasue we should not assuem that will always work | |
| 14:00:08 | sean-k-mooney | dmitriis: ya i think there are some broadcom cards that have it too not sure about intel | |
| 14:00:08 | dmitriis | sean-k-mooney: ack, agreed | |
| 14:00:28 | sean-k-mooney | in princiapl vpd is all that is needed | |
| 14:00:57 | sean-k-mooney | well and switch deve prot representer netdevs i guess for ovs-vif | |
| 14:01:01 | sean-k-mooney | *ovn-vif | |
| 14:01:05 | dmitriis | yes | |
| 14:01:40 | sean-k-mooney | hopefully other manufactures will implement both and they can just be enabeld using the work you already did | |
| 14:02:05 | dmitriis | sean-k-mooney: for now the switchdev-capable NIC kernel driver. Plenty of drivers in DPDK support representors so we are looking into that too | |
| 14:02:26 | sean-k-mooney | the dpdk ones are userspace only however | |
| 14:02:32 | sean-k-mooney | we dont have support for them in openstack | |
| 14:02:50 | sean-k-mooney | so you would have to add supprot for that end to end in a similar way | |
| 14:03:17 | sean-k-mooney | although i assume they just use vhost-user on the qemu side | |
| 14:03:23 | dmitriis | sean-k-mooney: hmm, why would the hypervisor side VF care how a VF representor is handled at the remote side? | |
| 14:03:44 | dmitriis | i.e. it could be ovs + DPDK at the BF2 side while the hypervisor side VF would use anything it wants | |
| 14:03:59 | sean-k-mooney | because dpdk does not require VFs | |
| 14:04:26 | sean-k-mooney | it support userspace only represntors that use intels sio cabliteis too | |
| 14:04:46 | sean-k-mooney | so in generally you shoudl not assuem with dpdk that we woudl have a vf attach to the vm at all | |
| 14:05:04 | sean-k-mooney | it could just be a vhost-user prot that is then interally connect to a subfunction | |
| 14:05:44 | sean-k-mooney | useing ovs-dpdk on the BF2 side is different and likely can be hiddne since you are enabelign a VF based approch | |
| 14:05:57 | dmitriis | yeah | |
| 14:05:59 | dmitriis | https://doc.dpdk.org/guides/prog_guide/switch_representation.html#port-representors | |
| 14:06:00 | sean-k-mooney | dpdk woudl just be used for faster flow programing via dpdk flow instead fo tc_flower | |
| 14:06:12 | dmitriis | sean-k-mooney: yep, that's where I was going | |
| 14:06:31 | sean-k-mooney | ya so that is not the same as dpdk port-represntors feature | |
| 14:06:48 | dmitriis | right, I see | |
| 14:06:55 | dmitriis | so this would be to have a broader coverage at the DPU side (BF2 or not) | |
| 14:07:02 | sean-k-mooney | you can use vfs with it but it is more general | |
| 14:08:03 | sean-k-mooney | you should be able to use dpdk with what you have provided today | |
| 14:08:20 | opendevreview | Balazs Gibizer proposed openstack/nova master: Record SRIOV PF MAC in the binding profile https://review.opendev.org/c/openstack/nova/+/829248 | |
| 14:08:26 | sean-k-mooney | since we still have a VF on the host side | |
| 14:08:52 | sean-k-mooney | dmitriis: it would only change if we wanted to supprot subfunciton or ovs-dpdk runnign on the host with hardware offload | |
| 14:09:05 | dmitriis | sean-k-mooney: yes, probably with some minor modifications to os-vif but at the OpenStack side things would remain the same | |
| 14:09:36 | dmitriis | sean-k-mooney: yes, subfunctions need a little bit more thinking since they are more dynamic | |
| 14:09:52 | dmitriis | we currently just enable SR-IOV and expect VF representors to appear the other side | |
| 14:10:18 | sean-k-mooney | ya | |
| 14:10:42 | sean-k-mooney | so i proably would still prefer to track subfunciton in the pci tracker as a new type | |
| 14:10:47 | sean-k-mooney | like vdpa | |
| 14:11:05 | sean-k-mooney | even if they are not a vf and may not have a pci adress | |
| 14:11:21 | sean-k-mooney | at which point we might want to rename the module to be host_dev tracker or something | |
| 14:11:42 | sean-k-mooney | i kind of want it to eventualy track mdevs too | |