| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-02-15 | |||
| 11:51:55 | sean-k-mooney | is this something you will have time to work on | |
| 11:52:58 | sean-k-mooney | i have not booted the old servers i used to use for sriov work in the better part of a year maybe more so not sure if they are still working. i can review and maybe try and test the fix at some point but likely not until after FF | |
| 11:53:26 | sean-k-mooney | we will see how much progress i make on the health checks this week/next | |
| 11:53:52 | sean-k-mooney | im not sure if they will get done before FF but if not im hoping early next cycle they can land | |
| 11:54:50 | sean-k-mooney | gibi: by the way there is a slim change i can partly test dmitriis off-path code too. | |
| 11:55:46 | sean-k-mooney | i dont have bluefield 2 nics but i have a conectx-6 and if i lie to ovn and nova about it beign remote managed i should be able to test it end to end | |
| 11:56:06 | sean-k-mooney | but again likely wont get to that until FF has passed but im hoping to do it before RC1 | |
| 11:56:33 | gibi | sean-k-mooney: I have priority on this bug now from downstream I will work on it. But it should not effect me reviewing patches before FF | |
| 11:56:57 | gibi | so if you have anything just link me up | |
| 11:57:20 | dmitriis | sean-k-mooney: I am looking into adding more func tests and into doing a multi-node test. Unfortunately, SR-IOV is hidden by the QEMU's VFIO layer when PFs are passed through to a VM so I cannot reuse 2 PFs on the same host to fake 2 computes. | |
| 11:57:25 | dmitriis | brb in ~ 1h | |
| 11:57:42 | sean-k-mooney | dmitriis: yep it is | |
| 11:57:44 | gibi | I cannot test dmitriis patch I have only old SRIOV capable PFs | |
| 11:58:01 | sean-k-mooney | dmitriis: you can pass the PF to the vm but it nologner has the sriov extion | |
| 11:58:08 | dmitriis | https://git.qemu.org/?p=qemu.git;a=commitdiff;h=e37dac06dc4e85a2f46c24261c0dfdf2a30b50e3 | |
| 11:58:08 | dmitriis | Still there in recent versions of QEMU: https://github.com/qemu/qemu/blob/v6.2.0/hw/vfio/pci.c#L2094-L2097 | |
| 11:58:27 | dmitriis | this landed upstream https://lore.kernel.org/linux-pci/20200319184653.6c10638b.cohuck@redhat.com/T/ but the QEMU limitation is still there | |
| 11:58:28 | sean-k-mooney | oh.. | |
| 11:59:08 | opendevreview | Merged openstack/nova stable/xena: libvirt: Add announce-self post live-migration workaround https://review.opendev.org/c/openstack/nova/+/825064 | |
| 11:59:08 | dmitriis | (jumped into a meeting) | |
| 11:59:30 | sean-k-mooney | i wonder what happens if you jsut remove the sriov case there | |
| 12:01:37 | sean-k-mooney | /* Read-only VF BARs confuse OVMF */ implies its a uefi limitation | |
| 12:02:35 | sean-k-mooney | https://github.com/qemu/qemu/commit/e37dac06dc4e85a2f46c24261c0dfdf2a30b50e3 | |
| 13:03:01 | opendevreview | David Hill proposed openstack/nova master: Validate burst value is equal or smaller than 4194303 https://review.opendev.org/c/openstack/nova/+/829064 | |
| 13:14:47 | dmitriis | sean-k-mooney: yeah, the commit message says "OVMF tries to size the VF BARs and comes up with absurd results, ending with an assert". Haven't looked at the OVMF in-depth but I suspect it might be around this code which haven't seen much change besides automatic style fixes in 13 years: | |
| 13:14:47 | dmitriis | https://github.com/tianocore/edk2/blame/c095122d4b5f3152417cd97dabecfe31cc3b6508/MdeModulePkg/Bus/Pci/PciBusDxe/PciEnumeratorSupport.c#L494-L502 | |
| 13:14:47 | dmitriis | https://github.com/tianocore/edk2/blame/c095122d4b5f3152417cd97dabecfe31cc3b6508/MdeModulePkg/Bus/Pci/PciBusDxe/PciEnumeratorSupport.c#L1554-L1578 | |
| 13:35:31 | opendevreview | Jonathan Race proposed openstack/nova master: object/notification for Adds Pick guest CPU architecture based on host arch in libvirt driver support https://review.opendev.org/c/openstack/nova/+/828369 | |
| 13:35:31 | opendevreview | Jonathan Race proposed openstack/nova master: driver/secheduler/docs for Adds Pick guest CPU architecture based on host arch in libvirt driver support https://review.opendev.org/c/openstack/nova/+/822053 | |
| 13:35:32 | opendevreview | Jonathan Race proposed openstack/nova master: zuul-job for Adds Pick guest CPU architecture based on host arch in libvirt driver support https://review.opendev.org/c/openstack/nova/+/828372 | |
| 13:43:32 | opendevreview | Jonathan Race proposed openstack/nova master: object/notification for Adds Pick guest CPU architecture based on host arch in libvirt driver support https://review.opendev.org/c/openstack/nova/+/828369 | |
| 13:43:33 | opendevreview | Jonathan Race proposed openstack/nova master: driver/secheduler/docs for Adds Pick guest CPU architecture based on host arch in libvirt driver support https://review.opendev.org/c/openstack/nova/+/822053 | |
| 13:43:33 | opendevreview | Jonathan Race proposed openstack/nova master: zuul-job for Adds Pick guest CPU architecture based on host arch in libvirt driver support https://review.opendev.org/c/openstack/nova/+/828372 | |
| 13:46:46 | dmitriis | sean-k-mooney: One other way of working around this which might work is using two containers for Nova compute services (I've certainly ran Libvirt even in unprivileged containers before https://dshcherb.github.io/2017/12/04/qemu-kvm-virtual-machines-in-unprivileged-lxd.html). PCI device access isn't namespaced: we'd need privileged containers and | |
| 13:46:46 | dmitriis | have passthrough_whitelist to use "VFs of a PF" type of config in two different containers: | |
| 13:46:46 | dmitriis | `passthrough_whitelist = {"vendor_id":"*", "product_id":"15b3", "address": "0000:82:00.0", "physical_network": null, "remote_managed": "true"}` | |
| 13:46:46 | dmitriis | `passthrough_whitelist = {"vendor_id":"*", "product_id":"15b3", "address": "0000:82:00.1", "physical_network": null, "remote_managed": "true"}` | |
| 13:46:46 | dmitriis | where 0000:82:00.0 and 0000:82:00.1 are PF addresses. | |
| 13:49:35 | dmitriis | That should give us two "compute hosts" with different hostnames. We would be able to test the control plane path at least while faking the notion of a different host. In the live migration case a QEMU process would migrate locally into a different container and get a VF allocated from a different PF. | |
| 13:51:17 | sean-k-mooney | dmitriis: ya you can do that | |
| 13:51:33 | sean-k-mooney | but its not quite the same as a real live migration to differnt hosts | |
| 13:51:40 | sean-k-mooney | however it would work | |
| 13:52:51 | sean-k-mooney | that is proably more work then i want to do to do some testing but i have a 2 port connectx-6 so i could consider it | |
| 13:53:19 | sean-k-mooney | you woudl need two copies of libvirt too | |
| 13:53:30 | sean-k-mooney | with different hostnames in each contaienr | |
| 13:53:55 | sean-k-mooney | its doable but a lot of work. hopefully you have access to 2 hosts and can test this properly | |
| 13:53:56 | dmitriis | sean-k-mooney: yeah, a UTS namespace should give me that. IPC is namespaced too so shouldn't run into an issue with unix sockets. | |
| 13:54:08 | dmitriis | sean-k-mooney: agreed, just testing the control path in this case. I have a host with ConnectX5 as well that can be used for the live migration case (from BF2 to ConnectX 5). | |
| 13:54:22 | sean-k-mooney | dmitriis: by the way are you working on updating the melonox sriov ci to test this | |
| 13:54:34 | sean-k-mooney | i think it proably should be extended to test this | |
| 13:54:56 | sean-k-mooney | dmitriis: well in production that would not work | |
| 13:55:08 | sean-k-mooney | live migration form BF2 to connectx 5 | |
| 13:55:19 | sean-k-mooney | they woudl use differnt vnic types form a neutorn point of view | |
| 13:55:34 | sean-k-mooney | so that is not a capablity you will be able to leverage in an openstack env | |
| 13:55:41 | dmitriis | sean-k-mooney: I was hoping to get some clarity on the mellanox CI last week but that was postponed, I'm still pursuing it though | |
| 13:55:43 | sean-k-mooney | even though it could be made work | |
| 13:55:51 | sean-k-mooney | ack | |
| 13:56:28 | dmitriis | sean-k-mooney: I could configure the remote-managed feature to work with connectx as well since it's not mandatory to have networking agents on a different host | |
| 13:56:55 | dmitriis | i.e. I could tell Nova that a VF is remote-managed but run ovs-vswitchd and ovn-controller locally | |
| 13:56:59 | sean-k-mooney | technically however i dont think we shoudl really support that | |
| 13:57:08 | sean-k-mooney | we certenly should not document it in any upstream docs | |
| 13:57:25 | sean-k-mooney | except perhaps a developer docs | |
| 13:57:26 | dmitriis | yes, I don't intend to to avoid a clash with the existing approach | |
| 13:58:24 | sean-k-mooney | well its not so much that it would be a clash its more that its semanticaly incorrect. im also not sure what would add teh representor vf to ovs in that case | |
| 13:58:34 | sean-k-mooney | since os-vif wont do that nor will nova | |
| 13:58:46 | dmitriis | sean-k-mooney: that would be ovn-controller with ovn-vif | |
| 13:58:47 | sean-k-mooney | ovn would have to do that | |
| 13:58:49 | sean-k-mooney | ya | |
| 13:58:58 | dmitriis | yep, much like we do on BF2 | |
| 13:58:58 | sean-k-mooney | so you would have to also deploy that | |
| 13:59:03 | dmitriis | yes | |
| 13:59:17 | sean-k-mooney | so for a dev setup it coudl be used to fake a BF2 | |
| 13:59:34 | dmitriis | yes, exactly. ConnectX devices also have a VPD | |
| 13:59:35 | sean-k-mooney | but we dont want operators doing that really becasue we should not assuem that will always work | |
| 14:00:08 | sean-k-mooney | dmitriis: ya i think there are some broadcom cards that have it too not sure about intel | |
| 14:00:08 | dmitriis | sean-k-mooney: ack, agreed | |
| 14:00:28 | sean-k-mooney | in princiapl vpd is all that is needed | |
| 14:00:57 | sean-k-mooney | well and switch deve prot representer netdevs i guess for ovs-vif | |
| 14:01:01 | sean-k-mooney | *ovn-vif | |
| 14:01:05 | dmitriis | yes | |
| 14:01:40 | sean-k-mooney | hopefully other manufactures will implement both and they can just be enabeld using the work you already did | |
| 14:02:05 | dmitriis | sean-k-mooney: for now the switchdev-capable NIC kernel driver. Plenty of drivers in DPDK support representors so we are looking into that too | |
| 14:02:26 | sean-k-mooney | the dpdk ones are userspace only however | |
| 14:02:32 | sean-k-mooney | we dont have support for them in openstack | |
| 14:02:50 | sean-k-mooney | so you would have to add supprot for that end to end in a similar way | |
| 14:03:17 | sean-k-mooney | although i assume they just use vhost-user on the qemu side | |
| 14:03:23 | dmitriis | sean-k-mooney: hmm, why would the hypervisor side VF care how a VF representor is handled at the remote side? | |
| 14:03:44 | dmitriis | i.e. it could be ovs + DPDK at the BF2 side while the hypervisor side VF would use anything it wants | |
| 14:03:59 | sean-k-mooney | because dpdk does not require VFs | |
| 14:04:26 | sean-k-mooney | it support userspace only represntors that use intels sio cabliteis too | |
| 14:04:46 | sean-k-mooney | so in generally you shoudl not assuem with dpdk that we woudl have a vf attach to the vm at all | |
| 14:05:04 | sean-k-mooney | it could just be a vhost-user prot that is then interally connect to a subfunction | |
| 14:05:44 | sean-k-mooney | useing ovs-dpdk on the BF2 side is different and likely can be hiddne since you are enabelign a VF based approch | |
| 14:05:57 | dmitriis | yeah | |
| 14:05:59 | dmitriis | https://doc.dpdk.org/guides/prog_guide/switch_representation.html#port-representors | |
| 14:06:00 | sean-k-mooney | dpdk woudl just be used for faster flow programing via dpdk flow instead fo tc_flower | |
| 14:06:12 | dmitriis | sean-k-mooney: yep, that's where I was going | |
| 14:06:31 | sean-k-mooney | ya so that is not the same as dpdk port-represntors feature | |
| 14:06:48 | dmitriis | right, I see | |
| 14:06:55 | dmitriis | so this would be to have a broader coverage at the DPU side (BF2 or not) | |