| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-02-15 | |||
| 13:43:33 | opendevreview | Jonathan Race proposed openstack/nova master: driver/secheduler/docs for Adds Pick guest CPU architecture based on host arch in libvirt driver support https://review.opendev.org/c/openstack/nova/+/822053 | |
| 13:43:33 | opendevreview | Jonathan Race proposed openstack/nova master: zuul-job for Adds Pick guest CPU architecture based on host arch in libvirt driver support https://review.opendev.org/c/openstack/nova/+/828372 | |
| 13:46:46 | dmitriis | sean-k-mooney: One other way of working around this which might work is using two containers for Nova compute services (I've certainly ran Libvirt even in unprivileged containers before https://dshcherb.github.io/2017/12/04/qemu-kvm-virtual-machines-in-unprivileged-lxd.html). PCI device access isn't namespaced: we'd need privileged containers and | |
| 13:46:46 | dmitriis | have passthrough_whitelist to use "VFs of a PF" type of config in two different containers: | |
| 13:46:46 | dmitriis | `passthrough_whitelist = {"vendor_id":"*", "product_id":"15b3", "address": "0000:82:00.0", "physical_network": null, "remote_managed": "true"}` | |
| 13:46:46 | dmitriis | `passthrough_whitelist = {"vendor_id":"*", "product_id":"15b3", "address": "0000:82:00.1", "physical_network": null, "remote_managed": "true"}` | |
| 13:46:46 | dmitriis | where 0000:82:00.0 and 0000:82:00.1 are PF addresses. | |
| 13:49:35 | dmitriis | That should give us two "compute hosts" with different hostnames. We would be able to test the control plane path at least while faking the notion of a different host. In the live migration case a QEMU process would migrate locally into a different container and get a VF allocated from a different PF. | |
| 13:51:17 | sean-k-mooney | dmitriis: ya you can do that | |
| 13:51:33 | sean-k-mooney | but its not quite the same as a real live migration to differnt hosts | |
| 13:51:40 | sean-k-mooney | however it would work | |
| 13:52:51 | sean-k-mooney | that is proably more work then i want to do to do some testing but i have a 2 port connectx-6 so i could consider it | |
| 13:53:19 | sean-k-mooney | you woudl need two copies of libvirt too | |
| 13:53:30 | sean-k-mooney | with different hostnames in each contaienr | |
| 13:53:55 | sean-k-mooney | its doable but a lot of work. hopefully you have access to 2 hosts and can test this properly | |
| 13:53:56 | dmitriis | sean-k-mooney: yeah, a UTS namespace should give me that. IPC is namespaced too so shouldn't run into an issue with unix sockets. | |
| 13:54:08 | dmitriis | sean-k-mooney: agreed, just testing the control path in this case. I have a host with ConnectX5 as well that can be used for the live migration case (from BF2 to ConnectX 5). | |
| 13:54:22 | sean-k-mooney | dmitriis: by the way are you working on updating the melonox sriov ci to test this | |
| 13:54:34 | sean-k-mooney | i think it proably should be extended to test this | |
| 13:54:56 | sean-k-mooney | dmitriis: well in production that would not work | |
| 13:55:08 | sean-k-mooney | live migration form BF2 to connectx 5 | |
| 13:55:19 | sean-k-mooney | they woudl use differnt vnic types form a neutorn point of view | |
| 13:55:34 | sean-k-mooney | so that is not a capablity you will be able to leverage in an openstack env | |
| 13:55:41 | dmitriis | sean-k-mooney: I was hoping to get some clarity on the mellanox CI last week but that was postponed, I'm still pursuing it though | |
| 13:55:43 | sean-k-mooney | even though it could be made work | |
| 13:55:51 | sean-k-mooney | ack | |
| 13:56:28 | dmitriis | sean-k-mooney: I could configure the remote-managed feature to work with connectx as well since it's not mandatory to have networking agents on a different host | |
| 13:56:55 | dmitriis | i.e. I could tell Nova that a VF is remote-managed but run ovs-vswitchd and ovn-controller locally | |
| 13:56:59 | sean-k-mooney | technically however i dont think we shoudl really support that | |
| 13:57:08 | sean-k-mooney | we certenly should not document it in any upstream docs | |
| 13:57:25 | sean-k-mooney | except perhaps a developer docs | |
| 13:57:26 | dmitriis | yes, I don't intend to to avoid a clash with the existing approach | |
| 13:58:24 | sean-k-mooney | well its not so much that it would be a clash its more that its semanticaly incorrect. im also not sure what would add teh representor vf to ovs in that case | |
| 13:58:34 | sean-k-mooney | since os-vif wont do that nor will nova | |
| 13:58:46 | dmitriis | sean-k-mooney: that would be ovn-controller with ovn-vif | |
| 13:58:47 | sean-k-mooney | ovn would have to do that | |
| 13:58:49 | sean-k-mooney | ya | |
| 13:58:58 | dmitriis | yep, much like we do on BF2 | |
| 13:58:58 | sean-k-mooney | so you would have to also deploy that | |
| 13:59:03 | dmitriis | yes | |
| 13:59:17 | sean-k-mooney | so for a dev setup it coudl be used to fake a BF2 | |
| 13:59:34 | dmitriis | yes, exactly. ConnectX devices also have a VPD | |
| 13:59:35 | sean-k-mooney | but we dont want operators doing that really becasue we should not assuem that will always work | |
| 14:00:08 | sean-k-mooney | dmitriis: ya i think there are some broadcom cards that have it too not sure about intel | |
| 14:00:08 | dmitriis | sean-k-mooney: ack, agreed | |
| 14:00:28 | sean-k-mooney | in princiapl vpd is all that is needed | |
| 14:00:57 | sean-k-mooney | well and switch deve prot representer netdevs i guess for ovs-vif | |
| 14:01:01 | sean-k-mooney | *ovn-vif | |
| 14:01:05 | dmitriis | yes | |
| 14:01:40 | sean-k-mooney | hopefully other manufactures will implement both and they can just be enabeld using the work you already did | |
| 14:02:05 | dmitriis | sean-k-mooney: for now the switchdev-capable NIC kernel driver. Plenty of drivers in DPDK support representors so we are looking into that too | |
| 14:02:26 | sean-k-mooney | the dpdk ones are userspace only however | |
| 14:02:32 | sean-k-mooney | we dont have support for them in openstack | |
| 14:02:50 | sean-k-mooney | so you would have to add supprot for that end to end in a similar way | |
| 14:03:17 | sean-k-mooney | although i assume they just use vhost-user on the qemu side | |
| 14:03:23 | dmitriis | sean-k-mooney: hmm, why would the hypervisor side VF care how a VF representor is handled at the remote side? | |
| 14:03:44 | dmitriis | i.e. it could be ovs + DPDK at the BF2 side while the hypervisor side VF would use anything it wants | |
| 14:03:59 | sean-k-mooney | because dpdk does not require VFs | |
| 14:04:26 | sean-k-mooney | it support userspace only represntors that use intels sio cabliteis too | |
| 14:04:46 | sean-k-mooney | so in generally you shoudl not assuem with dpdk that we woudl have a vf attach to the vm at all | |
| 14:05:04 | sean-k-mooney | it could just be a vhost-user prot that is then interally connect to a subfunction | |
| 14:05:44 | sean-k-mooney | useing ovs-dpdk on the BF2 side is different and likely can be hiddne since you are enabelign a VF based approch | |
| 14:05:57 | dmitriis | yeah | |
| 14:05:59 | dmitriis | https://doc.dpdk.org/guides/prog_guide/switch_representation.html#port-representors | |
| 14:06:00 | sean-k-mooney | dpdk woudl just be used for faster flow programing via dpdk flow instead fo tc_flower | |
| 14:06:12 | dmitriis | sean-k-mooney: yep, that's where I was going | |
| 14:06:31 | sean-k-mooney | ya so that is not the same as dpdk port-represntors feature | |
| 14:06:48 | dmitriis | right, I see | |
| 14:06:55 | dmitriis | so this would be to have a broader coverage at the DPU side (BF2 or not) | |
| 14:07:02 | sean-k-mooney | you can use vfs with it but it is more general | |
| 14:08:03 | sean-k-mooney | you should be able to use dpdk with what you have provided today | |
| 14:08:20 | opendevreview | Balazs Gibizer proposed openstack/nova master: Record SRIOV PF MAC in the binding profile https://review.opendev.org/c/openstack/nova/+/829248 | |
| 14:08:26 | sean-k-mooney | since we still have a VF on the host side | |
| 14:08:52 | sean-k-mooney | dmitriis: it would only change if we wanted to supprot subfunciton or ovs-dpdk runnign on the host with hardware offload | |
| 14:09:05 | dmitriis | sean-k-mooney: yes, probably with some minor modifications to os-vif but at the OpenStack side things would remain the same | |
| 14:09:36 | dmitriis | sean-k-mooney: yes, subfunctions need a little bit more thinking since they are more dynamic | |
| 14:09:52 | dmitriis | we currently just enable SR-IOV and expect VF representors to appear the other side | |
| 14:10:18 | sean-k-mooney | ya | |
| 14:10:42 | sean-k-mooney | so i proably would still prefer to track subfunciton in the pci tracker as a new type | |
| 14:10:47 | sean-k-mooney | like vdpa | |
| 14:11:05 | sean-k-mooney | even if they are not a vf and may not have a pci adress | |
| 14:11:21 | sean-k-mooney | at which point we might want to rename the module to be host_dev tracker or something | |
| 14:11:42 | sean-k-mooney | i kind of want it to eventualy track mdevs too | |
| 14:12:01 | sean-k-mooney | so it woudl be nice if we could unify it all in a common module | |
| 14:12:10 | sean-k-mooney | but that is not currently a priority for me | |
| 14:12:44 | sean-k-mooney | we alternitivly can use the generic resouces table to track these liek we do for persictent memroy namespaces | |
| 14:12:50 | sean-k-mooney | it was created to be generic | |
| 14:13:02 | sean-k-mooney | it might be a better fit for subfunciton and mdevs | |
| 14:13:19 | dmitriis | sean-k-mooney: on module renaming: yes, kind of going along the way VFIO is going (it has a common part and bus-specific parts) | |
| 14:13:28 | sean-k-mooney | but i have not really spent time looking, at least not in the last 2 years | |
| 14:15:26 | dmitriis | sean-k-mooney: one other direction which we haven't covered is ways for an operator to choose which device will be selected. E.g. in case of multiple DPUs per host or in the cloud we might want to supply product id-like info. | |
| 14:15:36 | dmitriis | currently InstancePCIRequest gets a generated spec | |
| 14:16:07 | sean-k-mooney | no | |
| 14:16:11 | sean-k-mooney | we do not | |
| 14:16:37 | sean-k-mooney | the operator gets to select which device are allowed | |
| 14:16:57 | sean-k-mooney | and can also associate them with a neutron phsyical network | |
| 14:17:18 | sean-k-mooney | but we explcitly do not want them to be able to select device via the pci aliase or the request by other means | |
| 14:17:41 | sean-k-mooney | what we could support is traits requests | |
| 14:17:58 | sean-k-mooney | or perhaps resouce classes | |
| 14:18:20 | sean-k-mooney | i have covered both of those in my pci in placment spec | |