Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2022-02-15
11:33:45 sean-k-mooney actully it looks like they tried to work around this once before by adding that
11:34:28 gibi today nova writes the port.mac_address for PF ports before the first binding. So if there won't be a reset in neutron then after unbind the port would keep the mac of a real device
11:34:46 sean-k-mooney https://bugs.launchpad.net/neutron/+bug/1830383
11:35:00 sean-k-mooney ya so that code was an incorrect fix for the current issue your trying to solve
11:35:16 sean-k-mooney gibi: that should be reverted
11:35:36 gibi but then what mac a PF port should have after it is unbound
11:35:38 gibi ?
11:35:48 gibi empty?
11:35:51 sean-k-mooney the same one it had when it was bound
11:35:59 sean-k-mooney nova is still telling the vm to use that mac
11:36:08 sean-k-mooney in the cold migration case
11:36:20 gibi hm
11:36:56 sean-k-mooney i can see why they did this but today since we are not updating the mac on the neutron port in the cold migration case this only help in the event we detach the port
11:36:57 gibi so if I boot a VM with a PF port, then that port gets the MAC of the PF today. Then I delete the VM. The port should forget the MAC of the PF
11:37:30 sean-k-mooney ya in that case the currnt reset hack works
11:37:40 gibi so we have to keep that reset for the unbinding case
11:37:48 sean-k-mooney we can yes
11:38:10 gibi for the rebinding there will be a new MAC from nova
11:38:49 sean-k-mooney however if you have 2 vm and 3 hosts and you move vm a form host a to host c and vm b form host b to host a today
11:39:01 sean-k-mooney vm b will get a dhcp responce for vm a
11:39:28 sean-k-mooney because in neutorn we are not updating the macs and vm a still has a port with the mac for port b
11:39:36 sean-k-mooney *for host a
11:39:49 sean-k-mooney assuming vm a and b are on the same netork
11:40:28 sean-k-mooney so the reset is not happeing today when we unbindi the port unle when we detach the port
11:40:59 gibi I have verified it. When I delete the VM neutron resets the MAC
11:41:02 gibi today
11:41:05 sean-k-mooney well i guess it depend on how you define "unbind"
11:41:09 sean-k-mooney yep
11:41:53 sean-k-mooney so if you update the host-id today the port will breifly go to vif_type=unbound
11:42:13 sean-k-mooney so i consider updating the host-id to be unbinding/rebinding the port
11:42:14 gibi but yes if I migrate the VM (and not delet it) then I can create the mixed MAC case you described
11:42:48 sean-k-mooney updating the device_id/device_onwer to '' i do not consider to be unbinding the port
11:42:55 sean-k-mooney that is detaching the port
11:43:56 gibi ahh I see
11:44:31 sean-k-mooney gibi: sorry if im beign pedantic with the terminology there it just the device_id and device_owner filed are not part of the port_bindign or port_binding_extended api extentions
11:44:43 gibi true
11:44:54 gibi just the host_id
11:45:04 gibi but I think that is OK
11:45:14 sean-k-mooney https://github.com/openstack/neutron-lib/blob/master/neutron_lib/api/definitions/portbindings_extended.py#L75-L105 yes
11:45:44 sean-k-mooney so that is why updating the host-id in the cold migrat case set the vif_type to unbound and then to the correct vif
11:45:59 sean-k-mooney it first unbindign the port in the ml2 driver then binds it again
11:46:13 sean-k-mooney potentially to a different ml2 driver
11:46:52 sean-k-mooney e.g. from linux bridge to ovs or whatever if both dirver supprot the vnic_type and you have different ones on different hosts
11:47:17 sean-k-mooney gibi when it the neutron meeting
11:47:59 gibi 14:00 UTC today
11:48:02 gibi afaik
11:48:04 sean-k-mooney i can try and attend but i think we are more or less on teh same page
11:48:21 sean-k-mooney ok so basicaly just over 2 hours form now
11:50:51 gibi no worries
11:51:08 gibi I agree that we are on the same page
11:51:55 sean-k-mooney is this something you will have time to work on
11:52:58 sean-k-mooney i have not booted the old servers i used to use for sriov work in the better part of a year maybe more so not sure if they are still working. i can review and maybe try and test the fix at some point but likely not until after FF
11:53:26 sean-k-mooney we will see how much progress i make on the health checks this week/next
11:53:52 sean-k-mooney im not sure if they will get done before FF but if not im hoping early next cycle they can land
11:54:50 sean-k-mooney gibi: by the way there is a slim change i can partly test dmitriis off-path code too.
11:55:46 sean-k-mooney i dont have bluefield 2 nics but i have a conectx-6 and if i lie to ovn and nova about it beign remote managed i should be able to test it end to end
11:56:06 sean-k-mooney but again likely wont get to that until FF has passed but im hoping to do it before RC1
11:56:33 gibi sean-k-mooney: I have priority on this bug now from downstream I will work on it. But it should not effect me reviewing patches before FF
11:56:57 gibi so if you have anything just link me up
11:57:20 dmitriis sean-k-mooney: I am looking into adding more func tests and into doing a multi-node test. Unfortunately, SR-IOV is hidden by the QEMU's VFIO layer when PFs are passed through to a VM so I cannot reuse 2 PFs on the same host to fake 2 computes.
11:57:25 dmitriis brb in ~ 1h
11:57:42 sean-k-mooney dmitriis: yep it is
11:57:44 gibi I cannot test dmitriis patch I have only old SRIOV capable PFs
11:58:01 sean-k-mooney dmitriis: you can pass the PF to the vm but it nologner has the sriov extion
11:58:08 dmitriis Still there in recent versions of QEMU: https://github.com/qemu/qemu/blob/v6.2.0/hw/vfio/pci.c#L2094-L2097
11:58:08 dmitriis https://git.qemu.org/?p=qemu.git;a=commitdiff;h=e37dac06dc4e85a2f46c24261c0dfdf2a30b50e3
11:58:27 dmitriis this landed upstream https://lore.kernel.org/linux-pci/20200319184653.6c10638b.cohuck@redhat.com/T/ but the QEMU limitation is still there
11:58:28 sean-k-mooney oh..
11:59:08 dmitriis (jumped into a meeting)
11:59:08 opendevreview Merged openstack/nova stable/xena: libvirt: Add announce-self post live-migration workaround https://review.opendev.org/c/openstack/nova/+/825064
11:59:30 sean-k-mooney i wonder what happens if you jsut remove the sriov case there
12:01:37 sean-k-mooney /* Read-only VF BARs confuse OVMF */ implies its a uefi limitation
12:02:35 sean-k-mooney https://github.com/qemu/qemu/commit/e37dac06dc4e85a2f46c24261c0dfdf2a30b50e3
13:03:01 opendevreview David Hill proposed openstack/nova master: Validate burst value is equal or smaller than 4194303 https://review.opendev.org/c/openstack/nova/+/829064
13:14:47 dmitriis https://github.com/tianocore/edk2/blame/c095122d4b5f3152417cd97dabecfe31cc3b6508/MdeModulePkg/Bus/Pci/PciBusDxe/PciEnumeratorSupport.c#L1554-L1578
13:14:47 dmitriis https://github.com/tianocore/edk2/blame/c095122d4b5f3152417cd97dabecfe31cc3b6508/MdeModulePkg/Bus/Pci/PciBusDxe/PciEnumeratorSupport.c#L494-L502
13:14:47 dmitriis sean-k-mooney: yeah, the commit message says "OVMF tries to size the VF BARs and comes up with absurd results, ending with an assert". Haven't looked at the OVMF in-depth but I suspect it might be around this code which haven't seen much change besides automatic style fixes in 13 years:
13:35:31 opendevreview Jonathan Race proposed openstack/nova master: driver/secheduler/docs for Adds Pick guest CPU architecture based on host arch in libvirt driver support https://review.opendev.org/c/openstack/nova/+/822053
13:35:31 opendevreview Jonathan Race proposed openstack/nova master: object/notification for Adds Pick guest CPU architecture based on host arch in libvirt driver support https://review.opendev.org/c/openstack/nova/+/828369
13:35:32 opendevreview Jonathan Race proposed openstack/nova master: zuul-job for Adds Pick guest CPU architecture based on host arch in libvirt driver support https://review.opendev.org/c/openstack/nova/+/828372
13:43:32 opendevreview Jonathan Race proposed openstack/nova master: object/notification for Adds Pick guest CPU architecture based on host arch in libvirt driver support https://review.opendev.org/c/openstack/nova/+/828369
13:43:33 opendevreview Jonathan Race proposed openstack/nova master: zuul-job for Adds Pick guest CPU architecture based on host arch in libvirt driver support https://review.opendev.org/c/openstack/nova/+/828372
13:43:33 opendevreview Jonathan Race proposed openstack/nova master: driver/secheduler/docs for Adds Pick guest CPU architecture based on host arch in libvirt driver support https://review.opendev.org/c/openstack/nova/+/822053
13:46:46 dmitriis where 0000:82:00.0 and 0000:82:00.1 are PF addresses.
13:46:46 dmitriis `passthrough_whitelist = {"vendor_id":"*", "product_id":"15b3", "address": "0000:82:00.1", "physical_network": null, "remote_managed": "true"}`
13:46:46 dmitriis `passthrough_whitelist = {"vendor_id":"*", "product_id":"15b3", "address": "0000:82:00.0", "physical_network": null, "remote_managed": "true"}`
13:46:46 dmitriis have passthrough_whitelist to use "VFs of a PF" type of config in two different containers:
13:46:46 dmitriis sean-k-mooney: One other way of working around this which might work is using two containers for Nova compute services (I've certainly ran Libvirt even in unprivileged containers before https://dshcherb.github.io/2017/12/04/qemu-kvm-virtual-machines-in-unprivileged-lxd.html). PCI device access isn't namespaced: we'd need privileged containers and
13:49:35 dmitriis That should give us two "compute hosts" with different hostnames. We would be able to test the control plane path at least while faking the notion of a different host. In the live migration case a QEMU process would migrate locally into a different container and get a VF allocated from a different PF.
13:51:17 sean-k-mooney dmitriis: ya you can do that
13:51:33 sean-k-mooney but its not quite the same as a real live migration to differnt hosts
13:51:40 sean-k-mooney however it would work
13:52:51 sean-k-mooney that is proably more work then i want to do to do some testing but i have a 2 port connectx-6 so i could consider it
13:53:19 sean-k-mooney you woudl need two copies of libvirt too
13:53:30 sean-k-mooney with different hostnames in each contaienr
13:53:55 sean-k-mooney its doable but a lot of work. hopefully you have access to 2 hosts and can test this properly
13:53:56 dmitriis sean-k-mooney: yeah, a UTS namespace should give me that. IPC is namespaced too so shouldn't run into an issue with unix sockets.
13:54:08 dmitriis sean-k-mooney: agreed, just testing the control path in this case. I have a host with ConnectX5 as well that can be used for the live migration case (from BF2 to ConnectX 5).
13:54:22 sean-k-mooney dmitriis: by the way are you working on updating the melonox sriov ci to test this
13:54:34 sean-k-mooney i think it proably should be extended to test this

Earlier   Later