Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2022-02-15
11:26:52 gibi and you have a point, if we actually update the mac in the db then we can simplify the logic in neutron. initially I planned to just overlay the binding mac over the port mac, but that is complicated from implementation perspective
11:28:27 gibi and one extra conditional that we only need to do this whole thing to PFs (direct-physical) ports. In the rest of the case the mac flows from neutron to the device, but for PF the mac flows from the device to neturon
11:29:05 sean-k-mooney well if we want to do that we can basicaly have neutron do (if host_id == profile:migrating_to then old_mac=mac; mac=profile:migration_mac profile:old_mac=old_mac;
11:29:18 sean-k-mooney )
11:30:33 sean-k-mooney we would need to record the old mac for revert althoug we wont have cleared the souce pci device allcoation so we might not need old_mac we can just look that up if we revert on the source
11:30:33 gibi I think we don't even have to save the old mac. either it is in an inactive binding already or the port is being unbound
11:31:00 gibi when neutron unbinds a PF it generates a random mac for it
11:31:16 sean-k-mooney ya we dont for cold migration we have it and for live migration we eitehr activate teh dest binding and never rever or we revert before the mac update
11:31:43 sean-k-mooney gibi: no nueton does not generate random macs for pfs
11:31:43 gibi but if we revert then we revert to the source binding that will have the source mac
11:31:54 sean-k-mooney it generate a mac when you first create the port
11:32:01 sean-k-mooney but that is the same for any port
11:32:15 sean-k-mooney it does not generate random macs after that when you change the bining
11:32:55 gibi sean-k-mooney: a/neutron/plugins/ml2/plugin.py
11:33:01 gibi sean-k-mooney: https://github.com/openstack/neutron/blob/master/neutron/plugins/ml2/plugin.py#L438
11:33:16 sean-k-mooney that looks like a bug
11:33:24 gibi called from https://github.com/openstack/neutron/blob/master/neutron/plugins/ml2/plugin.py#L1782
11:33:45 sean-k-mooney actully it looks like they tried to work around this once before by adding that
11:34:28 gibi today nova writes the port.mac_address for PF ports before the first binding. So if there won't be a reset in neutron then after unbind the port would keep the mac of a real device
11:34:46 sean-k-mooney https://bugs.launchpad.net/neutron/+bug/1830383
11:35:00 sean-k-mooney ya so that code was an incorrect fix for the current issue your trying to solve
11:35:16 sean-k-mooney gibi: that should be reverted
11:35:36 gibi but then what mac a PF port should have after it is unbound
11:35:38 gibi ?
11:35:48 gibi empty?
11:35:51 sean-k-mooney the same one it had when it was bound
11:35:59 sean-k-mooney nova is still telling the vm to use that mac
11:36:08 sean-k-mooney in the cold migration case
11:36:20 gibi hm
11:36:56 sean-k-mooney i can see why they did this but today since we are not updating the mac on the neutron port in the cold migration case this only help in the event we detach the port
11:36:57 gibi so if I boot a VM with a PF port, then that port gets the MAC of the PF today. Then I delete the VM. The port should forget the MAC of the PF
11:37:30 sean-k-mooney ya in that case the currnt reset hack works
11:37:40 gibi so we have to keep that reset for the unbinding case
11:37:48 sean-k-mooney we can yes
11:38:10 gibi for the rebinding there will be a new MAC from nova
11:38:49 sean-k-mooney however if you have 2 vm and 3 hosts and you move vm a form host a to host c and vm b form host b to host a today
11:39:01 sean-k-mooney vm b will get a dhcp responce for vm a
11:39:28 sean-k-mooney because in neutorn we are not updating the macs and vm a still has a port with the mac for port b
11:39:36 sean-k-mooney *for host a
11:39:49 sean-k-mooney assuming vm a and b are on the same netork
11:40:28 sean-k-mooney so the reset is not happeing today when we unbindi the port unle when we detach the port
11:40:59 gibi I have verified it. When I delete the VM neutron resets the MAC
11:41:02 gibi today
11:41:05 sean-k-mooney well i guess it depend on how you define "unbind"
11:41:09 sean-k-mooney yep
11:41:53 sean-k-mooney so if you update the host-id today the port will breifly go to vif_type=unbound
11:42:13 sean-k-mooney so i consider updating the host-id to be unbinding/rebinding the port
11:42:14 gibi but yes if I migrate the VM (and not delet it) then I can create the mixed MAC case you described
11:42:48 sean-k-mooney updating the device_id/device_onwer to '' i do not consider to be unbinding the port
11:42:55 sean-k-mooney that is detaching the port
11:43:56 gibi ahh I see
11:44:31 sean-k-mooney gibi: sorry if im beign pedantic with the terminology there it just the device_id and device_owner filed are not part of the port_bindign or port_binding_extended api extentions
11:44:43 gibi true
11:44:54 gibi just the host_id
11:45:04 gibi but I think that is OK
11:45:14 sean-k-mooney https://github.com/openstack/neutron-lib/blob/master/neutron_lib/api/definitions/portbindings_extended.py#L75-L105 yes
11:45:44 sean-k-mooney so that is why updating the host-id in the cold migrat case set the vif_type to unbound and then to the correct vif
11:45:59 sean-k-mooney it first unbindign the port in the ml2 driver then binds it again
11:46:13 sean-k-mooney potentially to a different ml2 driver
11:46:52 sean-k-mooney e.g. from linux bridge to ovs or whatever if both dirver supprot the vnic_type and you have different ones on different hosts
11:47:17 sean-k-mooney gibi when it the neutron meeting
11:47:59 gibi 14:00 UTC today
11:48:02 gibi afaik
11:48:04 sean-k-mooney i can try and attend but i think we are more or less on teh same page
11:48:21 sean-k-mooney ok so basicaly just over 2 hours form now
11:50:51 gibi no worries
11:51:08 gibi I agree that we are on the same page
11:51:55 sean-k-mooney is this something you will have time to work on
11:52:58 sean-k-mooney i have not booted the old servers i used to use for sriov work in the better part of a year maybe more so not sure if they are still working. i can review and maybe try and test the fix at some point but likely not until after FF
11:53:26 sean-k-mooney we will see how much progress i make on the health checks this week/next
11:53:52 sean-k-mooney im not sure if they will get done before FF but if not im hoping early next cycle they can land
11:54:50 sean-k-mooney gibi: by the way there is a slim change i can partly test dmitriis off-path code too.
11:55:46 sean-k-mooney i dont have bluefield 2 nics but i have a conectx-6 and if i lie to ovn and nova about it beign remote managed i should be able to test it end to end
11:56:06 sean-k-mooney but again likely wont get to that until FF has passed but im hoping to do it before RC1
11:56:33 gibi sean-k-mooney: I have priority on this bug now from downstream I will work on it. But it should not effect me reviewing patches before FF
11:56:57 gibi so if you have anything just link me up
11:57:20 dmitriis sean-k-mooney: I am looking into adding more func tests and into doing a multi-node test. Unfortunately, SR-IOV is hidden by the QEMU's VFIO layer when PFs are passed through to a VM so I cannot reuse 2 PFs on the same host to fake 2 computes.
11:57:25 dmitriis brb in ~ 1h
11:57:42 sean-k-mooney dmitriis: yep it is
11:57:44 gibi I cannot test dmitriis patch I have only old SRIOV capable PFs
11:58:01 sean-k-mooney dmitriis: you can pass the PF to the vm but it nologner has the sriov extion
11:58:08 dmitriis https://git.qemu.org/?p=qemu.git;a=commitdiff;h=e37dac06dc4e85a2f46c24261c0dfdf2a30b50e3
11:58:08 dmitriis Still there in recent versions of QEMU: https://github.com/qemu/qemu/blob/v6.2.0/hw/vfio/pci.c#L2094-L2097
11:58:27 dmitriis this landed upstream https://lore.kernel.org/linux-pci/20200319184653.6c10638b.cohuck@redhat.com/T/ but the QEMU limitation is still there
11:58:28 sean-k-mooney oh..
11:59:08 opendevreview Merged openstack/nova stable/xena: libvirt: Add announce-self post live-migration workaround https://review.opendev.org/c/openstack/nova/+/825064
11:59:08 dmitriis (jumped into a meeting)
11:59:30 sean-k-mooney i wonder what happens if you jsut remove the sriov case there
12:01:37 sean-k-mooney /* Read-only VF BARs confuse OVMF */ implies its a uefi limitation
12:02:35 sean-k-mooney https://github.com/qemu/qemu/commit/e37dac06dc4e85a2f46c24261c0dfdf2a30b50e3
13:03:01 opendevreview David Hill proposed openstack/nova master: Validate burst value is equal or smaller than 4194303 https://review.opendev.org/c/openstack/nova/+/829064
13:14:47 dmitriis sean-k-mooney: yeah, the commit message says "OVMF tries to size the VF BARs and comes up with absurd results, ending with an assert". Haven't looked at the OVMF in-depth but I suspect it might be around this code which haven't seen much change besides automatic style fixes in 13 years:
13:14:47 dmitriis https://github.com/tianocore/edk2/blame/c095122d4b5f3152417cd97dabecfe31cc3b6508/MdeModulePkg/Bus/Pci/PciBusDxe/PciEnumeratorSupport.c#L494-L502
13:14:47 dmitriis https://github.com/tianocore/edk2/blame/c095122d4b5f3152417cd97dabecfe31cc3b6508/MdeModulePkg/Bus/Pci/PciBusDxe/PciEnumeratorSupport.c#L1554-L1578
13:35:31 opendevreview Jonathan Race proposed openstack/nova master: object/notification for Adds Pick guest CPU architecture based on host arch in libvirt driver support https://review.opendev.org/c/openstack/nova/+/828369
13:35:31 opendevreview Jonathan Race proposed openstack/nova master: driver/secheduler/docs for Adds Pick guest CPU architecture based on host arch in libvirt driver support https://review.opendev.org/c/openstack/nova/+/822053
13:35:32 opendevreview Jonathan Race proposed openstack/nova master: zuul-job for Adds Pick guest CPU architecture based on host arch in libvirt driver support https://review.opendev.org/c/openstack/nova/+/828372
13:43:32 opendevreview Jonathan Race proposed openstack/nova master: object/notification for Adds Pick guest CPU architecture based on host arch in libvirt driver support https://review.opendev.org/c/openstack/nova/+/828369
13:43:33 opendevreview Jonathan Race proposed openstack/nova master: driver/secheduler/docs for Adds Pick guest CPU architecture based on host arch in libvirt driver support https://review.opendev.org/c/openstack/nova/+/822053
13:43:33 opendevreview Jonathan Race proposed openstack/nova master: zuul-job for Adds Pick guest CPU architecture based on host arch in libvirt driver support https://review.opendev.org/c/openstack/nova/+/828372

Earlier   Later