Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-12-12
13:52:38 adrianc sean-k-mooney: i had a discussion with moshele about it, apparently this is an old issue which branches off this mail: https://www.redhat.com/archives/libvir-list/2016-September/msg00075.html
13:53:13 sean-k-mooney ah ok
13:53:32 sean-k-mooney so i had considered creating an os-vif plugin for sriov this cycle.
13:53:38 adrianc sriov netdev mac address will be re-allocated only on driver rebind on the VF
13:54:02 sean-k-mooney we can do it in the nova tree but we could also jsut create the plugin and have it set the vf down
13:54:26 adrianc if you do passthrough then it will be properly cleaned, however on macvtap this is not the case
13:54:36 sean-k-mooney ya
13:54:51 adrianc i added a hook in my code to do: interface down in vif.py under unplug_hw_veb()
13:55:14 sean-k-mooney ok cool that should work for now.
13:56:02 sean-k-mooney i would like to move that into os-vif later i also want to move the trusted vf logic into os-vif which was why i was orignally gong to create the sriov plugin
13:56:25 sean-k-mooney adrianc: do you have a new patchset up with that change. ill redeploy and test
13:58:34 adrianc not yet, i need to deploy the change and make sure it actually does the job. it worked manually.
14:00:30 adrianc sean-k-mooney: it should not be tied to the neutron-sriov-live-migration work as it serves as a WA to an existing problem.
14:01:44 adrianc let me know if doing ifconfig <netdev> down on the source vf netdev solves the issue for you. (after migration)
14:01:49 sean-k-mooney that is true but you could make it the base patch of the series so its easy to test all of them
14:02:13 sean-k-mooney sure i need to restack but ill let you know in an hour or so
14:02:27 adrianc sean-k-mooney: you will still need some POC code in neutron for multiple port binding
14:03:42 sean-k-mooney ya i am useing that patch too
14:04:02 sean-k-mooney devstack can deploy with unmerged patches from gerrit
14:04:23 sean-k-mooney but it cant combine two sets fo unrelated patches
14:04:54 sean-k-mooney so to test at the moment i jsut need to point to the top patch on the chain in nova and the neutron patch
14:05:26 adrianc can be applied manually after deploy :)
14:06:12 sean-k-mooney yep ill just create a branch locally and cherrypick them
14:06:18 sean-k-mooney its fine
14:28:47 mriedem melwitt: a change of mine hit the same vif plug timeout thing on that test with 7 ports http://logs.openstack.org/46/623246/3/check/nova-multiattach/efa830b/logs/screen-n-cpu.txt.gz?level=TRACE#_Dec_12_00_01_13_179474
14:29:36 mriedem bauzas: can you look at this? https://review.openstack.org/#/c/615347/ - it drops old confusing compat which is actually contributing to lots of misleading warnings in the n-api logs and gate race failures
14:29:48 bauzas mriedem: sure
14:29:52 sean-k-mooney mriedem ya i have seen that a few times
14:33:43 mriedem it's always the first port that we're missing
14:33:54 mriedem we get 6 of the 7 network-vif-plugged events
14:33:59 sean-k-mooney mriedem: one edgecase that i am awre of is in some configuration we start waithing for the network events too late
14:34:26 sean-k-mooney we start waiting for the event just before we call plug on the interface
14:34:44 sean-k-mooney for some neutron backend they send that event when we call bind
14:34:58 sean-k-mooney i assume this is ml2/ovs however correct?
14:35:15 sean-k-mooney that should not hit the race but i wonde if ther i a missed event in the logs
14:35:24 mriedem yes ovs
14:37:32 sean-k-mooney mriedem: http://logs.openstack.org/46/623246/3/check/nova-multiattach/efa830b/logs/screen-n-cpu.txt.gz?#_Dec_12_00_01_07_481084
14:37:39 mriedem we plug it here
14:37:39 mriedem Dec 11 23:56:11.855811 ubuntu-xenial-inap-mtl01-0001136812 nova-compute[29399]: INFO os_vif [None req-f43f3c35-dc0b-4aa4-bfd0-4046045b315e tempest-TaggedBootDevicesTest-250467933 tempest-TaggedBootDevicesTest-250467933] Successfully plugged vif VIFOpenVSwitch(active=False,address=fa:16:3e:fb:dd:9f,bridge_name='br-int',has_traffic_filtering=True,id=472ab433-bc18-41b5-85ae-009611117b70,network=Network(4988ef3d-6a8f-477f-b0a2-29
14:37:39 mriedem 019494),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap472ab433-bc')
14:38:20 mriedem hmm wtf
14:38:39 mriedem i also see this right before that port is plugged
14:38:39 mriedem Dec 11 23:55:46.981194 ubuntu-xenial-inap-mtl01-0001136812 neutron-server[20899]: DEBUG neutron.notifiers.nova [-] Sending events: [{'tag': u'472ab433-bc18-41b5-85ae-009611117b70', 'name': 'network-changed', 'server_uuid': u'18e5987e-a535-4044-84aa-dd8f023cedcc'}] {{(pid=21001) send_events /opt/stack/new/neutron/neutron/notifiers/nova.py:245}}
14:39:31 mriedem sean-k-mooney: that's a different port
14:39:35 mriedem the unplugged event
14:39:55 sean-k-mooney mriedem: yes i just noticed that
14:42:23 mriedem asking in -neutron
14:53:58 frickler nova stable cores: https://review.openstack.org/619254 is waiting for a second +2, if it gets that, I can start nagging mriedem about stable releases instead ;-)
15:07:52 jangutter mriedem: yonks ago I saw something similar that happened if the neutron-ovsdb linkage got busted.
15:09:31 mriedem jangutter: i did see this in the neutron agent logs
15:09:31 mriedem Dec 11 23:56:12.725010 ubuntu-xenial-inap-mtl01-0001136812 neutron-openvswitch-agent[21879]: INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [None req-cc26f856-2521-411e-ae10-34ad96b7c665 None None] Port 472ab433-bc18-41b5-85ae-009611117b70 was not found on the integration bridge and will therefore not be processed
15:10:35 jangutter mriedem: there's some extra metadata added by libvirt that links the port uuid. years ago I saw neutron triggering off of that.
15:11:13 adrianc sean-k-mooney: OK, so apparently on my setup i need to unbind/bind the VF, taking down the interface is not sufficient. this may be NIC vendor specific, let me know if interface down works for you OR you need to rebind the VF to flush out the MAC from the source node
15:21:36 sean-k-mooney adrianc: rebinding the vf will be a problem if needed
15:22:03 sean-k-mooney nova doen not always have permissions to do that.
15:23:01 mriedem hello people - easy fix to get the postgresql job working again https://review.openstack.org/#/c/619061/
15:23:22 mriedem cdent: ^ since you seem to care about pg
15:23:26 mriedem slaweq: you too ^
15:23:41 adrianc sean-k-mooney: the fix should be in libvirt imo, once the guest is not on the machine, it should properly clean-up. however for the direct SRIOV you will not have this problem
15:24:07 cdent mriedem: it's less care about pg, more care about not-just-mysql
15:24:10 cdent but yeah, looking
15:24:17 sean-k-mooney adrianc: well libvirt or qemu but yes one of the two shoudl notify the pf to reset the vf
15:25:05 sean-k-mooney adrianc: yes direct mode should work as the vf will be reset when it detached form the guest kernel and released to the host kernel
15:25:09 jangutter sean-k-mooney, adrianc: I think qemu does a function-level-reset on the VF when doing iommu passthrough.
15:25:27 sean-k-mooney jangutter: on a live migration
15:25:34 jangutter sean-k-mooney, adrianc: and libvirt does driver binding and unbinding.
15:25:57 sean-k-mooney jangutter: in the macvtap case you do not bind the diriver to vfio-pci
15:26:09 slaweq mriedem: it is this fix which will finally fix neutron pgsql periodic job, right?
15:26:09 sean-k-mooney jangutter: you leave it bound to the host kernel driver at all times
15:26:23 jangutter sean-k-mooney: yep, but permission-wise, libvirt could bind/unbind it as a workaround.
15:26:46 jangutter sean-k-mooney: and generally, you _don't_ want to do a FLR on something bound to a driver.
15:26:53 sean-k-mooney jangutter: it could yes but it wont today meaning we would need a new libvirt release and need to check for it in the code
15:27:16 jangutter sean-k-mooney: yep, tell me about it, I know all about people wanting to propagate libvirt "fixes".
15:28:01 sean-k-mooney i think this would be somthing that the sriov nic agent could resonalby handel too
15:28:10 sean-k-mooney well maybe not
15:28:15 sean-k-mooney that could be racy
15:29:02 sean-k-mooney it would have to be nova/libvirt to avoid races with other vms booting
15:29:09 adrianc sean-k-mooney: there is no FLR involved during detach/attach, the driver unbinds from the guest, and binds on the host whether or not it triggers an FLR is something the driver decides AFAIK
15:29:39 jangutter adrianc: yep, FLR is mandatory on iommu passthrough I think, but not for driver bind/unbind
15:30:32 sean-k-mooney well it depend on if the vf is bound to say ixgbeVf when its detach or if it stays with vfio-pci
15:30:54 jangutter sean-k-mooney: yeah.
15:31:59 adrianc would have been easier if setting VF mac address via ip link on the VF idx would have synced the mac address on the corresponding VF netdev :)
15:32:00 sean-k-mooney in any case however in macvtap mode we aparent dont update the driver the vf is bound to or trigger a vf reset
15:32:33 sean-k-mooney adrianc: that silly talk why would a hardware/driver implementer ever do a thing like that :P
15:32:33 openstackgerrit Balazs Gibizer proposed openstack/nova master: Add python 3.7 unit and functional tox jobs https://review.openstack.org/624055
15:32:34 openstackgerrit Balazs Gibizer proposed openstack/nova master: Replace ThreadPoolExecutor with GreenThreadPoolExecutor https://review.openstack.org/624740
15:32:39 gibi cdent: ^^
15:32:48 cdent roger
15:33:16 mriedem slaweq: yeah
15:33:22 jangutter sean-k-mooney, adrianc: years ago we had looked at how to propagate MAC config from host to the VF and the conclusion we came to was that at the FLR is the safest way to do so. I don't think there was an async event able to propagate that info from the device to the driver without either a reset or rebind.
15:33:42 slaweq mriedem: thx for taking care of it :)
15:33:47 sean-k-mooney adrianc: speacking of ip link ther eis no way to reset it via ip tools right.
15:34:17 jangutter sean-k-mooney, adrianc: the only way I know to trigger an FLR is via setpci...
15:35:02 sean-k-mooney jangutter: i was hoping there would be a standard sysfs file we could echo too but i guess not
15:35:18 aspiers mriedem: if I understand the code right, the config drive disk mapping is built alongside the other disk mappings prior to instance launch (even though the config drive itself seems to be built late via post_xml_callback). so it's not hotplugged post-launch. Also I think it will either use "ide" or "virtio" bus (depending on whether iso9660 or vfat) and therefore we don't anticipate any problems with
15:35:19 aspiers SEV. Nevertheless we're testing to make sure
15:35:22 jangutter sean-k-mooney: there might be these days, though.
15:35:40 sean-k-mooney jangutter: it would still likely be vendor specific
15:36:15 jangutter sean-k-mooney: very, I believe our driver only recently could propagate MAC on rebind. Previously you _had_ to get an FLR.
15:36:20 sean-k-mooney adrianc: im just getting around to setting up that test env now but i will try a few different things and let you know

Earlier   Later