| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-12-12 | |||
| 13:46:27 | sean-k-mooney | :) | |
| 13:49:02 | adrianc | sean-k-mooney: so apparently there is some weird behavior with sriov macvtap when you tear down the VM the VF netdev remains up | |
| 13:49:33 | adrianc | so you have a scenario of duplicate macs in the network | |
| 13:50:22 | adrianc | doing ifconfig <VF-netdev> down on the source VF should fix the issue (at least on my setup it did) | |
| 13:52:38 | adrianc | sean-k-mooney: i had a discussion with moshele about it, apparently this is an old issue which branches off this mail: https://www.redhat.com/archives/libvir-list/2016-September/msg00075.html | |
| 13:53:13 | sean-k-mooney | ah ok | |
| 13:53:32 | sean-k-mooney | so i had considered creating an os-vif plugin for sriov this cycle. | |
| 13:53:38 | adrianc | sriov netdev mac address will be re-allocated only on driver rebind on the VF | |
| 13:54:02 | sean-k-mooney | we can do it in the nova tree but we could also jsut create the plugin and have it set the vf down | |
| 13:54:26 | adrianc | if you do passthrough then it will be properly cleaned, however on macvtap this is not the case | |
| 13:54:36 | sean-k-mooney | ya | |
| 13:54:51 | adrianc | i added a hook in my code to do: interface down in vif.py under unplug_hw_veb() | |
| 13:55:14 | sean-k-mooney | ok cool that should work for now. | |
| 13:56:02 | sean-k-mooney | i would like to move that into os-vif later i also want to move the trusted vf logic into os-vif which was why i was orignally gong to create the sriov plugin | |
| 13:56:25 | sean-k-mooney | adrianc: do you have a new patchset up with that change. ill redeploy and test | |
| 13:58:34 | adrianc | not yet, i need to deploy the change and make sure it actually does the job. it worked manually. | |
| 14:00:30 | adrianc | sean-k-mooney: it should not be tied to the neutron-sriov-live-migration work as it serves as a WA to an existing problem. | |
| 14:01:44 | adrianc | let me know if doing ifconfig <netdev> down on the source vf netdev solves the issue for you. (after migration) | |
| 14:01:49 | sean-k-mooney | that is true but you could make it the base patch of the series so its easy to test all of them | |
| 14:02:13 | sean-k-mooney | sure i need to restack but ill let you know in an hour or so | |
| 14:02:27 | adrianc | sean-k-mooney: you will still need some POC code in neutron for multiple port binding | |
| 14:03:42 | sean-k-mooney | ya i am useing that patch too | |
| 14:04:02 | sean-k-mooney | devstack can deploy with unmerged patches from gerrit | |
| 14:04:23 | sean-k-mooney | but it cant combine two sets fo unrelated patches | |
| 14:04:54 | sean-k-mooney | so to test at the moment i jsut need to point to the top patch on the chain in nova and the neutron patch | |
| 14:05:26 | adrianc | can be applied manually after deploy :) | |
| 14:06:12 | sean-k-mooney | yep ill just create a branch locally and cherrypick them | |
| 14:06:18 | sean-k-mooney | its fine | |
| 14:28:47 | mriedem | melwitt: a change of mine hit the same vif plug timeout thing on that test with 7 ports http://logs.openstack.org/46/623246/3/check/nova-multiattach/efa830b/logs/screen-n-cpu.txt.gz?level=TRACE#_Dec_12_00_01_13_179474 | |
| 14:29:36 | mriedem | bauzas: can you look at this? https://review.openstack.org/#/c/615347/ - it drops old confusing compat which is actually contributing to lots of misleading warnings in the n-api logs and gate race failures | |
| 14:29:48 | bauzas | mriedem: sure | |
| 14:29:52 | sean-k-mooney | mriedem ya i have seen that a few times | |
| 14:33:43 | mriedem | it's always the first port that we're missing | |
| 14:33:54 | mriedem | we get 6 of the 7 network-vif-plugged events | |
| 14:33:59 | sean-k-mooney | mriedem: one edgecase that i am awre of is in some configuration we start waithing for the network events too late | |
| 14:34:26 | sean-k-mooney | we start waiting for the event just before we call plug on the interface | |
| 14:34:44 | sean-k-mooney | for some neutron backend they send that event when we call bind | |
| 14:34:58 | sean-k-mooney | i assume this is ml2/ovs however correct? | |
| 14:35:15 | sean-k-mooney | that should not hit the race but i wonde if ther i a missed event in the logs | |
| 14:35:24 | mriedem | yes ovs | |
| 14:37:32 | sean-k-mooney | mriedem: http://logs.openstack.org/46/623246/3/check/nova-multiattach/efa830b/logs/screen-n-cpu.txt.gz?#_Dec_12_00_01_07_481084 | |
| 14:37:39 | mriedem | we plug it here | |
| 14:37:39 | mriedem | Dec 11 23:56:11.855811 ubuntu-xenial-inap-mtl01-0001136812 nova-compute[29399]: INFO os_vif [None req-f43f3c35-dc0b-4aa4-bfd0-4046045b315e tempest-TaggedBootDevicesTest-250467933 tempest-TaggedBootDevicesTest-250467933] Successfully plugged vif VIFOpenVSwitch(active=False,address=fa:16:3e:fb:dd:9f,bridge_name='br-int',has_traffic_filtering=True,id=472ab433-bc18-41b5-85ae-009611117b70,network=Network(4988ef3d-6a8f-477f-b0a2-29 | |
| 14:37:39 | mriedem | 019494),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap472ab433-bc') | |
| 14:38:20 | mriedem | hmm wtf | |
| 14:38:39 | mriedem | i also see this right before that port is plugged | |
| 14:38:39 | mriedem | Dec 11 23:55:46.981194 ubuntu-xenial-inap-mtl01-0001136812 neutron-server[20899]: DEBUG neutron.notifiers.nova [-] Sending events: [{'tag': u'472ab433-bc18-41b5-85ae-009611117b70', 'name': 'network-changed', 'server_uuid': u'18e5987e-a535-4044-84aa-dd8f023cedcc'}] {{(pid=21001) send_events /opt/stack/new/neutron/neutron/notifiers/nova.py:245}} | |
| 14:39:31 | mriedem | sean-k-mooney: that's a different port | |
| 14:39:35 | mriedem | the unplugged event | |
| 14:39:55 | sean-k-mooney | mriedem: yes i just noticed that | |
| 14:42:23 | mriedem | asking in -neutron | |
| 14:53:58 | frickler | nova stable cores: https://review.openstack.org/619254 is waiting for a second +2, if it gets that, I can start nagging mriedem about stable releases instead ;-) | |
| 15:07:52 | jangutter | mriedem: yonks ago I saw something similar that happened if the neutron-ovsdb linkage got busted. | |
| 15:09:31 | mriedem | jangutter: i did see this in the neutron agent logs | |
| 15:09:31 | mriedem | Dec 11 23:56:12.725010 ubuntu-xenial-inap-mtl01-0001136812 neutron-openvswitch-agent[21879]: INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [None req-cc26f856-2521-411e-ae10-34ad96b7c665 None None] Port 472ab433-bc18-41b5-85ae-009611117b70 was not found on the integration bridge and will therefore not be processed | |
| 15:10:35 | jangutter | mriedem: there's some extra metadata added by libvirt that links the port uuid. years ago I saw neutron triggering off of that. | |
| 15:11:13 | adrianc | sean-k-mooney: OK, so apparently on my setup i need to unbind/bind the VF, taking down the interface is not sufficient. this may be NIC vendor specific, let me know if interface down works for you OR you need to rebind the VF to flush out the MAC from the source node | |
| 15:21:36 | sean-k-mooney | adrianc: rebinding the vf will be a problem if needed | |
| 15:22:03 | sean-k-mooney | nova doen not always have permissions to do that. | |
| 15:23:01 | mriedem | hello people - easy fix to get the postgresql job working again https://review.openstack.org/#/c/619061/ | |
| 15:23:22 | mriedem | cdent: ^ since you seem to care about pg | |
| 15:23:26 | mriedem | slaweq: you too ^ | |
| 15:23:41 | adrianc | sean-k-mooney: the fix should be in libvirt imo, once the guest is not on the machine, it should properly clean-up. however for the direct SRIOV you will not have this problem | |
| 15:24:07 | cdent | mriedem: it's less care about pg, more care about not-just-mysql | |
| 15:24:10 | cdent | but yeah, looking | |
| 15:24:17 | sean-k-mooney | adrianc: well libvirt or qemu but yes one of the two shoudl notify the pf to reset the vf | |
| 15:25:05 | sean-k-mooney | adrianc: yes direct mode should work as the vf will be reset when it detached form the guest kernel and released to the host kernel | |
| 15:25:09 | jangutter | sean-k-mooney, adrianc: I think qemu does a function-level-reset on the VF when doing iommu passthrough. | |
| 15:25:27 | sean-k-mooney | jangutter: on a live migration | |
| 15:25:34 | jangutter | sean-k-mooney, adrianc: and libvirt does driver binding and unbinding. | |
| 15:25:57 | sean-k-mooney | jangutter: in the macvtap case you do not bind the diriver to vfio-pci | |
| 15:26:09 | slaweq | mriedem: it is this fix which will finally fix neutron pgsql periodic job, right? | |
| 15:26:09 | sean-k-mooney | jangutter: you leave it bound to the host kernel driver at all times | |
| 15:26:23 | jangutter | sean-k-mooney: yep, but permission-wise, libvirt could bind/unbind it as a workaround. | |
| 15:26:46 | jangutter | sean-k-mooney: and generally, you _don't_ want to do a FLR on something bound to a driver. | |
| 15:26:53 | sean-k-mooney | jangutter: it could yes but it wont today meaning we would need a new libvirt release and need to check for it in the code | |
| 15:27:16 | jangutter | sean-k-mooney: yep, tell me about it, I know all about people wanting to propagate libvirt "fixes". | |
| 15:28:01 | sean-k-mooney | i think this would be somthing that the sriov nic agent could resonalby handel too | |
| 15:28:10 | sean-k-mooney | well maybe not | |
| 15:28:15 | sean-k-mooney | that could be racy | |
| 15:29:02 | sean-k-mooney | it would have to be nova/libvirt to avoid races with other vms booting | |
| 15:29:09 | adrianc | sean-k-mooney: there is no FLR involved during detach/attach, the driver unbinds from the guest, and binds on the host whether or not it triggers an FLR is something the driver decides AFAIK | |
| 15:29:39 | jangutter | adrianc: yep, FLR is mandatory on iommu passthrough I think, but not for driver bind/unbind | |
| 15:30:32 | sean-k-mooney | well it depend on if the vf is bound to say ixgbeVf when its detach or if it stays with vfio-pci | |
| 15:30:54 | jangutter | sean-k-mooney: yeah. | |
| 15:31:59 | adrianc | would have been easier if setting VF mac address via ip link on the VF idx would have synced the mac address on the corresponding VF netdev :) | |
| 15:32:00 | sean-k-mooney | in any case however in macvtap mode we aparent dont update the driver the vf is bound to or trigger a vf reset | |
| 15:32:33 | sean-k-mooney | adrianc: that silly talk why would a hardware/driver implementer ever do a thing like that :P | |
| 15:32:33 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add python 3.7 unit and functional tox jobs https://review.openstack.org/624055 | |
| 15:32:34 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Replace ThreadPoolExecutor with GreenThreadPoolExecutor https://review.openstack.org/624740 | |
| 15:32:39 | gibi | cdent: ^^ | |
| 15:32:48 | cdent | roger | |
| 15:33:16 | mriedem | slaweq: yeah | |
| 15:33:22 | jangutter | sean-k-mooney, adrianc: years ago we had looked at how to propagate MAC config from host to the VF and the conclusion we came to was that at the FLR is the safest way to do so. I don't think there was an async event able to propagate that info from the device to the driver without either a reset or rebind. | |
| 15:33:42 | slaweq | mriedem: thx for taking care of it :) | |
| 15:33:47 | sean-k-mooney | adrianc: speacking of ip link ther eis no way to reset it via ip tools right. | |
| 15:34:17 | jangutter | sean-k-mooney, adrianc: the only way I know to trigger an FLR is via setpci... | |
| 15:35:02 | sean-k-mooney | jangutter: i was hoping there would be a standard sysfs file we could echo too but i guess not | |
| 15:35:18 | aspiers | mriedem: if I understand the code right, the config drive disk mapping is built alongside the other disk mappings prior to instance launch (even though the config drive itself seems to be built late via post_xml_callback). so it's not hotplugged post-launch. Also I think it will either use "ide" or "virtio" bus (depending on whether iso9660 or vfat) and therefore we don't anticipate any problems with | |
| 15:35:19 | aspiers | SEV. Nevertheless we're testing to make sure | |