| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-12-12 | |||
| 14:06:12 | sean-k-mooney | yep ill just create a branch locally and cherrypick them | |
| 14:06:18 | sean-k-mooney | its fine | |
| 14:28:47 | mriedem | melwitt: a change of mine hit the same vif plug timeout thing on that test with 7 ports http://logs.openstack.org/46/623246/3/check/nova-multiattach/efa830b/logs/screen-n-cpu.txt.gz?level=TRACE#_Dec_12_00_01_13_179474 | |
| 14:29:36 | mriedem | bauzas: can you look at this? https://review.openstack.org/#/c/615347/ - it drops old confusing compat which is actually contributing to lots of misleading warnings in the n-api logs and gate race failures | |
| 14:29:48 | bauzas | mriedem: sure | |
| 14:29:52 | sean-k-mooney | mriedem ya i have seen that a few times | |
| 14:33:43 | mriedem | it's always the first port that we're missing | |
| 14:33:54 | mriedem | we get 6 of the 7 network-vif-plugged events | |
| 14:33:59 | sean-k-mooney | mriedem: one edgecase that i am awre of is in some configuration we start waithing for the network events too late | |
| 14:34:26 | sean-k-mooney | we start waiting for the event just before we call plug on the interface | |
| 14:34:44 | sean-k-mooney | for some neutron backend they send that event when we call bind | |
| 14:34:58 | sean-k-mooney | i assume this is ml2/ovs however correct? | |
| 14:35:15 | sean-k-mooney | that should not hit the race but i wonde if ther i a missed event in the logs | |
| 14:35:24 | mriedem | yes ovs | |
| 14:37:32 | sean-k-mooney | mriedem: http://logs.openstack.org/46/623246/3/check/nova-multiattach/efa830b/logs/screen-n-cpu.txt.gz?#_Dec_12_00_01_07_481084 | |
| 14:37:39 | mriedem | 019494),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap472ab433-bc') | |
| 14:37:39 | mriedem | Dec 11 23:56:11.855811 ubuntu-xenial-inap-mtl01-0001136812 nova-compute[29399]: INFO os_vif [None req-f43f3c35-dc0b-4aa4-bfd0-4046045b315e tempest-TaggedBootDevicesTest-250467933 tempest-TaggedBootDevicesTest-250467933] Successfully plugged vif VIFOpenVSwitch(active=False,address=fa:16:3e:fb:dd:9f,bridge_name='br-int',has_traffic_filtering=True,id=472ab433-bc18-41b5-85ae-009611117b70,network=Network(4988ef3d-6a8f-477f-b0a2-29 | |
| 14:37:39 | mriedem | we plug it here | |
| 14:38:20 | mriedem | hmm wtf | |
| 14:38:39 | mriedem | Dec 11 23:55:46.981194 ubuntu-xenial-inap-mtl01-0001136812 neutron-server[20899]: DEBUG neutron.notifiers.nova [-] Sending events: [{'tag': u'472ab433-bc18-41b5-85ae-009611117b70', 'name': 'network-changed', 'server_uuid': u'18e5987e-a535-4044-84aa-dd8f023cedcc'}] {{(pid=21001) send_events /opt/stack/new/neutron/neutron/notifiers/nova.py:245}} | |
| 14:38:39 | mriedem | i also see this right before that port is plugged | |
| 14:39:31 | mriedem | sean-k-mooney: that's a different port | |
| 14:39:35 | mriedem | the unplugged event | |
| 14:39:55 | sean-k-mooney | mriedem: yes i just noticed that | |
| 14:42:23 | mriedem | asking in -neutron | |
| 14:53:58 | frickler | nova stable cores: https://review.openstack.org/619254 is waiting for a second +2, if it gets that, I can start nagging mriedem about stable releases instead ;-) | |
| 15:07:52 | jangutter | mriedem: yonks ago I saw something similar that happened if the neutron-ovsdb linkage got busted. | |
| 15:09:31 | mriedem | Dec 11 23:56:12.725010 ubuntu-xenial-inap-mtl01-0001136812 neutron-openvswitch-agent[21879]: INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [None req-cc26f856-2521-411e-ae10-34ad96b7c665 None None] Port 472ab433-bc18-41b5-85ae-009611117b70 was not found on the integration bridge and will therefore not be processed | |
| 15:09:31 | mriedem | jangutter: i did see this in the neutron agent logs | |
| 15:10:35 | jangutter | mriedem: there's some extra metadata added by libvirt that links the port uuid. years ago I saw neutron triggering off of that. | |
| 15:11:13 | adrianc | sean-k-mooney: OK, so apparently on my setup i need to unbind/bind the VF, taking down the interface is not sufficient. this may be NIC vendor specific, let me know if interface down works for you OR you need to rebind the VF to flush out the MAC from the source node | |
| 15:21:36 | sean-k-mooney | adrianc: rebinding the vf will be a problem if needed | |
| 15:22:03 | sean-k-mooney | nova doen not always have permissions to do that. | |
| 15:23:01 | mriedem | hello people - easy fix to get the postgresql job working again https://review.openstack.org/#/c/619061/ | |
| 15:23:22 | mriedem | cdent: ^ since you seem to care about pg | |
| 15:23:26 | mriedem | slaweq: you too ^ | |
| 15:23:41 | adrianc | sean-k-mooney: the fix should be in libvirt imo, once the guest is not on the machine, it should properly clean-up. however for the direct SRIOV you will not have this problem | |
| 15:24:07 | cdent | mriedem: it's less care about pg, more care about not-just-mysql | |
| 15:24:10 | cdent | but yeah, looking | |
| 15:24:17 | sean-k-mooney | adrianc: well libvirt or qemu but yes one of the two shoudl notify the pf to reset the vf | |
| 15:25:05 | sean-k-mooney | adrianc: yes direct mode should work as the vf will be reset when it detached form the guest kernel and released to the host kernel | |
| 15:25:09 | jangutter | sean-k-mooney, adrianc: I think qemu does a function-level-reset on the VF when doing iommu passthrough. | |
| 15:25:27 | sean-k-mooney | jangutter: on a live migration | |
| 15:25:34 | jangutter | sean-k-mooney, adrianc: and libvirt does driver binding and unbinding. | |
| 15:25:57 | sean-k-mooney | jangutter: in the macvtap case you do not bind the diriver to vfio-pci | |
| 15:26:09 | sean-k-mooney | jangutter: you leave it bound to the host kernel driver at all times | |
| 15:26:09 | slaweq | mriedem: it is this fix which will finally fix neutron pgsql periodic job, right? | |
| 15:26:23 | jangutter | sean-k-mooney: yep, but permission-wise, libvirt could bind/unbind it as a workaround. | |
| 15:26:46 | jangutter | sean-k-mooney: and generally, you _don't_ want to do a FLR on something bound to a driver. | |
| 15:26:53 | sean-k-mooney | jangutter: it could yes but it wont today meaning we would need a new libvirt release and need to check for it in the code | |
| 15:27:16 | jangutter | sean-k-mooney: yep, tell me about it, I know all about people wanting to propagate libvirt "fixes". | |
| 15:28:01 | sean-k-mooney | i think this would be somthing that the sriov nic agent could resonalby handel too | |
| 15:28:10 | sean-k-mooney | well maybe not | |
| 15:28:15 | sean-k-mooney | that could be racy | |
| 15:29:02 | sean-k-mooney | it would have to be nova/libvirt to avoid races with other vms booting | |
| 15:29:09 | adrianc | sean-k-mooney: there is no FLR involved during detach/attach, the driver unbinds from the guest, and binds on the host whether or not it triggers an FLR is something the driver decides AFAIK | |
| 15:29:39 | jangutter | adrianc: yep, FLR is mandatory on iommu passthrough I think, but not for driver bind/unbind | |
| 15:30:32 | sean-k-mooney | well it depend on if the vf is bound to say ixgbeVf when its detach or if it stays with vfio-pci | |
| 15:30:54 | jangutter | sean-k-mooney: yeah. | |
| 15:31:59 | adrianc | would have been easier if setting VF mac address via ip link on the VF idx would have synced the mac address on the corresponding VF netdev :) | |
| 15:32:00 | sean-k-mooney | in any case however in macvtap mode we aparent dont update the driver the vf is bound to or trigger a vf reset | |
| 15:32:33 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add python 3.7 unit and functional tox jobs https://review.openstack.org/624055 | |
| 15:32:33 | sean-k-mooney | adrianc: that silly talk why would a hardware/driver implementer ever do a thing like that :P | |
| 15:32:34 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Replace ThreadPoolExecutor with GreenThreadPoolExecutor https://review.openstack.org/624740 | |
| 15:32:39 | gibi | cdent: ^^ | |
| 15:32:48 | cdent | roger | |
| 15:33:16 | mriedem | slaweq: yeah | |
| 15:33:22 | jangutter | sean-k-mooney, adrianc: years ago we had looked at how to propagate MAC config from host to the VF and the conclusion we came to was that at the FLR is the safest way to do so. I don't think there was an async event able to propagate that info from the device to the driver without either a reset or rebind. | |
| 15:33:42 | slaweq | mriedem: thx for taking care of it :) | |
| 15:33:47 | sean-k-mooney | adrianc: speacking of ip link ther eis no way to reset it via ip tools right. | |
| 15:34:17 | jangutter | sean-k-mooney, adrianc: the only way I know to trigger an FLR is via setpci... | |
| 15:35:02 | sean-k-mooney | jangutter: i was hoping there would be a standard sysfs file we could echo too but i guess not | |
| 15:35:18 | aspiers | mriedem: if I understand the code right, the config drive disk mapping is built alongside the other disk mappings prior to instance launch (even though the config drive itself seems to be built late via post_xml_callback). so it's not hotplugged post-launch. Also I think it will either use "ide" or "virtio" bus (depending on whether iso9660 or vfat) and therefore we don't anticipate any problems with | |
| 15:35:19 | aspiers | SEV. Nevertheless we're testing to make sure | |
| 15:35:22 | jangutter | sean-k-mooney: there might be these days, though. | |
| 15:35:40 | sean-k-mooney | jangutter: it would still likely be vendor specific | |
| 15:36:15 | jangutter | sean-k-mooney: very, I believe our driver only recently could propagate MAC on rebind. Previously you _had_ to get an FLR. | |
| 15:36:20 | sean-k-mooney | adrianc: im just getting around to setting up that test env now but i will try a few different things and let you know | |
| 15:37:20 | sean-k-mooney | should i put this in the spec by the way. im inclined to say no as this is too technical/hardware specific | |
| 15:37:31 | adrianc | sean-k-mooney: ack, let me know if you can find a nicer WA | |
| 15:38:01 | adrianc | well i dont think its related to the spec, its a long existing problem IMO and not related to live-migration | |
| 15:38:19 | sean-k-mooney | we need to adress this in the code obviously but i think code commetns/dev docs | |
| 15:38:47 | sean-k-mooney | adrianc: i guess you could consider it a cold migration bug | |
| 15:38:48 | adrianc | there are additional gaps with macVtap like multiple port binding so | |
| 15:40:38 | adrianc | sean-k-mooney: it can happen also without LM being involved, easiest is to boot a VM on node A delete and boot it on node B | |
| 15:41:02 | adrianc | the mac will be reused and you hit the issue | |
| 15:41:14 | sean-k-mooney | adrianc: reusing the same neutorn port so that the mac and vlan remain the same | |
| 15:41:49 | adrianc | adrianc: yes, using the same neutron port | |
| 15:42:13 | adrianc | sean-k-mooney: ^ | |
| 15:42:22 | sean-k-mooney | ya ok that is proably ground enought to just treat this as a seperate bug and maybe backport the fix | |
| 15:43:53 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Add workaround to cleanup instance dir during evac with rbd https://review.openstack.org/618478 | |
| 15:45:41 | lyarwood | mriedem / mdbooth / melwitt ; ^ thanks again for the reviews, I appreciate just how ugly this is but hopefully this is btw but with downstream customers asking for this to be fixed back in Newton I'm being forced into a corner here tbh. | |
| 15:46:02 | lyarwood | thanks weechat-- | |
| 15:46:53 | sean-k-mooney | lyarwood: what wrong with weechat++ | |
| 15:47:53 | lyarwood | ^ the above mess, I was trying to edit the line and it decided to send it out, fun times. | |
| 16:02:27 | mriedem | melwitt: did you see my question in your spec for configurable max volumes? https://review.openstack.org/#/c/597306/1/specs/stein/approved/conf-max-attach-volumes.rst@44 | |
| 16:05:01 | melwitt | mriedem: I did, sorry. the immediate answer was "not 100% sure, need to check my notes again". it was "fun" tracing it all and I can't remember whether bfv vs attach additional end up using the same code path that leads to the limit of 26. I need to double check it in order to reply and haven't gotten to it yet | |
| 16:09:09 | mriedem | ok, but i'm assuming even if it's not the same code path, bfv would need a check somewhere for the config option to limit the number of volumes attached to an instance during spawn | |
| 16:10:21 | mriedem | it might have to be here https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1598 | |
| 16:10:34 | mriedem | or a new driver method called from _prep_block_devices | |