| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-04-20 | |||
| 00:03:05 | imacdonn | note that plug_vifs() is called before _create_domain() | |
| 00:03:08 | mriedem | right that's where we start with os-vif | |
| 00:03:09 | mriedem | yeah | |
| 00:03:38 | mriedem | kevinbenton: are you around? | |
| 00:04:01 | kevinbenton | mriedem: AHOy! | |
| 00:04:16 | mriedem | kevinbenton: ^ whats set the MTU on the tap interface? | |
| 00:04:21 | mriedem | for vif_type == bridge? | |
| 00:05:04 | mriedem | i thought it was os-vif | |
| 00:05:15 | kevinbenton | mriedem: no | |
| 00:05:30 | kevinbenton | mriedem: it's either libvirt internals or it inherits from bridge as part of linux | |
| 00:05:33 | mriedem | or does neutron do it once we plug the vif and neutron agent picks that up? | |
| 00:05:43 | mriedem | hmm | |
| 00:06:15 | kevinbenton | mriedem: this is sort of why i wanted to move away from libvirt plugging things into the bridge | |
| 00:06:34 | imacdonn | inheriting from the bridge doesn't work, because the bridge is initially created with MTU 1500, and it won't go any higher until a interface is added .. then it assumes the lowest MTU amongst all member interfaces | |
| 00:06:48 | kevinbenton | imacdonn: hang on | |
| 00:06:51 | imacdonn | k | |
| 00:06:59 | kevinbenton | MTU is set on the bridge by os-vif | |
| 00:07:13 | mriedem | i thought it was this https://github.com/openstack/os-vif/blob/master/vif_plug_linux_bridge/linux_net.py#L91 | |
| 00:07:23 | imacdonn | it can't be | |
| 00:07:28 | imacdonn | or .. maybe I have a bug in the OS | |
| 00:08:09 | imacdonn | # brctl addbr test-bridge | |
| 00:08:17 | imacdonn | # ip l set test-bridge mtu 9000 | |
| 00:08:21 | imacdonn | RTNETLINK answers: Invalid argument | |
| 00:09:15 | kevinbenton | oh hang on | |
| 00:09:24 | kevinbenton | maybe not | |
| 00:09:49 | kevinbenton | i think that branch in os-vif is not exercised with neutron | |
| 00:10:20 | kevinbenton | because https://github.com/openstack/os-vif/blob/master/vif_plug_linux_bridge/linux_bridge.py#L98 is not true with neutron | |
| 00:10:59 | imacdonn | yeah, I only have should_provide_bridge | |
| 00:15:36 | mriedem | ok, well i guess that's why then | |
| 00:16:01 | openstack | Launchpad bug 1399064 in linux (Ubuntu) "bridges cannot have a mtu > 1500 by themselves" [Medium,Triaged] | |
| 00:16:01 | kevinbenton | https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1399064 | |
| 00:16:22 | kevinbenton | so i wonder if we have a race condition here | |
| 00:16:43 | kevinbenton | where nova adding a tap interface to the bridge results in 1500 mtu | |
| 00:16:51 | kevinbenton | since it's first | |
| 00:17:09 | kevinbenton | and then neutron adds vxlan interface | |
| 00:17:17 | kevinbenton | with real network mtu | |
| 00:18:03 | imacdonn | yeah, so then when I destroy nova instance, and the tap interface goes away, the bridge inherits the MTU of the VLAN interface, which is 9000 (in my case) | |
| 00:18:23 | imacdonn | then I launch another nova instance, and its tap interface picks up the bridge's MTU of 9000, and it works | |
| 00:18:33 | kevinbenton | ack | |
| 00:18:39 | kevinbenton | that sounds exactly like the case then | |
| 00:18:56 | imacdonn | it's not really a "race condition" in the multi-threading sense | |
| 00:19:31 | kevinbenton | well it is in the sense that the neutron agent could conceivably beat nova | |
| 00:19:40 | kevinbenton | after libvirt creates bridge | |
| 00:19:43 | imacdonn | even if it was possible to set the MTU on an empty bridge, there's still nothign in the code that will set the bridge's MTU | |
| 00:19:45 | kevinbenton | but before it adds interface | |
| 00:19:58 | kevinbenton | neutron could wire up the vlan/vxlan interface | |
| 00:20:20 | imacdonn | oh, you;re saying that if neutron added the VLAN interface before libvirt created the domain, it'd work ? | |
| 00:20:36 | kevinbenton | right | |
| 00:20:45 | imacdonn | that can't happen, because the neutron l2 agent triggers on the tap interface's existence | |
| 00:21:10 | kevinbenton | oh duh | |
| 00:21:12 | imacdonn | well, I guess it could happen if there's a window where the tap interface exists but hasn't been added the bridge yet | |
| 00:21:20 | kevinbenton | yeah | |
| 00:21:33 | kevinbenton | but the reason we don't see this in the gate is probably because of the dhcp agents, etc | |
| 00:21:49 | imacdonn | BTW, this was solved in the neutron agent in the past | |
| 00:22:01 | imacdonn | but then it got un-solved by https://git.openstack.org/cgit/openstack/neutron/commit/?id=d352661c56d5f03713e615b7e0c2c9c8688e0132 | |
| 00:22:31 | kevinbenton | how would that solve it? | |
| 00:22:33 | imacdonn | the previous solution does seem a little kludgey | |
| 00:22:41 | kevinbenton | bridge MTU would also be too small | |
| 00:22:51 | imacdonn | it looks at the the MTU of the VLAN interface and forces it onto the tap interface | |
| 00:23:54 | kevinbenton | ah, i see | |
| 00:24:01 | kevinbenton | it's ignoring bridge mtu completely | |
| 00:24:23 | imacdonn | yeah, the bridge just inherits the lowest amongst all its members | |
| 00:25:18 | kevinbenton | yes, so this was result in packet loss every time a vm connected | |
| 00:26:06 | imacdonn | I wonder if libvirt provides any way to specify the MTU for a bridge interface | |
| 00:26:43 | imacdonn | seems that that's probably where it should be done ? | |
| 00:26:49 | clarkb | this is only a problem if trying to use larger MTUs right? | |
| 00:26:55 | clarkb | thats why we wouldnt see it in the gate | |
| 00:27:04 | imacdonn | yeah | |
| 00:27:10 | kevinbenton | well we also wouldn't see it in single node gate | |
| 00:27:20 | kevinbenton | because dhcp agent would get attached before VM | |
| 00:27:29 | kevinbenton | and DHCP agent plugs using correct MTU | |
| 00:27:48 | clarkb | kevinbenton: right but until we test with jumbo frames we would never see it regardless of multinode or single node | |
| 00:27:51 | clarkb | (and we have multinode testing) | |
| 00:27:58 | imacdonn | FWIW, the dhcp_agent's tap interface does get set to that of the network .. but that's obviously nothing to do with nova | |
| 00:28:45 | kevinbenton | so yeah, there are two possible fixes i can see | |
| 00:28:53 | kevinbenton | tell libvirt about MTU if it supports that | |
| 00:29:04 | imacdonn | yeah, if | |
| 00:29:10 | kevinbenton | or proceed with https://review.openstack.org/#/c/447150/ | |
| 00:29:25 | kevinbenton | which will stop libvirt from plugging stuff into linux bridges | |
| 00:29:35 | kevinbenton | and then re-add something like https://git.openstack.org/cgit/openstack/neutron/commit/?id=d352661c56d5f03713e615b7e0c2c9c8688e0132 | |
| 00:29:45 | imacdonn | hmm ineresting | |
| 00:29:46 | kevinbenton | to set MTU before it gets added to the bridge | |
| 00:29:48 | imacdonn | interesting* | |
| 00:29:52 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Avoid forcing translation on logging calls https://review.openstack.org/413876 | |
| 00:31:55 | imacdonn | interesting hackaround ... http://linuxaleph.blogspot.com/2013/01/how-to-network-jumbo-frames-to-kvm-guest.html | |
| 00:33:27 | kevinbenton | imacdonn: can you check if libvirt can take MTU as an option? | |
| 00:33:35 | imacdonn | Googling on that now | |
| 00:34:02 | kevinbenton | because the problem with my patch is that it requires a permissions change due to qemu sillyness | |
| 00:34:20 | kevinbenton | it requires more permissions to not create a bridge than to create one :) | |
| 00:34:26 | imacdonn | heh | |
| 00:34:49 | clarkb | kevinbenton: well qemu-bridge-helper is a thing, maybe you just make that nop? | |
| 00:34:57 | clarkb | (thought that might confuse people greatly) | |
| 00:35:48 | kevinbenton | clarkb: yeah, that also makes detection harder because i wouldn't be using a different vif_type on the neutron side | |
| 00:36:18 | kevinbenton | it would be hard to distinguish in the agent if a given nova port was going to try to plug itself into a bridge | |
| 00:36:38 | clarkb | gotcha | |
| 00:37:04 | clarkb | (you only need to attach to the bridge if on a network connected to something?) | |
| 00:37:27 | imacdonn | https://libvirt.org/formatnetwork.html <- trying to comprehend this .. I think it's saying that it only works for libvirt-managed networks | |
| 00:38:10 | imacdonn | "Network types not specifically mentioned here don't support having an MTU set in the libvirt network config." | |
| 00:38:58 | kevinbenton | clarkb: we attach ports to bridges that were created by agents. we leave attaching to bridge up to nova for compute ports | |
| 00:39:22 | kevinbenton | clarkb: so we skip based on device_owner 'compute' right now | |