| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-04-20 | |||
| 00:09:24 | kevinbenton | maybe not | |
| 00:09:49 | kevinbenton | i think that branch in os-vif is not exercised with neutron | |
| 00:10:20 | kevinbenton | because https://github.com/openstack/os-vif/blob/master/vif_plug_linux_bridge/linux_bridge.py#L98 is not true with neutron | |
| 00:10:59 | imacdonn | yeah, I only have should_provide_bridge | |
| 00:15:36 | mriedem | ok, well i guess that's why then | |
| 00:16:01 | openstack | Launchpad bug 1399064 in linux (Ubuntu) "bridges cannot have a mtu > 1500 by themselves" [Medium,Triaged] | |
| 00:16:01 | kevinbenton | https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1399064 | |
| 00:16:22 | kevinbenton | so i wonder if we have a race condition here | |
| 00:16:43 | kevinbenton | where nova adding a tap interface to the bridge results in 1500 mtu | |
| 00:16:51 | kevinbenton | since it's first | |
| 00:17:09 | kevinbenton | and then neutron adds vxlan interface | |
| 00:17:17 | kevinbenton | with real network mtu | |
| 00:18:03 | imacdonn | yeah, so then when I destroy nova instance, and the tap interface goes away, the bridge inherits the MTU of the VLAN interface, which is 9000 (in my case) | |
| 00:18:23 | imacdonn | then I launch another nova instance, and its tap interface picks up the bridge's MTU of 9000, and it works | |
| 00:18:33 | kevinbenton | ack | |
| 00:18:39 | kevinbenton | that sounds exactly like the case then | |
| 00:18:56 | imacdonn | it's not really a "race condition" in the multi-threading sense | |
| 00:19:31 | kevinbenton | well it is in the sense that the neutron agent could conceivably beat nova | |
| 00:19:40 | kevinbenton | after libvirt creates bridge | |
| 00:19:43 | imacdonn | even if it was possible to set the MTU on an empty bridge, there's still nothign in the code that will set the bridge's MTU | |
| 00:19:45 | kevinbenton | but before it adds interface | |
| 00:19:58 | kevinbenton | neutron could wire up the vlan/vxlan interface | |
| 00:20:20 | imacdonn | oh, you;re saying that if neutron added the VLAN interface before libvirt created the domain, it'd work ? | |
| 00:20:36 | kevinbenton | right | |
| 00:20:45 | imacdonn | that can't happen, because the neutron l2 agent triggers on the tap interface's existence | |
| 00:21:10 | kevinbenton | oh duh | |
| 00:21:12 | imacdonn | well, I guess it could happen if there's a window where the tap interface exists but hasn't been added the bridge yet | |
| 00:21:20 | kevinbenton | yeah | |
| 00:21:33 | kevinbenton | but the reason we don't see this in the gate is probably because of the dhcp agents, etc | |
| 00:21:49 | imacdonn | BTW, this was solved in the neutron agent in the past | |
| 00:22:01 | imacdonn | but then it got un-solved by https://git.openstack.org/cgit/openstack/neutron/commit/?id=d352661c56d5f03713e615b7e0c2c9c8688e0132 | |
| 00:22:31 | kevinbenton | how would that solve it? | |
| 00:22:33 | imacdonn | the previous solution does seem a little kludgey | |
| 00:22:41 | kevinbenton | bridge MTU would also be too small | |
| 00:22:51 | imacdonn | it looks at the the MTU of the VLAN interface and forces it onto the tap interface | |
| 00:23:54 | kevinbenton | ah, i see | |
| 00:24:01 | kevinbenton | it's ignoring bridge mtu completely | |
| 00:24:23 | imacdonn | yeah, the bridge just inherits the lowest amongst all its members | |
| 00:25:18 | kevinbenton | yes, so this was result in packet loss every time a vm connected | |
| 00:26:06 | imacdonn | I wonder if libvirt provides any way to specify the MTU for a bridge interface | |
| 00:26:43 | imacdonn | seems that that's probably where it should be done ? | |
| 00:26:49 | clarkb | this is only a problem if trying to use larger MTUs right? | |
| 00:26:55 | clarkb | thats why we wouldnt see it in the gate | |
| 00:27:04 | imacdonn | yeah | |
| 00:27:10 | kevinbenton | well we also wouldn't see it in single node gate | |
| 00:27:20 | kevinbenton | because dhcp agent would get attached before VM | |
| 00:27:29 | kevinbenton | and DHCP agent plugs using correct MTU | |
| 00:27:48 | clarkb | kevinbenton: right but until we test with jumbo frames we would never see it regardless of multinode or single node | |
| 00:27:51 | clarkb | (and we have multinode testing) | |
| 00:27:58 | imacdonn | FWIW, the dhcp_agent's tap interface does get set to that of the network .. but that's obviously nothing to do with nova | |
| 00:28:45 | kevinbenton | so yeah, there are two possible fixes i can see | |
| 00:28:53 | kevinbenton | tell libvirt about MTU if it supports that | |
| 00:29:04 | imacdonn | yeah, if | |
| 00:29:10 | kevinbenton | or proceed with https://review.openstack.org/#/c/447150/ | |
| 00:29:25 | kevinbenton | which will stop libvirt from plugging stuff into linux bridges | |
| 00:29:35 | kevinbenton | and then re-add something like https://git.openstack.org/cgit/openstack/neutron/commit/?id=d352661c56d5f03713e615b7e0c2c9c8688e0132 | |
| 00:29:45 | imacdonn | hmm ineresting | |
| 00:29:46 | kevinbenton | to set MTU before it gets added to the bridge | |
| 00:29:48 | imacdonn | interesting* | |
| 00:29:52 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Avoid forcing translation on logging calls https://review.openstack.org/413876 | |
| 00:31:55 | imacdonn | interesting hackaround ... http://linuxaleph.blogspot.com/2013/01/how-to-network-jumbo-frames-to-kvm-guest.html | |
| 00:33:27 | kevinbenton | imacdonn: can you check if libvirt can take MTU as an option? | |
| 00:33:35 | imacdonn | Googling on that now | |
| 00:34:02 | kevinbenton | because the problem with my patch is that it requires a permissions change due to qemu sillyness | |
| 00:34:20 | kevinbenton | it requires more permissions to not create a bridge than to create one :) | |
| 00:34:26 | imacdonn | heh | |
| 00:34:49 | clarkb | kevinbenton: well qemu-bridge-helper is a thing, maybe you just make that nop? | |
| 00:34:57 | clarkb | (thought that might confuse people greatly) | |
| 00:35:48 | kevinbenton | clarkb: yeah, that also makes detection harder because i wouldn't be using a different vif_type on the neutron side | |
| 00:36:18 | kevinbenton | it would be hard to distinguish in the agent if a given nova port was going to try to plug itself into a bridge | |
| 00:36:38 | clarkb | gotcha | |
| 00:37:04 | clarkb | (you only need to attach to the bridge if on a network connected to something?) | |
| 00:37:27 | imacdonn | https://libvirt.org/formatnetwork.html <- trying to comprehend this .. I think it's saying that it only works for libvirt-managed networks | |
| 00:38:10 | imacdonn | "Network types not specifically mentioned here don't support having an MTU set in the libvirt network config." | |
| 00:38:58 | kevinbenton | clarkb: we attach ports to bridges that were created by agents. we leave attaching to bridge up to nova for compute ports | |
| 00:39:22 | kevinbenton | clarkb: so we skip based on device_owner 'compute' right now | |
| 00:39:42 | kevinbenton | clarkb: ideally i want to get rid of that, but for one cycle i will need to know about old compute ports trying to plug themselves in | |
| 00:40:32 | kevinbenton | clarkb: so with my current approach we just set the vif_type='tap' for any newly bound linux bridge ports | |
| 00:40:48 | kevinbenton | clarkb: and that will make it easy to tell | |
| 00:41:38 | clarkb | and is there a reason you can't set the mtu on the bridge when it is created (because you don't know what the mtu of the devices that attach to it later will be?) | |
| 00:42:08 | kevinbenton | clarkb: we could | |
| 00:42:11 | imacdonn | linux doesn't seem to allow that .. at least the kernel I'm running | |
| 00:42:13 | kevinbenton | clarkb: but that leads to the other bug | |
| 00:42:28 | openstack | Launchpad bug 1399064 in linux (Ubuntu) "bridges cannot have a mtu > 1500 by themselves" [Medium,Triaged] | |
| 00:42:28 | kevinbenton | clarkb: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1399064 | |
| 00:42:35 | clarkb | ah | |
| 00:42:50 | clarkb | we set them lower in the multinode tests when we used linux bridges and not ovs | |
| 00:42:53 | clarkb | good to know thanks | |
| 00:45:20 | kevinbenton | imacdonn: yeah, that doesn't sound promising for libvirt | |
| 00:45:28 | imacdonn | ack | |
| 00:46:16 | kevinbenton | imacdonn: i suppose we can put the logic back in the neutron agent | |
| 00:46:30 | kevinbenton | the tap device inherits from the bridge, right? | |
| 00:46:44 | imacdonn | it appears so, yes | |
| 00:47:01 | imacdonn | well... | |
| 00:47:08 | kevinbenton | so once the bridge is embiggened, we don't reduce by adding a tap device | |
| 00:47:12 | kevinbenton | :) | |
| 00:47:12 | imacdonn | it picks up the bridge's MTU when it gets added to the bridge | |
| 00:47:21 | imacdonn | if you change the bridge later, I don't think the tap will follow | |
| 00:47:27 | kevinbenton | yeah, that's fine | |
| 00:47:37 | kevinbenton | i'm just trying to make sure the bridge wouldn't shrink every time | |