| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-04-19 | |||
| 23:45:08 | mriedem | UK | |
| 23:45:27 | mriedem | so he'd be around in the morning | |
| 23:46:05 | imacdonn | OK, I'll look for him them ... I can post what I did somewhere, but I'm fairly sure it's not "right" | |
| 23:46:27 | mriedem | hard to know if you don't post it | |
| 23:46:40 | mriedem | posting to gerrit would be preferrable, else you could attach a patch file in the bug report | |
| 23:47:44 | imacdonn | I think it's too much of a hack to put in a review .. should I attach to my bug (the one marked as a dup) ? | |
| 23:49:41 | mriedem | nothing is too much of a hack to put it in a review - just put WIP in the prefix of the commit title | |
| 23:49:47 | mriedem | or -Workflow it in gerrit | |
| 23:49:53 | mriedem | no one is going to think less of you :) | |
| 23:50:15 | imacdonn | heh | |
| 23:50:45 | imacdonn | I'll take another look at it first ... don't want to embarrass myself any more than necessary :) | |
| 23:52:49 | imacdonn | I have another thing I wanted to get your opinion on (completely unrelated) | |
| 23:55:36 | mriedem | ok | |
| 23:55:52 | imacdonn | so when nova plugs an instance's vif into a neutron network that has MTU != 1500, something needs to set the MTU on the tap interface. Whose job is that? nova-compute? os_vif? the neutron l2 agent? ... ? | |
| 23:56:02 | imacdonn | asking because it's currently not getting done, and I'm not sure who to blame. | |
| 23:56:18 | imacdonn | For more detail, see https://ask.openstack.org/en/question/105521/whos-supposed-to-set-the-mtu-on-a-nova-instances-vifs-tap-interface/ | |
| 23:56:52 | mriedem | sean-k-mooney: ^ | |
| 23:57:02 | mriedem | imacdonn: in part it depends on the vif type | |
| 23:57:15 | mriedem | the libvirt driver isn't using os-vif yet for all vif types | |
| 23:57:35 | imacdonn | it is in my case .. vif_type is bridge | |
| 23:58:08 | mriedem | then i believe os-vif does it | |
| 23:58:28 | imacdonn | I think I have the terminology right there ... in any case, vif_plug_linux_bridge under os_vif creates the bridge | |
| 23:58:52 | imacdonn | then libvirt creates the domain, which creates the tap interface, and puts it on the bridge | |
| 23:58:57 | mriedem | does the network in neutron have the mtu attribute set? | |
| 23:59:06 | imacdonn | yes | |
| 23:59:40 | imacdonn | I've studied the code, and I can't find anything that's supposed to do the job | |
| 23:59:56 | imacdonn | I don't think os_vif can do it, because the tap interface doesn't exist yet | |
| #openstack-nova - 2017-04-20 | |||
| 00:00:09 | imacdonn | it doesn't exist until libvirt creates the domain | |
| 00:02:50 | imacdonn | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L5079 | |
| 00:02:55 | mriedem | do you see Converting VIF in the nova-compute logs at debug level? | |
| 00:03:05 | imacdonn | note that plug_vifs() is called before _create_domain() | |
| 00:03:08 | mriedem | right that's where we start with os-vif | |
| 00:03:09 | mriedem | yeah | |
| 00:03:38 | mriedem | kevinbenton: are you around? | |
| 00:04:01 | kevinbenton | mriedem: AHOy! | |
| 00:04:16 | mriedem | kevinbenton: ^ whats set the MTU on the tap interface? | |
| 00:04:21 | mriedem | for vif_type == bridge? | |
| 00:05:04 | mriedem | i thought it was os-vif | |
| 00:05:15 | kevinbenton | mriedem: no | |
| 00:05:30 | kevinbenton | mriedem: it's either libvirt internals or it inherits from bridge as part of linux | |
| 00:05:33 | mriedem | or does neutron do it once we plug the vif and neutron agent picks that up? | |
| 00:05:43 | mriedem | hmm | |
| 00:06:15 | kevinbenton | mriedem: this is sort of why i wanted to move away from libvirt plugging things into the bridge | |
| 00:06:34 | imacdonn | inheriting from the bridge doesn't work, because the bridge is initially created with MTU 1500, and it won't go any higher until a interface is added .. then it assumes the lowest MTU amongst all member interfaces | |
| 00:06:48 | kevinbenton | imacdonn: hang on | |
| 00:06:51 | imacdonn | k | |
| 00:06:59 | kevinbenton | MTU is set on the bridge by os-vif | |
| 00:07:13 | mriedem | i thought it was this https://github.com/openstack/os-vif/blob/master/vif_plug_linux_bridge/linux_net.py#L91 | |
| 00:07:23 | imacdonn | it can't be | |
| 00:07:28 | imacdonn | or .. maybe I have a bug in the OS | |
| 00:08:09 | imacdonn | # brctl addbr test-bridge | |
| 00:08:17 | imacdonn | # ip l set test-bridge mtu 9000 | |
| 00:08:21 | imacdonn | RTNETLINK answers: Invalid argument | |
| 00:09:15 | kevinbenton | oh hang on | |
| 00:09:24 | kevinbenton | maybe not | |
| 00:09:49 | kevinbenton | i think that branch in os-vif is not exercised with neutron | |
| 00:10:20 | kevinbenton | because https://github.com/openstack/os-vif/blob/master/vif_plug_linux_bridge/linux_bridge.py#L98 is not true with neutron | |
| 00:10:59 | imacdonn | yeah, I only have should_provide_bridge | |
| 00:15:36 | mriedem | ok, well i guess that's why then | |
| 00:16:01 | kevinbenton | https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1399064 | |
| 00:16:01 | openstack | Launchpad bug 1399064 in linux (Ubuntu) "bridges cannot have a mtu > 1500 by themselves" [Medium,Triaged] | |
| 00:16:22 | kevinbenton | so i wonder if we have a race condition here | |
| 00:16:43 | kevinbenton | where nova adding a tap interface to the bridge results in 1500 mtu | |
| 00:16:51 | kevinbenton | since it's first | |
| 00:17:09 | kevinbenton | and then neutron adds vxlan interface | |
| 00:17:17 | kevinbenton | with real network mtu | |
| 00:18:03 | imacdonn | yeah, so then when I destroy nova instance, and the tap interface goes away, the bridge inherits the MTU of the VLAN interface, which is 9000 (in my case) | |
| 00:18:23 | imacdonn | then I launch another nova instance, and its tap interface picks up the bridge's MTU of 9000, and it works | |
| 00:18:33 | kevinbenton | ack | |
| 00:18:39 | kevinbenton | that sounds exactly like the case then | |
| 00:18:56 | imacdonn | it's not really a "race condition" in the multi-threading sense | |
| 00:19:31 | kevinbenton | well it is in the sense that the neutron agent could conceivably beat nova | |
| 00:19:40 | kevinbenton | after libvirt creates bridge | |
| 00:19:43 | imacdonn | even if it was possible to set the MTU on an empty bridge, there's still nothign in the code that will set the bridge's MTU | |
| 00:19:45 | kevinbenton | but before it adds interface | |
| 00:19:58 | kevinbenton | neutron could wire up the vlan/vxlan interface | |
| 00:20:20 | imacdonn | oh, you;re saying that if neutron added the VLAN interface before libvirt created the domain, it'd work ? | |
| 00:20:36 | kevinbenton | right | |
| 00:20:45 | imacdonn | that can't happen, because the neutron l2 agent triggers on the tap interface's existence | |
| 00:21:10 | kevinbenton | oh duh | |
| 00:21:12 | imacdonn | well, I guess it could happen if there's a window where the tap interface exists but hasn't been added the bridge yet | |
| 00:21:20 | kevinbenton | yeah | |
| 00:21:33 | kevinbenton | but the reason we don't see this in the gate is probably because of the dhcp agents, etc | |
| 00:21:49 | imacdonn | BTW, this was solved in the neutron agent in the past | |
| 00:22:01 | imacdonn | but then it got un-solved by https://git.openstack.org/cgit/openstack/neutron/commit/?id=d352661c56d5f03713e615b7e0c2c9c8688e0132 | |
| 00:22:31 | kevinbenton | how would that solve it? | |
| 00:22:33 | imacdonn | the previous solution does seem a little kludgey | |
| 00:22:41 | kevinbenton | bridge MTU would also be too small | |
| 00:22:51 | imacdonn | it looks at the the MTU of the VLAN interface and forces it onto the tap interface | |
| 00:23:54 | kevinbenton | ah, i see | |
| 00:24:01 | kevinbenton | it's ignoring bridge mtu completely | |
| 00:24:23 | imacdonn | yeah, the bridge just inherits the lowest amongst all its members | |
| 00:25:18 | kevinbenton | yes, so this was result in packet loss every time a vm connected | |
| 00:26:06 | imacdonn | I wonder if libvirt provides any way to specify the MTU for a bridge interface | |
| 00:26:43 | imacdonn | seems that that's probably where it should be done ? | |
| 00:26:49 | clarkb | this is only a problem if trying to use larger MTUs right? | |
| 00:26:55 | clarkb | thats why we wouldnt see it in the gate | |
| 00:27:04 | imacdonn | yeah | |
| 00:27:10 | kevinbenton | well we also wouldn't see it in single node gate | |
| 00:27:20 | kevinbenton | because dhcp agent would get attached before VM | |