| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-14 | |||
| 18:43:44 | mnaser | so afaik odl just 'notices' ports appearing, and in neutron-server there is a web socket connection to odl which notifies it that a port is appeared, so it sets it to active | |
| 18:44:01 | melwitt | then we added another patch to only not wait if linuxbridge because we thought ovs was detecting the unplug/plug | |
| 18:44:16 | mnaser | and so what happens is that, after the hard reboot, the port is always still plugged and didnt get a state change, so nothing happens in neutron and no notification goes out | |
| 18:44:28 | jaypipes | mnaser: wow. I had no idea there were hard-coded hooks in the neutron base db plugin thing. :( | |
| 18:44:32 | mnaser | well, the unplug in os_vif when using ovs literally does nothing | |
| 18:44:35 | mnaser | jaypipes: yeah, TIL yesterday | |
| 18:44:40 | mnaser | or YIL i guess | |
| 18:44:48 | mnaser | i thought it was a bit more complex than that | |
| 18:44:50 | jaypipes | heh | |
| 18:45:31 | melwitt | yeah, but the plug? we asked people in #openstack-neutron and the consensus at the time was that the ovs agent would detect things and send events | |
| 18:45:37 | mnaser | now question is does anyone know why os_vif_ovs does nothing when nova uses the 'unplug' method in it? win32 seems to remove the port from the bridge | |
| 18:45:42 | mriedem | melwitt: this isn't 'ovs' this, | |
| 18:45:44 | mriedem | it's opendaylight | |
| 18:45:59 | melwitt | okay, so things are working fine for ovs then | |
| 18:46:03 | mnaser | correct | |
| 18:46:04 | mriedem | apparently same vif type, | |
| 18:46:09 | mriedem | different backend implementation | |
| 18:46:09 | mnaser | ^ and that too | |
| 18:46:17 | mriedem | does the vif type == networking API or something? | |
| 18:46:23 | mriedem | and odl implements the ovs api? | |
| 18:46:25 | melwitt | sorry, I got confused by the earlier mention of adding VIF_TYPE_OVS to the blacklist? | |
| 18:46:37 | melwitt | is that because opendaylight falls under that vif type? | |
| 18:46:41 | mriedem | melwitt: yes | |
| 18:46:41 | mnaser | correct | |
| 18:46:48 | jaypipes | mnaser: I'll be honest. I don't know why for OVS we don't do anything for OVS when !Windows. | |
| 18:46:56 | melwitt | okay, gotcha now | |
| 18:47:13 | mriedem | if we had to add ovs to that blacklist, then there is really no point in even waiting for vif plug events during a hard reboot with the libvirt driver | |
| 18:47:20 | mnaser | pretty much :\ | |
| 18:47:21 | melwitt | right | |
| 18:47:30 | mnaser | because i think if os_vif_ovs did an actual unplug, the state in neutron would be updated, and on the actual plug later, the state would be updated again and a notification goes out | |
| 18:47:40 | jaypipes | mnaser: I'm going to git blame it... one sec | |
| 18:47:41 | mnaser | i'm *assuming* neutron-openvswitch-agent magically realizes things have been unplugged and removes them | |
| 18:48:12 | mnaser | jaypipes: i tried to git blame for a while and didn't get anywhere productive but my git-fu isn't strong | |
| 18:49:19 | jaypipes | mnaser: it was Rawlin Peters who added this code. | |
| 18:49:22 | mnaser | again it could be something odl should be handling, does nova contact neutron for the unplug at the api layer? | |
| 18:49:26 | jaypipes | mnaser: I don't know Rawlin. | |
| 18:49:37 | mnaser | :< | |
| 18:49:42 | jaypipes | https://github.com/openstack/os-vif/commit/d5b119ba37d5c724d1279cb2292dce05c32b51a0 | |
| 18:49:54 | mriedem | mnaser: when we were talking about the differences between openvswith and linuxbridge agents here, sean-k-mooney was saying that ovs will send a notification which the neutron agent proxies to the server which is why ovs works and we don't miss plug events, but LB does polling and there could be a window where we miss it and then timeout waiting for an event that neutron never sends | |
| 18:49:58 | jaypipes | mnaser: https://review.openstack.org/#/c/333486/ | |
| 18:50:10 | mriedem | mnaser: no it's not a REST API call | |
| 18:50:13 | jaypipes | perhaps we should ask danpb and sean-k-mooney since they approved this patch. | |
| 18:50:20 | mriedem | it's a change on the host which the neutron agent routes up to neutron server | |
| 18:50:38 | sean-k-mooney | i saw my name | |
| 18:50:41 | sean-k-mooney | reading | |
| 18:50:47 | mnaser | ok i see, so given that neutron agent doesnt exist on the host, it would never end up hitting neutron | |
| 18:50:55 | melwitt | mnaser: yeah, we only do a unplug and plug through os-vif, no calls to neutron (unless os-vif does any, which I don't think it does) | |
| 18:51:04 | mriedem | melwitt: it doesn't | |
| 18:51:09 | jaypipes | melwitt: it does not, no. | |
| 18:51:12 | mriedem | it's like os-brick, just all host level stuff | |
| 18:51:14 | jaypipes | jinx | |
| 18:51:19 | melwitt | cool | |
| 18:51:27 | jaypipes | mriedem: it's like os-brick, only it bricked the host. | |
| 18:51:32 | mnaser | so maybe if we actually unplugged in os-vif, then it probably wont "hurt" the ovs-agent and things like odl would notice that the port is gone | |
| 18:51:48 | mnaser | and then update neutron in their own little mechanisms | |
| 18:51:54 | sean-k-mooney | oh yes the neutron l2 agent subscribes to notification from the ovs db instead of polling so it gets notified when we do plug and unplug on a hard reboot | |
| 18:52:15 | mnaser | sean-k-mooney: but os_vif unplug with ovs does nothing | |
| 18:52:35 | mnaser | https://github.com/openstack/os-vif/blob/master/vif_plug_ovs/ovs.py#L266-L268 | |
| 18:52:44 | mnaser | so im not sure how n-ovs-agent even knows something changed | |
| 18:53:00 | mriedem | jaypipes: rimshot'ed | |
| 18:53:04 | sean-k-mooney | mnaser: yes libvirt does the port plug on the ovs bridge in that case | |
| 18:53:08 | jaypipes | mriedem: :) | |
| 18:53:21 | mnaser | sean-k-mooney: ok, so pretty much it doesn't know things were unplugged, but it knows when they're plugged | |
| 18:53:22 | melwitt | how come odl doesn't have its own plugin for os-vif? | |
| 18:53:33 | sean-k-mooney | odl is not a network backend | |
| 18:53:42 | sean-k-mooney | odl mananges ovs | |
| 18:53:43 | melwitt | oh | |
| 18:53:43 | jaypipes | it's an SDN controller. | |
| 18:53:55 | sean-k-mooney | same for ovn | |
| 18:54:43 | melwitt | so odl manages ovs, but in the setup there are no ovs agents running on compute hosts | |
| 18:54:48 | mnaser | melwitt: correct | |
| 18:55:14 | mriedem | it's possible there is something in the port binding profile that tells us it's odl and not native ovs, but then again if we had to check that level of detail in the driver logic, it's going to suck | |
| 18:55:21 | mriedem | given all of the different neutron backends | |
| 18:55:23 | sean-k-mooney | neutron is repsocible for managing l2 and up nova/os-vif handel l1 netowrking for instances | |
| 18:55:50 | melwitt | yeah, I didn't realize the ovs agent was optional if the backend is ovs | |
| 18:56:10 | mnaser | backend is odl* :) | |
| 18:56:18 | mriedem | i didn't realize you could have different backends for the same vif type either | |
| 18:56:38 | sean-k-mooney | melwitt: well the reference implentation use the agent but, ovn,onos and odl can all be used instead of the agents | |
| 18:56:41 | melwitt | sean-k-mooney said odl isn't a network backend. sorry, I lack knowledge on these | |
| 18:56:56 | melwitt | okay, so they replace the agents. thanks | |
| 18:57:02 | sean-k-mooney | melwitt: for a neutron vif_type perspctive its nots | |
| 18:58:37 | sean-k-mooney | by the way i think there might be a way to get rid of the polling in the linux bridge agent if you use pyinotify but i have not run the idea past any of the neutron folks | |
| 18:58:40 | mriedem | actually there isn't really anything in the port information that would help us distinguish odl vifs http://logs.openstack.org/22/552922/1/check/networking-odl-tempest-oxygen/277adb4/controller/logs/screen-n-cpu.txt.gz#_Mar_14_15_53_40_140157 | |
| 18:58:58 | sean-k-mooney | mriedem: correct infact that is by design | |
| 18:59:18 | sean-k-mooney | mriedem: if there was the tenant could also tell | |
| 18:59:49 | mriedem | welp, we should probably not wait for vif plugged events on hard reboot in the libvirt driver then | |
| 18:59:56 | mriedem | if we can't determine accurately when that will work | |
| 19:00:11 | mriedem | which means reverting all of these https://review.openstack.org/#/q/Ib08afad3822f2ca95cfeea18d7f4fc4cb407b4d6 | |
| 19:00:17 | jaypipes | yeah | |
| 19:00:18 | melwitt | urgh | |
| 19:00:28 | mriedem | dansmith: ^ fyi | |
| 19:00:36 | sean-k-mooney | i missed the start of this convo. are ye discussing the fact that odl says it is finished wiring up the port imediatly when you bind the port | |
| 19:00:42 | dansmith | yeah I'm following | |
| 19:00:48 | dansmith | that does indeed suck | |
| 19:00:51 | mriedem | sean-k-mooney: odl doesn't send the vif plugged event | |
| 19:01:00 | mnaser | mriedem: it does send it, but only when the port is first created | |
| 19:01:02 | mriedem | but is the ovs vif type | |
| 19:01:07 | mriedem | mnaser: ok | |
| 19:01:19 | mnaser | mriedem: the port is never ever unplugged from the openvswitch port on shutdown (when nova calls unplug()) | |
| 19:01:23 | sean-k-mooney | mriedem: it send it when the port is bound not when the port is wired up | |