| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-14 | |||
| 18:25:11 | mnaser | this was noticed when the tempest start_stop tests failed because the startup would time out | |
| 18:26:02 | jaypipes | hm | |
| 18:26:03 | mnaser | http://logs.openstack.org/22/552922/1/check/networking-odl-tempest-oxygen/277adb4/testr_results.html.gz (you can look at nova logs there, but pretty much test_stop_start_server / test_reboot_server_hard / etc are the ones that arent working now) | |
| 18:26:43 | mnaser | and failure is .. "Details: (ServerActionsTestJSON:test_stop_start_server) Server da9cead9-c217-495e-97a8-65a6adacf37c failed to reach ACTIVE status and task state "None" within the required time (196 s). Current status: SHUTOFF. Current task state: powering-on." .. nova logs shows it timing out after 5 minutes | |
| 18:28:07 | jaypipes | mnaser: k. just a minute. reading through these patches... | |
| 18:28:13 | mnaser | sure thing | |
| 18:35:31 | jaypipes | mnaser: ok, done. so are you suggesting the fix here is to add VIF_TYPE_OVS to line 5392 here? https://review.openstack.org/#/c/541442/6/nova/virt/libvirt/driver.py | |
| 18:36:18 | mnaser | jaypipes: that would be a fix, or os_vif *actually* unplugging things could be a fix too | |
| 18:36:27 | mnaser | i do feel if that list starts growing, it might start become confusing for users :< | |
| 18:36:43 | jaypipes | mnaser: agreed. | |
| 18:37:12 | jaypipes | mnaser: for os-vif linux bridge, though, I'm not entirely sure what should be done on unplug... | |
| 18:37:50 | mnaser | jaypipes: yeah.. that's beyond me, but for openvswitch, im not sure if there is a link state, or maybe just deleting the actual port if its 'unplugged' | |
| 18:38:05 | mnaser | im sure there's a really good reason why it's not being deleted though, but i don't know why | |
| 18:38:37 | jaypipes | mnaser: I'm also not sure whether that would *ensure* that a vif-unplugged *event* from Neutron would be received... | |
| 18:38:43 | jaypipes | sean-k-mooney: you still around? | |
| 18:39:21 | mnaser | jaypipes: well, nova doesnt care about vif-unplugged afaik, issue is surrounding vif-plugged event | |
| 18:39:50 | mnaser | jaypipes: based on my reading, the nova notifications are a hook to the db model in neutron when an object changes. so if it goes from active => active, nothing is being updated and it never sent anything | |
| 18:40:03 | mnaser | now, maybe neutron-openvswitch-agent does things differently to trigger a change, i'm not sure. | |
| 18:40:03 | mriedem | lyarwood: melwitt: probably want to listen to ^ | |
| 18:40:34 | jaypipes | mnaser: not the neutron DB. the ovsdb... | |
| 18:40:35 | mriedem | related https://review.openstack.org/#/c/550046/ | |
| 18:40:41 | jaypipes | mnaser: and that's only for OVS of course. | |
| 18:40:47 | melwitt | mriedem: thanks | |
| 18:41:52 | mriedem | adding VIF_TYPE_OVS to the blacklist would defeat the purpose of the change, which is apparently working for ovs | |
| 18:41:57 | mriedem | we use ovs in our normal gate jobs | |
| 18:42:16 | mnaser | https://github.com/openstack/neutron/blob/master/neutron/db/db_base_plugin_v2.py#L155-L169 | |
| 18:42:17 | mriedem | building a giant list of which vifs types we should not wait on is going to be shitty | |
| 18:42:29 | melwitt | yeah, we were under the impression that ovs *does* send events for the plug in the hard reboot case | |
| 18:42:37 | melwitt | and that it was only linuxbridge that doesn't | |
| 18:42:46 | mnaser | right, but for context, i'm talking about the case when opendaylight is used here | |
| 18:42:54 | mriedem | does that use vif type ovs? | |
| 18:42:56 | jaypipes | mriedem: you mean like this? https://github.com/openstack/os-vif/blob/master/vif_plug_ovs/ovs.py#L266-L268 | |
| 18:42:57 | mnaser | yes | |
| 18:42:59 | mriedem | guh | |
| 18:43:14 | jaypipes | mriedem: welcome to our own private Idahell. | |
| 18:43:15 | mriedem | jaypipes: heh shitty indeed | |
| 18:43:21 | mriedem | can i be keanu? | |
| 18:43:23 | melwitt | originally, the patch never waited for any events because we thought during a hard reboot, the neutron agent would never detect us doing a os-vif unplug/plug | |
| 18:43:24 | jaypipes | no. | |
| 18:43:44 | mnaser | so afaik odl just 'notices' ports appearing, and in neutron-server there is a web socket connection to odl which notifies it that a port is appeared, so it sets it to active | |
| 18:44:01 | melwitt | then we added another patch to only not wait if linuxbridge because we thought ovs was detecting the unplug/plug | |
| 18:44:16 | mnaser | and so what happens is that, after the hard reboot, the port is always still plugged and didnt get a state change, so nothing happens in neutron and no notification goes out | |
| 18:44:28 | jaypipes | mnaser: wow. I had no idea there were hard-coded hooks in the neutron base db plugin thing. :( | |
| 18:44:32 | mnaser | well, the unplug in os_vif when using ovs literally does nothing | |
| 18:44:35 | mnaser | jaypipes: yeah, TIL yesterday | |
| 18:44:40 | mnaser | or YIL i guess | |
| 18:44:48 | mnaser | i thought it was a bit more complex than that | |
| 18:44:50 | jaypipes | heh | |
| 18:45:31 | melwitt | yeah, but the plug? we asked people in #openstack-neutron and the consensus at the time was that the ovs agent would detect things and send events | |
| 18:45:37 | mnaser | now question is does anyone know why os_vif_ovs does nothing when nova uses the 'unplug' method in it? win32 seems to remove the port from the bridge | |
| 18:45:42 | mriedem | melwitt: this isn't 'ovs' this, | |
| 18:45:44 | mriedem | it's opendaylight | |
| 18:45:59 | melwitt | okay, so things are working fine for ovs then | |
| 18:46:03 | mnaser | correct | |
| 18:46:04 | mriedem | apparently same vif type, | |
| 18:46:09 | mnaser | ^ and that too | |
| 18:46:09 | mriedem | different backend implementation | |
| 18:46:17 | mriedem | does the vif type == networking API or something? | |
| 18:46:23 | mriedem | and odl implements the ovs api? | |
| 18:46:25 | melwitt | sorry, I got confused by the earlier mention of adding VIF_TYPE_OVS to the blacklist? | |
| 18:46:37 | melwitt | is that because opendaylight falls under that vif type? | |
| 18:46:41 | mnaser | correct | |
| 18:46:41 | mriedem | melwitt: yes | |
| 18:46:48 | jaypipes | mnaser: I'll be honest. I don't know why for OVS we don't do anything for OVS when !Windows. | |
| 18:46:56 | melwitt | okay, gotcha now | |
| 18:47:13 | mriedem | if we had to add ovs to that blacklist, then there is really no point in even waiting for vif plug events during a hard reboot with the libvirt driver | |
| 18:47:20 | mnaser | pretty much :\ | |
| 18:47:21 | melwitt | right | |
| 18:47:30 | mnaser | because i think if os_vif_ovs did an actual unplug, the state in neutron would be updated, and on the actual plug later, the state would be updated again and a notification goes out | |
| 18:47:40 | jaypipes | mnaser: I'm going to git blame it... one sec | |
| 18:47:41 | mnaser | i'm *assuming* neutron-openvswitch-agent magically realizes things have been unplugged and removes them | |
| 18:48:12 | mnaser | jaypipes: i tried to git blame for a while and didn't get anywhere productive but my git-fu isn't strong | |
| 18:49:19 | jaypipes | mnaser: it was Rawlin Peters who added this code. | |
| 18:49:22 | mnaser | again it could be something odl should be handling, does nova contact neutron for the unplug at the api layer? | |
| 18:49:26 | jaypipes | mnaser: I don't know Rawlin. | |
| 18:49:37 | mnaser | :< | |
| 18:49:42 | jaypipes | https://github.com/openstack/os-vif/commit/d5b119ba37d5c724d1279cb2292dce05c32b51a0 | |
| 18:49:54 | mriedem | mnaser: when we were talking about the differences between openvswith and linuxbridge agents here, sean-k-mooney was saying that ovs will send a notification which the neutron agent proxies to the server which is why ovs works and we don't miss plug events, but LB does polling and there could be a window where we miss it and then timeout waiting for an event that neutron never sends | |
| 18:49:58 | jaypipes | mnaser: https://review.openstack.org/#/c/333486/ | |
| 18:50:10 | mriedem | mnaser: no it's not a REST API call | |
| 18:50:13 | jaypipes | perhaps we should ask danpb and sean-k-mooney since they approved this patch. | |
| 18:50:20 | mriedem | it's a change on the host which the neutron agent routes up to neutron server | |
| 18:50:38 | sean-k-mooney | i saw my name | |
| 18:50:41 | sean-k-mooney | reading | |
| 18:50:47 | mnaser | ok i see, so given that neutron agent doesnt exist on the host, it would never end up hitting neutron | |
| 18:50:55 | melwitt | mnaser: yeah, we only do a unplug and plug through os-vif, no calls to neutron (unless os-vif does any, which I don't think it does) | |
| 18:51:04 | mriedem | melwitt: it doesn't | |
| 18:51:09 | jaypipes | melwitt: it does not, no. | |
| 18:51:12 | mriedem | it's like os-brick, just all host level stuff | |
| 18:51:14 | jaypipes | jinx | |
| 18:51:19 | melwitt | cool | |
| 18:51:27 | jaypipes | mriedem: it's like os-brick, only it bricked the host. | |
| 18:51:32 | mnaser | so maybe if we actually unplugged in os-vif, then it probably wont "hurt" the ovs-agent and things like odl would notice that the port is gone | |
| 18:51:48 | mnaser | and then update neutron in their own little mechanisms | |
| 18:51:54 | sean-k-mooney | oh yes the neutron l2 agent subscribes to notification from the ovs db instead of polling so it gets notified when we do plug and unplug on a hard reboot | |
| 18:52:15 | mnaser | sean-k-mooney: but os_vif unplug with ovs does nothing | |
| 18:52:35 | mnaser | https://github.com/openstack/os-vif/blob/master/vif_plug_ovs/ovs.py#L266-L268 | |
| 18:52:44 | mnaser | so im not sure how n-ovs-agent even knows something changed | |
| 18:53:00 | mriedem | jaypipes: rimshot'ed | |
| 18:53:04 | sean-k-mooney | mnaser: yes libvirt does the port plug on the ovs bridge in that case | |