| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-14 | |||
| 18:19:13 | jaypipes | mnaser: crap, sorry man. got distracted. looking now while tests are running locally. | |
| 18:19:38 | mnaser | jaypipes: np! i can get you a bit to what point i reached from my debugging | |
| 18:20:18 | jaypipes | sure thing. | |
| 18:20:21 | jaypipes | got for it. | |
| 18:21:13 | mnaser | jaypipes: so https://review.openstack.org/#/q/Ib08afad3822f2ca95cfeea18d7f4fc4cb407b4d6 which was merged in master and in the process of getting backported changed behavior (after another change) where now, nova expects a vif-network-plugged event even on reboots | |
| 18:21:45 | mnaser | now, it looks like linuxbridge broke because it didn't send that event, so a little work around was added in that patch above to skip that. now, opendaylight is broken because it doesn't send a notification either. | |
| 18:22:07 | mnaser | upon digging on *why* it doesn't send one, it looks like unplug with ovs on linux is a noop in os_vif | |
| 18:22:31 | mnaser | which means that the port is never really unplugged in odl, so when it tries to 'plug' it, neutron never really does anything because the port state never changes, and the server times out booting because vif_plugging_timeout | |
| 18:22:57 | mnaser | noop unplug here = https://github.com/openstack/os-vif/blob/master/vif_plug_ovs/ovs.py#L266-L268 | |
| 18:23:13 | jaypipes | mnaser: and this is only on hard reboot, yes? | |
| 18:23:26 | mnaser | jaypipes: correct, but afaik a clean state is a hard reboot too in the nova codebase? | |
| 18:23:42 | mnaser | clean start* | |
| 18:23:54 | mnaser | so first time you start an instance, it'll be okay, but stop and start again will cause it to want a plugged event | |
| 18:23:58 | jaypipes | mnaser: no. a clean start does not issue a call to unplug() in the os-vif API. | |
| 18:24:42 | mnaser | jaypipes: right, but a clean start means the port is created the first time and the vif plugged event comes through fine, but when the instance is stopped, nova calls unplug (but nothing happens) and when it starts again, it waits for vif plugged event (those patches changed that behaviour) | |
| 18:25:11 | mnaser | this was noticed when the tempest start_stop tests failed because the startup would time out | |
| 18:26:02 | jaypipes | hm | |
| 18:26:03 | mnaser | http://logs.openstack.org/22/552922/1/check/networking-odl-tempest-oxygen/277adb4/testr_results.html.gz (you can look at nova logs there, but pretty much test_stop_start_server / test_reboot_server_hard / etc are the ones that arent working now) | |
| 18:26:43 | mnaser | and failure is .. "Details: (ServerActionsTestJSON:test_stop_start_server) Server da9cead9-c217-495e-97a8-65a6adacf37c failed to reach ACTIVE status and task state "None" within the required time (196 s). Current status: SHUTOFF. Current task state: powering-on." .. nova logs shows it timing out after 5 minutes | |
| 18:28:07 | jaypipes | mnaser: k. just a minute. reading through these patches... | |
| 18:28:13 | mnaser | sure thing | |
| 18:35:31 | jaypipes | mnaser: ok, done. so are you suggesting the fix here is to add VIF_TYPE_OVS to line 5392 here? https://review.openstack.org/#/c/541442/6/nova/virt/libvirt/driver.py | |
| 18:36:18 | mnaser | jaypipes: that would be a fix, or os_vif *actually* unplugging things could be a fix too | |
| 18:36:27 | mnaser | i do feel if that list starts growing, it might start become confusing for users :< | |
| 18:36:43 | jaypipes | mnaser: agreed. | |
| 18:37:12 | jaypipes | mnaser: for os-vif linux bridge, though, I'm not entirely sure what should be done on unplug... | |
| 18:37:50 | mnaser | jaypipes: yeah.. that's beyond me, but for openvswitch, im not sure if there is a link state, or maybe just deleting the actual port if its 'unplugged' | |
| 18:38:05 | mnaser | im sure there's a really good reason why it's not being deleted though, but i don't know why | |
| 18:38:37 | jaypipes | mnaser: I'm also not sure whether that would *ensure* that a vif-unplugged *event* from Neutron would be received... | |
| 18:38:43 | jaypipes | sean-k-mooney: you still around? | |
| 18:39:21 | mnaser | jaypipes: well, nova doesnt care about vif-unplugged afaik, issue is surrounding vif-plugged event | |
| 18:39:50 | mnaser | jaypipes: based on my reading, the nova notifications are a hook to the db model in neutron when an object changes. so if it goes from active => active, nothing is being updated and it never sent anything | |
| 18:40:03 | mriedem | lyarwood: melwitt: probably want to listen to ^ | |
| 18:40:03 | mnaser | now, maybe neutron-openvswitch-agent does things differently to trigger a change, i'm not sure. | |
| 18:40:34 | jaypipes | mnaser: not the neutron DB. the ovsdb... | |
| 18:40:35 | mriedem | related https://review.openstack.org/#/c/550046/ | |
| 18:40:41 | jaypipes | mnaser: and that's only for OVS of course. | |
| 18:40:47 | melwitt | mriedem: thanks | |
| 18:41:52 | mriedem | adding VIF_TYPE_OVS to the blacklist would defeat the purpose of the change, which is apparently working for ovs | |
| 18:41:57 | mriedem | we use ovs in our normal gate jobs | |
| 18:42:16 | mnaser | https://github.com/openstack/neutron/blob/master/neutron/db/db_base_plugin_v2.py#L155-L169 | |
| 18:42:17 | mriedem | building a giant list of which vifs types we should not wait on is going to be shitty | |
| 18:42:29 | melwitt | yeah, we were under the impression that ovs *does* send events for the plug in the hard reboot case | |
| 18:42:37 | melwitt | and that it was only linuxbridge that doesn't | |
| 18:42:46 | mnaser | right, but for context, i'm talking about the case when opendaylight is used here | |
| 18:42:54 | mriedem | does that use vif type ovs? | |
| 18:42:56 | jaypipes | mriedem: you mean like this? https://github.com/openstack/os-vif/blob/master/vif_plug_ovs/ovs.py#L266-L268 | |
| 18:42:57 | mnaser | yes | |
| 18:42:59 | mriedem | guh | |
| 18:43:14 | jaypipes | mriedem: welcome to our own private Idahell. | |
| 18:43:15 | mriedem | jaypipes: heh shitty indeed | |
| 18:43:21 | mriedem | can i be keanu? | |
| 18:43:23 | melwitt | originally, the patch never waited for any events because we thought during a hard reboot, the neutron agent would never detect us doing a os-vif unplug/plug | |
| 18:43:24 | jaypipes | no. | |
| 18:43:44 | mnaser | so afaik odl just 'notices' ports appearing, and in neutron-server there is a web socket connection to odl which notifies it that a port is appeared, so it sets it to active | |
| 18:44:01 | melwitt | then we added another patch to only not wait if linuxbridge because we thought ovs was detecting the unplug/plug | |
| 18:44:16 | mnaser | and so what happens is that, after the hard reboot, the port is always still plugged and didnt get a state change, so nothing happens in neutron and no notification goes out | |
| 18:44:28 | jaypipes | mnaser: wow. I had no idea there were hard-coded hooks in the neutron base db plugin thing. :( | |
| 18:44:32 | mnaser | well, the unplug in os_vif when using ovs literally does nothing | |
| 18:44:35 | mnaser | jaypipes: yeah, TIL yesterday | |
| 18:44:40 | mnaser | or YIL i guess | |
| 18:44:48 | mnaser | i thought it was a bit more complex than that | |
| 18:44:50 | jaypipes | heh | |
| 18:45:31 | melwitt | yeah, but the plug? we asked people in #openstack-neutron and the consensus at the time was that the ovs agent would detect things and send events | |
| 18:45:37 | mnaser | now question is does anyone know why os_vif_ovs does nothing when nova uses the 'unplug' method in it? win32 seems to remove the port from the bridge | |
| 18:45:42 | mriedem | melwitt: this isn't 'ovs' this, | |
| 18:45:44 | mriedem | it's opendaylight | |
| 18:45:59 | melwitt | okay, so things are working fine for ovs then | |
| 18:46:03 | mnaser | correct | |
| 18:46:04 | mriedem | apparently same vif type, | |
| 18:46:09 | mriedem | different backend implementation | |
| 18:46:09 | mnaser | ^ and that too | |
| 18:46:17 | mriedem | does the vif type == networking API or something? | |
| 18:46:23 | mriedem | and odl implements the ovs api? | |
| 18:46:25 | melwitt | sorry, I got confused by the earlier mention of adding VIF_TYPE_OVS to the blacklist? | |
| 18:46:37 | melwitt | is that because opendaylight falls under that vif type? | |
| 18:46:41 | mriedem | melwitt: yes | |
| 18:46:41 | mnaser | correct | |
| 18:46:48 | jaypipes | mnaser: I'll be honest. I don't know why for OVS we don't do anything for OVS when !Windows. | |
| 18:46:56 | melwitt | okay, gotcha now | |
| 18:47:13 | mriedem | if we had to add ovs to that blacklist, then there is really no point in even waiting for vif plug events during a hard reboot with the libvirt driver | |
| 18:47:20 | mnaser | pretty much :\ | |
| 18:47:21 | melwitt | right | |
| 18:47:30 | mnaser | because i think if os_vif_ovs did an actual unplug, the state in neutron would be updated, and on the actual plug later, the state would be updated again and a notification goes out | |
| 18:47:40 | jaypipes | mnaser: I'm going to git blame it... one sec | |
| 18:47:41 | mnaser | i'm *assuming* neutron-openvswitch-agent magically realizes things have been unplugged and removes them | |
| 18:48:12 | mnaser | jaypipes: i tried to git blame for a while and didn't get anywhere productive but my git-fu isn't strong | |
| 18:49:19 | jaypipes | mnaser: it was Rawlin Peters who added this code. | |
| 18:49:22 | mnaser | again it could be something odl should be handling, does nova contact neutron for the unplug at the api layer? | |
| 18:49:26 | jaypipes | mnaser: I don't know Rawlin. | |
| 18:49:37 | mnaser | :< | |
| 18:49:42 | jaypipes | https://github.com/openstack/os-vif/commit/d5b119ba37d5c724d1279cb2292dce05c32b51a0 | |
| 18:49:54 | mriedem | mnaser: when we were talking about the differences between openvswith and linuxbridge agents here, sean-k-mooney was saying that ovs will send a notification which the neutron agent proxies to the server which is why ovs works and we don't miss plug events, but LB does polling and there could be a window where we miss it and then timeout waiting for an event that neutron never sends | |
| 18:49:58 | jaypipes | mnaser: https://review.openstack.org/#/c/333486/ | |
| 18:50:10 | mriedem | mnaser: no it's not a REST API call | |
| 18:50:13 | jaypipes | perhaps we should ask danpb and sean-k-mooney since they approved this patch. | |
| 18:50:20 | mriedem | it's a change on the host which the neutron agent routes up to neutron server | |
| 18:50:38 | sean-k-mooney | i saw my name | |
| 18:50:41 | sean-k-mooney | reading | |
| 18:50:47 | mnaser | ok i see, so given that neutron agent doesnt exist on the host, it would never end up hitting neutron | |