Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-03-14
19:20:03 dansmith mnaser: yeah less of a concern on reboot for that reason
19:20:05 sean-k-mooney mnaser: on hard reboot it is unpluged
19:20:15 sean-k-mooney mnaser: on soft reboot it is not
19:20:26 mnaser sean-k-mooney: but the unplug in os_vif seems to be noop from what i see (or maybe im misunderstanding things)
19:20:36 mnaser sean-k-mooney: https://github.com/openstack/os-vif/blob/master/vif_plug_ovs/ovs.py#L266-L268 nothing happens here...
19:20:45 sean-k-mooney mnaser: that only because in that code path the plugin is done by libvirt
19:20:59 mnaser oh okay sorry i'm following you now
19:21:06 mnaser the actual unplug happens by libvirt which is then picked up by n-ovs-agent
19:21:34 sean-k-mooney yep when we do the domain destroy on hard reboot libvirt remove the interface form ovs and deletes teh tap
19:21:54 mnaser ok i understand now
19:21:56 mnaser thank youy
19:22:17 cfriesen my organization has a local patch to make plug_vifs() optionally wait until the vifs are actually plugged in order to ensure that things work reliably during a live migration...we added that a long time ago and I was wondering if it's still needed, but it sounds like it is.
19:22:44 sean-k-mooney cfriesen: when you say actully plugged what do you wait for
19:23:06 dansmith we can't know that the plumbing is done behind the neutron curtain without the event right?
19:23:39 sean-k-mooney cfriesen: if its external-ids:iface-status=active in the ovs db we hard code that here https://github.com/openstack/os-vif/blob/master/vif_plug_ovs/linux_net.py#L71
19:24:35 sean-k-mooney dansmith: technically for ovs and ovs agents only we shoudl be able to use external-ids:iface-status=active but i dont know for odl
19:24:54 dansmith but that's a very specific case
19:24:55 sean-k-mooney dansmith: for lb or vpp or whatever else we have no way to tell
19:24:57 dansmith for that one type
19:24:59 dansmith yeah
19:25:36 mnaser mriedem: https://bugs.launchpad.net/nova/+bug/1755890 i tried to write a basic description and referenced this conversation but yeah feel free to update it with any lacking details
19:25:37 openstack Launchpad bug 1755890 in OpenStack Compute (nova) "Instances fail to hard reboot when using OpenDaylight" [Undecided,New]
19:25:38 cfriesen sean-k-mooney: actually, never mind. the only one with a useful query is for our custom networking thing, the others are assumed to always be up. My bad.
19:25:47 melwitt I have to run for now but I'm cool with reverting those patches if that's the best option we have. bbl
19:25:59 arvindn05 mriedem: jaypipes: updated the spec based on comments Patch set 9 should address the issues https://review.openstack.org/#/c/541507/9
19:27:09 sean-k-mooney cfriesen: the dpdk based titainium server vswitch i assuem
19:27:17 cfriesen yep
19:28:07 cfriesen it'll query the vswitch API
19:29:03 sean-k-mooney cfriesen: ya the issue is that bar seeting external-ids:iface-status=active which is an ovs agent specif thing there is no way to tell for ovs in general.
19:31:51 sean-k-mooney when odl first intregrated support into the ml2 framework this gap was discussed back in icehouse but they had no way syncronis state in odl with neutron without a intoducing an sdn constrolller specific api, b having odl emit notifcation on the rabbitmq bus or be haveing the neutron server poll odl for state change
19:32:27 sean-k-mooney the chose to do non of the above an just report vif_plugged on port binding instead and its been that way ever since
19:32:59 mnaser sean-k-mooney: maybe this is a networking-odl bug? i know that right now it creates a websocket that listens for state updates
19:33:01 mnaser https://github.com/openstack/networking-odl/blob/master/networking_odl/ml2/port_status_update.py
19:33:28 mnaser https://bugs.launchpad.net/networking-odl/+bug/1686023
19:33:29 openstack Launchpad bug 1686023 in networking-odl "networking-odl dynamic port status update full support missing" [Low,In progress]
19:33:32 mriedem jaypipes: replied in https://review.openstack.org/#/c/539605/ - need bauzas to probably elaborate at this point since i don't remember this being talked about at the ptg
19:33:42 mnaser i guess they support DOWN => ACTIVE but not ACTIVE=>DOWN
19:34:44 mnaser i wonder if there's an easy way to let neutron know that the port has gone down in there
19:34:45 mnaser and avoid the revert
19:34:48 sean-k-mooney mnaser: well i think its a ml2 framework feature request. e.g. allow agentless backend notify neutron of state changes
19:35:15 mriedem if we can blame this all on an incomplete neutron backend then that works for me
19:35:19 mnaser lolol
19:35:19 sean-k-mooney mnaser: that bug seams to focus on admin state which is different
19:36:01 mriedem https://review.openstack.org/#/c/465463/
19:36:04 mnaser sean-k-mooney: https://github.com/openstack/networking-odl/blob/master/networking_odl/ml2/port_status_update.py#L91-L95 ever that little bit? the bug talks about admin state but it seems to get the actual port state
19:37:20 sean-k-mooney mnaser: yest that should adress this but it seams that odl is not detecting the removal of the port and readding it
19:38:22 sean-k-mooney mnaser: perhaps the networking-odl ml2 driver is just not sending the notification to nova when it recives the notification from odl
19:38:41 mriedem shouldn't the driver go through the normal notificatoin code in neutron?
19:39:05 mnaser sean-k-mooney: its certainly not sending it when its being unplugged, but it looks like the code to watch for state doesnt even update it to 'down' or 'unplugged' or whatever state it should be in neutron
19:39:17 sean-k-mooney mriedem: that notification code is triggered by port status updates on the rpc bus
19:39:28 sean-k-mooney mriedem: these update from odl are from the websocket
19:41:19 sean-k-mooney mriedem: since we are not storing the state changes from https://github.com/openstack/networking-odl/blob/master/networking_odl/ml2/port_status_update.py#L91-L95 in the db i dont think https://github.com/openstack/neutron/blob/master/neutron/db/db_base_plugin_v2.py#L155-L169 will be invoked
19:41:52 mnaser sean-k-mooney: is there an example of a plugin that is storing/updating state changes in db which might be good to reference
19:42:58 sean-k-mooney maybe look at the ovs agent code.
19:43:05 mnaser https://github.com/openstack/networking-ovn/blob/master/networking_ovn/ml2/mech_driver.py#L713-L729
19:43:13 mnaser i guess i can use that as a reference and try submitting something to networking-odl
19:44:27 sean-k-mooney i dont think we actully need to store it in the db. we just need to send the notification to nova
19:45:07 mnaser i guess one thing though is talking about the revert is... if this is going to affect other drivers too
19:45:16 mnaser aka we fix odl but find 4 other vif types that are affected by that issue
19:46:17 sean-k-mooney mnaser: we are already special casing lb and we need to skip waithing i think for sriov too so basically we can only wait for ovs with agents today
19:46:54 sean-k-mooney maybe we can wait for ovn? but not sure.
19:49:47 mriedem unless the port profile/details add some specific flag that says we can expect events on replug, i don't think it's safe to just have the blanket wait that we have today
19:50:12 openstackgerrit Ed Leafe proposed openstack/nova master: Add 'member_of' param to GET /allocation_candidates https://review.openstack.org/552098
19:50:15 edleafe jaypipes: cdent: ^^ Addressed your comments
19:50:42 sean-k-mooney mriedem: that is true. i think this is something that first needs to made a requirement for all neutron backend before nova can start to depend on it.
19:51:13 sean-k-mooney mriedem: one option is add a nova config opetion to the compute node? we wait there correct? but not ideal
19:52:42 mriedem it's an option yes but not a good one imo
19:52:53 mriedem we already have vif_plugging_timeout and vif_plugging_is_fatal
19:53:11 mriedem we'd have to add a vif_plugging_reboot_if_hard_and_libvirt_and_thursday_timeout
19:54:46 sean-k-mooney hehe well it could be vif_wait_for_network but ya only ovs + reference agents could set it to true
19:55:11 sean-k-mooney it would have to default to false and proably should be deprecated from the start
19:56:01 mnaser i guess a revert seems like the cleanest possible choice? :x
19:56:10 jaypipes edleafe: +2 from me.
19:56:10 sean-k-mooney mnaser: ya
19:56:30 mnaser i'm not sure how a revert would happen because this seems to be 2 patches. a patch that reverts both referencing both commits?
19:56:36 mnaser unless someone else volunteers to do this
19:57:34 sean-k-mooney what were the 2 commits again. it will be next week if i do it at the earliest.
19:57:56 mriedem i think it's just this one https://review.openstack.org/#/q/Ib08afad3822f2ca95cfeea18d7f4fc4cb407b4d6
19:58:21 mriedem that's the one that adding the wait back in
19:58:23 mnaser what about https://review.openstack.org/#/q/Ib0cf5d55750f13d0499a570f14024dca551ed4d4 ?
19:58:39 mriedem that's what you want to happen, not wait
19:58:56 mriedem there were 3 changes:
19:59:21 mriedem 1. full hard reboot blasting everything away and re-plug; that caused issues because we were waiting for something that wouldn't happen
19:59:29 mriedem 2. to workaround ^, we stopped waiting
19:59:46 mriedem 3. because not waiting when it will work for some (ovs but not lb), the final patch was added which added the bridge conditional
19:59:53 mriedem but as we've now seen, that doesn't work for ovs (odl)
20:00:04 mriedem so revert #3 to go back to not waiting
20:00:15 mriedem which is https://review.openstack.org/#/c/541442/
20:00:25 sean-k-mooney mriedem: right https://review.openstack.org/#/c/541442/6/nova/virt/libvirt/driver.py skip waiting for lb and then larwood was extending it for sriov then odl came up
20:00:45 mriedem yeah so it's whack-a-mole at this point
20:01:05 mriedem heh, we could just revert all of them
20:01:06 mnaser so revert master and then cherry pick the reverts once they land?
20:01:08 mriedem going back to https://review.openstack.org/#/c/400384/
20:01:18 mriedem mnaser: cherry picking reverts is weird,
20:01:22 sean-k-mooney mriedem: so one patch to just remove waithing on reboot and backporting that is proable the simpelst thing
20:01:24 mriedem i'd just revert individually on all branches
20:01:26 mnaser ok i see
20:01:37 sean-k-mooney mriedem: or that.
20:02:21 mriedem i was never fully comfortable with https://review.openstack.org/#/c/400384/
20:02:34 mriedem it was trying to fix a problem with encrypted volumes and overshot probably to include vifs

Earlier   Later