| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-14 | |||
| 18:51:19 | melwitt | cool | |
| 18:51:27 | jaypipes | mriedem: it's like os-brick, only it bricked the host. | |
| 18:51:32 | mnaser | so maybe if we actually unplugged in os-vif, then it probably wont "hurt" the ovs-agent and things like odl would notice that the port is gone | |
| 18:51:48 | mnaser | and then update neutron in their own little mechanisms | |
| 18:51:54 | sean-k-mooney | oh yes the neutron l2 agent subscribes to notification from the ovs db instead of polling so it gets notified when we do plug and unplug on a hard reboot | |
| 18:52:15 | mnaser | sean-k-mooney: but os_vif unplug with ovs does nothing | |
| 18:52:35 | mnaser | https://github.com/openstack/os-vif/blob/master/vif_plug_ovs/ovs.py#L266-L268 | |
| 18:52:44 | mnaser | so im not sure how n-ovs-agent even knows something changed | |
| 18:53:00 | mriedem | jaypipes: rimshot'ed | |
| 18:53:04 | sean-k-mooney | mnaser: yes libvirt does the port plug on the ovs bridge in that case | |
| 18:53:08 | jaypipes | mriedem: :) | |
| 18:53:21 | mnaser | sean-k-mooney: ok, so pretty much it doesn't know things were unplugged, but it knows when they're plugged | |
| 18:53:22 | melwitt | how come odl doesn't have its own plugin for os-vif? | |
| 18:53:33 | sean-k-mooney | odl is not a network backend | |
| 18:53:42 | sean-k-mooney | odl mananges ovs | |
| 18:53:43 | jaypipes | it's an SDN controller. | |
| 18:53:43 | melwitt | oh | |
| 18:53:55 | sean-k-mooney | same for ovn | |
| 18:54:43 | melwitt | so odl manages ovs, but in the setup there are no ovs agents running on compute hosts | |
| 18:54:48 | mnaser | melwitt: correct | |
| 18:55:14 | mriedem | it's possible there is something in the port binding profile that tells us it's odl and not native ovs, but then again if we had to check that level of detail in the driver logic, it's going to suck | |
| 18:55:21 | mriedem | given all of the different neutron backends | |
| 18:55:23 | sean-k-mooney | neutron is repsocible for managing l2 and up nova/os-vif handel l1 netowrking for instances | |
| 18:55:50 | melwitt | yeah, I didn't realize the ovs agent was optional if the backend is ovs | |
| 18:56:10 | mnaser | backend is odl* :) | |
| 18:56:18 | mriedem | i didn't realize you could have different backends for the same vif type either | |
| 18:56:38 | sean-k-mooney | melwitt: well the reference implentation use the agent but, ovn,onos and odl can all be used instead of the agents | |
| 18:56:41 | melwitt | sean-k-mooney said odl isn't a network backend. sorry, I lack knowledge on these | |
| 18:56:56 | melwitt | okay, so they replace the agents. thanks | |
| 18:57:02 | sean-k-mooney | melwitt: for a neutron vif_type perspctive its nots | |
| 18:58:37 | sean-k-mooney | by the way i think there might be a way to get rid of the polling in the linux bridge agent if you use pyinotify but i have not run the idea past any of the neutron folks | |
| 18:58:40 | mriedem | actually there isn't really anything in the port information that would help us distinguish odl vifs http://logs.openstack.org/22/552922/1/check/networking-odl-tempest-oxygen/277adb4/controller/logs/screen-n-cpu.txt.gz#_Mar_14_15_53_40_140157 | |
| 18:58:58 | sean-k-mooney | mriedem: correct infact that is by design | |
| 18:59:18 | sean-k-mooney | mriedem: if there was the tenant could also tell | |
| 18:59:49 | mriedem | welp, we should probably not wait for vif plugged events on hard reboot in the libvirt driver then | |
| 18:59:56 | mriedem | if we can't determine accurately when that will work | |
| 19:00:11 | mriedem | which means reverting all of these https://review.openstack.org/#/q/Ib08afad3822f2ca95cfeea18d7f4fc4cb407b4d6 | |
| 19:00:17 | jaypipes | yeah | |
| 19:00:18 | melwitt | urgh | |
| 19:00:28 | mriedem | dansmith: ^ fyi | |
| 19:00:36 | sean-k-mooney | i missed the start of this convo. are ye discussing the fact that odl says it is finished wiring up the port imediatly when you bind the port | |
| 19:00:42 | dansmith | yeah I'm following | |
| 19:00:48 | dansmith | that does indeed suck | |
| 19:00:51 | mriedem | sean-k-mooney: odl doesn't send the vif plugged event | |
| 19:01:00 | mnaser | mriedem: it does send it, but only when the port is first created | |
| 19:01:02 | mriedem | but is the ovs vif type | |
| 19:01:07 | mriedem | mnaser: ok | |
| 19:01:19 | mnaser | mriedem: the port is never ever unplugged from the openvswitch port on shutdown (when nova calls unplug()) | |
| 19:01:23 | sean-k-mooney | mriedem: it send it when the port is bound not when the port is wired up | |
| 19:01:30 | mnaser | so when the vm is started again, it wont send it, because the port is already uo | |
| 19:01:35 | melwitt | sean-k-mooney: we were trying to Do The Right Thing and wait for vif plug events when we unplug and replug vifs during a hard reboot. but that's messing up for odl because it doesn't send a plug event in that scenario | |
| 19:01:53 | melwitt | and by unplug and replug we mean through os-vif only | |
| 19:01:58 | sean-k-mooney | melwitt: to make that work with odl we would have to bind the port again | |
| 19:01:59 | mriedem | and according to that other patch from lyarwood, there are other cases | |
| 19:02:09 | mriedem | https://review.openstack.org/#/c/550046/ | |
| 19:02:17 | mnaser | yeah so that's gonna be a little list of 'things that don't send notifications' | |
| 19:02:29 | mnaser | (i can imagine that you'll have other closed source sdns showing up soon too) | |
| 19:02:30 | mriedem | sean-k-mooney: and that would be a bigger change, to re-bind the port | |
| 19:02:57 | sean-k-mooney | mriedem: yes it would. we could explore that at some point but prehaps not now | |
| 19:03:16 | melwitt | yeah. I was wondering, is there a call to neutron we could make that would be mostly a no-op, not actually rebind it but update it to the same binding profile or something like that? | |
| 19:03:35 | melwitt | I don't know if neutron would emit an event if nothing changed | |
| 19:03:51 | mnaser | melwitt: it would not, i looked into the logic, its a db hook that checks if states changed | |
| 19:03:54 | sean-k-mooney | as context the odl ml2 works the way it does today because odl does not have acess to rabitmq to be able to notify nova/neutron when its finished wireing things up | |
| 19:03:59 | melwitt | mnaser: ah, rats | |
| 19:04:03 | mnaser | https://github.com/openstack/neutron/blob/master/neutron/db/db_base_plugin_v2.py#L155-L169 | |
| 19:04:28 | mnaser | https://github.com/openstack/neutron/blob/master/neutron/notifiers/nova.py | |
| 19:04:31 | mnaser | and it explicitely ignores noops | |
| 19:05:03 | mriedem | nova would have to unbind/bind or change the state of the binding (i think) | |
| 19:05:08 | mriedem | and that all seems scary and hacky | |
| 19:05:18 | mnaser | mriedem: and not something you'd want to backport all the way to stable/ocata either | |
| 19:05:26 | melwitt | yeah, I don't want to do that | |
| 19:05:38 | sean-k-mooney | mriedem: ya we would have to set the host_id in the binding profile to None then back to the hostname to get it to work | |
| 19:06:24 | mriedem | well the good news is the original change didn't make it to newton | |
| 19:07:01 | sean-k-mooney | mriedem: maybe we can use the multiple port bindings neutron are adding to adress this in the future | |
| 19:07:05 | melwitt | I was thinking if we could not actually change it, update it to the same, and it would emit a notification, that might be cool. but I definitely don't want to unbind and rebind it just for this. guess it all depends on what's the impact of not waiting for the event. instance says ACTIVE when it might not actually have networking yet? | |
| 19:07:24 | mriedem | melwitt: yes | |
| 19:07:57 | mriedem | doing that for hard reboot is less shitty than initial create, at least from our CI system | |
| 19:08:13 | sean-k-mooney | mriedem: how would you feel about nova calling activate on the port binding again on a hard reboot? | |
| 19:08:15 | openstackgerrit | Arvind Nadendla proposed openstack/nova-specs master: Support traits in Glance https://review.openstack.org/541507 | |
| 19:08:21 | mriedem | any places in tempest that fallout as a result could wait on the port being ACTIVE in addition to the server being ACTIVE | |
| 19:08:38 | mriedem | sean-k-mooney: calling activate? | |
| 19:08:48 | melwitt | yeah, what's that do? | |
| 19:09:07 | sean-k-mooney | in the multiple port binding spec there is a new api call to activate a binding as part of live migration | |
| 19:09:16 | mriedem | if it relies on the new porting binding api, that doesn't exist yet, and isn't backportable :) | |
| 19:09:33 | mriedem | sean-k-mooney: you just mean changing the binding status to 'active' right? | |
| 19:09:46 | mriedem | sean-k-mooney: for a hard-reboot with a single host binding, it would already be active | |
| 19:09:53 | sean-k-mooney | well my taught would be allow that to tell neutron to validate its wired up again because did something(reboot) you man not have seen | |
| 19:10:47 | melwitt | that sounds like what we want, but as mriedem said, not backportable | |
| 19:10:58 | mriedem | mnaser: other good news is ocata isn't going anywhere :) https://review.openstack.org/#/c/548916/ | |
| 19:11:02 | sean-k-mooney | mriedem: yes it would. but since that api is not in a release yet we could ammend it to say calling activate on an active binding forces it to wire it up again or at least check its in the active state | |
| 19:11:28 | mriedem | sean-k-mooney: that's pretty hacky, | |
| 19:11:37 | melwitt | and validating it would cause an event to be sent? | |
| 19:11:39 | mriedem | i think you'd have to deactivate the binding, and then activate it | |
| 19:12:11 | mriedem | /ports/{id}/binding/{host}/action :) | |
| 19:12:12 | sean-k-mooney | melwitt: validating would sent the event if it actully changed something | |
| 19:12:13 | mriedem | i can see it now | |
| 19:12:33 | melwitt | sean-k-mooney: yeah, I guess that seems similar to just unbinding it and binding it again | |
| 19:12:33 | mriedem | so for present day options, | |
| 19:12:44 | mriedem | i think we're talking revert right? | |
| 19:12:48 | sean-k-mooney | melwitt: yep it is | |