| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-14 | |||
| 19:12:50 | mriedem | back through queens and pike | |
| 19:13:01 | melwitt | I guess so. doesn't seem like we have any choice | |
| 19:13:10 | sean-k-mooney | mriedem: revert waiting. yes i think so | |
| 19:13:33 | mriedem | dansmith: jaypipes: how are you feeling about a revert | |
| 19:13:37 | mriedem | i don't see other very good options | |
| 19:13:44 | dansmith | yeah I mean, are there other options? | |
| 19:13:54 | mriedem | not outside of changing the port binding on reboot | |
| 19:13:55 | sean-k-mooney | i think we can/shoudl fix it for rocky so we can reenable waiting | |
| 19:13:57 | mriedem | which is a bigger change | |
| 19:14:00 | melwitt | unbinding the port and binding it again | |
| 19:14:30 | mriedem | sean-k-mooney: i think that would also require talking to the neutron team about whether or not that would do what we need | |
| 19:14:40 | mriedem | i can only talk to miguel once per day, he said | |
| 19:14:46 | melwitt | haha | |
| 19:14:56 | cfriesen | are there any races that come from not waiting for vifs to actually be plugged? | |
| 19:15:05 | mriedem | mnaser: is there a bug report for the ODL issue? | |
| 19:15:08 | mriedem | cfriesen: yes | |
| 19:15:08 | dansmith | yes | |
| 19:15:09 | sean-k-mooney | "in general" un binding and binding the port "should" be safe but it can fail | |
| 19:15:16 | mriedem | tempest races assuming it can ssh into the guest | |
| 19:15:18 | melwitt | cfriesen: instance can go to ACTIVE state without having networking yet | |
| 19:15:30 | mnaser | mriedem: in ODL world yes, but not in nova world, but i can make one | |
| 19:15:37 | mriedem | mnaser: yes please for tracking | |
| 19:15:43 | mnaser | ok | |
| 19:15:47 | melwitt | well, for ssh I know from using devstack that the instance says ACTIVE before ssh daemon is ready | |
| 19:15:52 | sean-k-mooney | cfriesen: odl has no mechanisium to notify nova today that they have been plugged | |
| 19:15:54 | melwitt | I thought | |
| 19:16:01 | sean-k-mooney | * notify neutron | |
| 19:16:19 | mriedem | melwitt: if that were true, | |
| 19:16:27 | mriedem | our CI would shit its pants daily | |
| 19:16:47 | sean-k-mooney | mriedem: we get around that by a retry loop | |
| 19:16:50 | dansmith | mriedem: we retry waiting for ssh | |
| 19:16:57 | mriedem | ah yes | |
| 19:17:01 | dansmith | but that's a totally different problem than not having networking and DHCP ready during boot | |
| 19:17:03 | mriedem | so we hide the poo | |
| 19:17:13 | dansmith | it's totally legit to not expect the instance is ready when the state goes to active, | |
| 19:17:21 | dansmith | it's not legit for the instance to boot up and expect networking but have none | |
| 19:17:31 | sean-k-mooney | tempest waits for active then wiats for ping to work then ssh's i think | |
| 19:17:41 | dansmith | and then tries ssh a few times with long timeouts | |
| 19:17:43 | mriedem | sean-k-mooney: yes i forgot about the latter | |
| 19:17:58 | mnaser | i mean, dhcp could be slow too, so it doesnt matter if internet is wired up.. | |
| 19:18:02 | mnaser | start up of the OS could be slow | |
| 19:18:14 | mnaser | unless you setup something inside the OS that tells nova the vm is ready, we'll always have ACTIVE vms that arent accessible | |
| 19:18:22 | melwitt | yeah, +1 dansmith. could this screw boot up entirely? if DHCP not ready while it's coming up? or will it always recover | |
| 19:18:33 | sean-k-mooney | dansmith: active and ready in an ironic case is partcallarly different things. vms tend to be ready soon after they are active | |
| 19:18:38 | dansmith | melwitt: yes if networking isn't running | |
| 19:18:43 | melwitt | urgh | |
| 19:18:52 | dansmith | melwitt: some OSes may try DHCP briefly, and if it gets none, then it never retries | |
| 19:19:05 | dansmith | if you are a cloud image and don't know what kind of cloud you're on, | |
| 19:19:13 | melwitt | that would majorly suck to just have a junk instance if this is hit | |
| 19:19:26 | dansmith | you really have to just poke around until you figure it out because it could be a variety of things | |
| 19:19:31 | mnaser | ...but then again this is something that has been in nova for 3-4 releases, so a fix would be nice, but i dont think its a issue thats happening often enough? | |
| 19:19:36 | sean-k-mooney | melwitt: also oftend cloud images will wait for cloud init to finish before starting ssh so that can take a while too | |
| 19:19:45 | mnaser | considering the port is never really unplugged in ovs, most things are already wired up | |
| 19:19:48 | melwitt | I see | |
| 19:20:03 | dansmith | mnaser: yeah less of a concern on reboot for that reason | |
| 19:20:05 | sean-k-mooney | mnaser: on hard reboot it is unpluged | |
| 19:20:15 | sean-k-mooney | mnaser: on soft reboot it is not | |
| 19:20:26 | mnaser | sean-k-mooney: but the unplug in os_vif seems to be noop from what i see (or maybe im misunderstanding things) | |
| 19:20:36 | mnaser | sean-k-mooney: https://github.com/openstack/os-vif/blob/master/vif_plug_ovs/ovs.py#L266-L268 nothing happens here... | |
| 19:20:45 | sean-k-mooney | mnaser: that only because in that code path the plugin is done by libvirt | |
| 19:20:59 | mnaser | oh okay sorry i'm following you now | |
| 19:21:06 | mnaser | the actual unplug happens by libvirt which is then picked up by n-ovs-agent | |
| 19:21:34 | sean-k-mooney | yep when we do the domain destroy on hard reboot libvirt remove the interface form ovs and deletes teh tap | |
| 19:21:54 | mnaser | ok i understand now | |
| 19:21:56 | mnaser | thank youy | |
| 19:22:17 | cfriesen | my organization has a local patch to make plug_vifs() optionally wait until the vifs are actually plugged in order to ensure that things work reliably during a live migration...we added that a long time ago and I was wondering if it's still needed, but it sounds like it is. | |
| 19:22:44 | sean-k-mooney | cfriesen: when you say actully plugged what do you wait for | |
| 19:23:06 | dansmith | we can't know that the plumbing is done behind the neutron curtain without the event right? | |
| 19:23:39 | sean-k-mooney | cfriesen: if its external-ids:iface-status=active in the ovs db we hard code that here https://github.com/openstack/os-vif/blob/master/vif_plug_ovs/linux_net.py#L71 | |
| 19:24:35 | sean-k-mooney | dansmith: technically for ovs and ovs agents only we shoudl be able to use external-ids:iface-status=active but i dont know for odl | |
| 19:24:54 | dansmith | but that's a very specific case | |
| 19:24:55 | sean-k-mooney | dansmith: for lb or vpp or whatever else we have no way to tell | |
| 19:24:57 | dansmith | for that one type | |
| 19:24:59 | dansmith | yeah | |
| 19:25:36 | mnaser | mriedem: https://bugs.launchpad.net/nova/+bug/1755890 i tried to write a basic description and referenced this conversation but yeah feel free to update it with any lacking details | |
| 19:25:37 | openstack | Launchpad bug 1755890 in OpenStack Compute (nova) "Instances fail to hard reboot when using OpenDaylight" [Undecided,New] | |
| 19:25:38 | cfriesen | sean-k-mooney: actually, never mind. the only one with a useful query is for our custom networking thing, the others are assumed to always be up. My bad. | |
| 19:25:47 | melwitt | I have to run for now but I'm cool with reverting those patches if that's the best option we have. bbl | |
| 19:25:59 | arvindn05 | mriedem: jaypipes: updated the spec based on comments Patch set 9 should address the issues https://review.openstack.org/#/c/541507/9 | |
| 19:27:09 | sean-k-mooney | cfriesen: the dpdk based titainium server vswitch i assuem | |
| 19:27:17 | cfriesen | yep | |
| 19:28:07 | cfriesen | it'll query the vswitch API | |
| 19:29:03 | sean-k-mooney | cfriesen: ya the issue is that bar seeting external-ids:iface-status=active which is an ovs agent specif thing there is no way to tell for ovs in general. | |
| 19:31:51 | sean-k-mooney | when odl first intregrated support into the ml2 framework this gap was discussed back in icehouse but they had no way syncronis state in odl with neutron without a intoducing an sdn constrolller specific api, b having odl emit notifcation on the rabbitmq bus or be haveing the neutron server poll odl for state change | |
| 19:32:27 | sean-k-mooney | the chose to do non of the above an just report vif_plugged on port binding instead and its been that way ever since | |
| 19:32:59 | mnaser | sean-k-mooney: maybe this is a networking-odl bug? i know that right now it creates a websocket that listens for state updates | |
| 19:33:01 | mnaser | https://github.com/openstack/networking-odl/blob/master/networking_odl/ml2/port_status_update.py | |
| 19:33:28 | mnaser | https://bugs.launchpad.net/networking-odl/+bug/1686023 | |
| 19:33:29 | openstack | Launchpad bug 1686023 in networking-odl "networking-odl dynamic port status update full support missing" [Low,In progress] | |
| 19:33:32 | mriedem | jaypipes: replied in https://review.openstack.org/#/c/539605/ - need bauzas to probably elaborate at this point since i don't remember this being talked about at the ptg | |
| 19:33:42 | mnaser | i guess they support DOWN => ACTIVE but not ACTIVE=>DOWN | |
| 19:34:44 | mnaser | i wonder if there's an easy way to let neutron know that the port has gone down in there | |
| 19:34:45 | mnaser | and avoid the revert | |
| 19:34:48 | sean-k-mooney | mnaser: well i think its a ml2 framework feature request. e.g. allow agentless backend notify neutron of state changes | |
| 19:35:15 | mriedem | if we can blame this all on an incomplete neutron backend then that works for me | |
| 19:35:19 | sean-k-mooney | mnaser: that bug seams to focus on admin state which is different | |
| 19:35:19 | mnaser | lolol | |
| 19:36:01 | mriedem | https://review.openstack.org/#/c/465463/ | |
| 19:36:04 | mnaser | sean-k-mooney: https://github.com/openstack/networking-odl/blob/master/networking_odl/ml2/port_status_update.py#L91-L95 ever that little bit? the bug talks about admin state but it seems to get the actual port state | |
| 19:37:20 | sean-k-mooney | mnaser: yest that should adress this but it seams that odl is not detecting the removal of the port and readding it | |