| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-14 | |||
| 19:08:21 | mriedem | any places in tempest that fallout as a result could wait on the port being ACTIVE in addition to the server being ACTIVE | |
| 19:08:38 | mriedem | sean-k-mooney: calling activate? | |
| 19:08:48 | melwitt | yeah, what's that do? | |
| 19:09:07 | sean-k-mooney | in the multiple port binding spec there is a new api call to activate a binding as part of live migration | |
| 19:09:16 | mriedem | if it relies on the new porting binding api, that doesn't exist yet, and isn't backportable :) | |
| 19:09:33 | mriedem | sean-k-mooney: you just mean changing the binding status to 'active' right? | |
| 19:09:46 | mriedem | sean-k-mooney: for a hard-reboot with a single host binding, it would already be active | |
| 19:09:53 | sean-k-mooney | well my taught would be allow that to tell neutron to validate its wired up again because did something(reboot) you man not have seen | |
| 19:10:47 | melwitt | that sounds like what we want, but as mriedem said, not backportable | |
| 19:10:58 | mriedem | mnaser: other good news is ocata isn't going anywhere :) https://review.openstack.org/#/c/548916/ | |
| 19:11:02 | sean-k-mooney | mriedem: yes it would. but since that api is not in a release yet we could ammend it to say calling activate on an active binding forces it to wire it up again or at least check its in the active state | |
| 19:11:28 | mriedem | sean-k-mooney: that's pretty hacky, | |
| 19:11:37 | melwitt | and validating it would cause an event to be sent? | |
| 19:11:39 | mriedem | i think you'd have to deactivate the binding, and then activate it | |
| 19:12:11 | mriedem | /ports/{id}/binding/{host}/action :) | |
| 19:12:12 | sean-k-mooney | melwitt: validating would sent the event if it actully changed something | |
| 19:12:13 | mriedem | i can see it now | |
| 19:12:33 | mriedem | so for present day options, | |
| 19:12:33 | melwitt | sean-k-mooney: yeah, I guess that seems similar to just unbinding it and binding it again | |
| 19:12:44 | mriedem | i think we're talking revert right? | |
| 19:12:48 | sean-k-mooney | melwitt: yep it is | |
| 19:12:50 | mriedem | back through queens and pike | |
| 19:13:01 | melwitt | I guess so. doesn't seem like we have any choice | |
| 19:13:10 | sean-k-mooney | mriedem: revert waiting. yes i think so | |
| 19:13:33 | mriedem | dansmith: jaypipes: how are you feeling about a revert | |
| 19:13:37 | mriedem | i don't see other very good options | |
| 19:13:44 | dansmith | yeah I mean, are there other options? | |
| 19:13:54 | mriedem | not outside of changing the port binding on reboot | |
| 19:13:55 | sean-k-mooney | i think we can/shoudl fix it for rocky so we can reenable waiting | |
| 19:13:57 | mriedem | which is a bigger change | |
| 19:14:00 | melwitt | unbinding the port and binding it again | |
| 19:14:30 | mriedem | sean-k-mooney: i think that would also require talking to the neutron team about whether or not that would do what we need | |
| 19:14:40 | mriedem | i can only talk to miguel once per day, he said | |
| 19:14:46 | melwitt | haha | |
| 19:14:56 | cfriesen | are there any races that come from not waiting for vifs to actually be plugged? | |
| 19:15:05 | mriedem | mnaser: is there a bug report for the ODL issue? | |
| 19:15:08 | dansmith | yes | |
| 19:15:08 | mriedem | cfriesen: yes | |
| 19:15:09 | sean-k-mooney | "in general" un binding and binding the port "should" be safe but it can fail | |
| 19:15:16 | mriedem | tempest races assuming it can ssh into the guest | |
| 19:15:18 | melwitt | cfriesen: instance can go to ACTIVE state without having networking yet | |
| 19:15:30 | mnaser | mriedem: in ODL world yes, but not in nova world, but i can make one | |
| 19:15:37 | mriedem | mnaser: yes please for tracking | |
| 19:15:43 | mnaser | ok | |
| 19:15:47 | melwitt | well, for ssh I know from using devstack that the instance says ACTIVE before ssh daemon is ready | |
| 19:15:52 | sean-k-mooney | cfriesen: odl has no mechanisium to notify nova today that they have been plugged | |
| 19:15:54 | melwitt | I thought | |
| 19:16:01 | sean-k-mooney | * notify neutron | |
| 19:16:19 | mriedem | melwitt: if that were true, | |
| 19:16:27 | mriedem | our CI would shit its pants daily | |
| 19:16:47 | sean-k-mooney | mriedem: we get around that by a retry loop | |
| 19:16:50 | dansmith | mriedem: we retry waiting for ssh | |
| 19:16:57 | mriedem | ah yes | |
| 19:17:01 | dansmith | but that's a totally different problem than not having networking and DHCP ready during boot | |
| 19:17:03 | mriedem | so we hide the poo | |
| 19:17:13 | dansmith | it's totally legit to not expect the instance is ready when the state goes to active, | |
| 19:17:21 | dansmith | it's not legit for the instance to boot up and expect networking but have none | |
| 19:17:31 | sean-k-mooney | tempest waits for active then wiats for ping to work then ssh's i think | |
| 19:17:41 | dansmith | and then tries ssh a few times with long timeouts | |
| 19:17:43 | mriedem | sean-k-mooney: yes i forgot about the latter | |
| 19:17:58 | mnaser | i mean, dhcp could be slow too, so it doesnt matter if internet is wired up.. | |
| 19:18:02 | mnaser | start up of the OS could be slow | |
| 19:18:14 | mnaser | unless you setup something inside the OS that tells nova the vm is ready, we'll always have ACTIVE vms that arent accessible | |
| 19:18:22 | melwitt | yeah, +1 dansmith. could this screw boot up entirely? if DHCP not ready while it's coming up? or will it always recover | |
| 19:18:33 | sean-k-mooney | dansmith: active and ready in an ironic case is partcallarly different things. vms tend to be ready soon after they are active | |
| 19:18:38 | dansmith | melwitt: yes if networking isn't running | |
| 19:18:43 | melwitt | urgh | |
| 19:18:52 | dansmith | melwitt: some OSes may try DHCP briefly, and if it gets none, then it never retries | |
| 19:19:05 | dansmith | if you are a cloud image and don't know what kind of cloud you're on, | |
| 19:19:13 | melwitt | that would majorly suck to just have a junk instance if this is hit | |
| 19:19:26 | dansmith | you really have to just poke around until you figure it out because it could be a variety of things | |
| 19:19:31 | mnaser | ...but then again this is something that has been in nova for 3-4 releases, so a fix would be nice, but i dont think its a issue thats happening often enough? | |
| 19:19:36 | sean-k-mooney | melwitt: also oftend cloud images will wait for cloud init to finish before starting ssh so that can take a while too | |
| 19:19:45 | mnaser | considering the port is never really unplugged in ovs, most things are already wired up | |
| 19:19:48 | melwitt | I see | |
| 19:20:03 | dansmith | mnaser: yeah less of a concern on reboot for that reason | |
| 19:20:05 | sean-k-mooney | mnaser: on hard reboot it is unpluged | |
| 19:20:15 | sean-k-mooney | mnaser: on soft reboot it is not | |
| 19:20:26 | mnaser | sean-k-mooney: but the unplug in os_vif seems to be noop from what i see (or maybe im misunderstanding things) | |
| 19:20:36 | mnaser | sean-k-mooney: https://github.com/openstack/os-vif/blob/master/vif_plug_ovs/ovs.py#L266-L268 nothing happens here... | |
| 19:20:45 | sean-k-mooney | mnaser: that only because in that code path the plugin is done by libvirt | |
| 19:20:59 | mnaser | oh okay sorry i'm following you now | |
| 19:21:06 | mnaser | the actual unplug happens by libvirt which is then picked up by n-ovs-agent | |
| 19:21:34 | sean-k-mooney | yep when we do the domain destroy on hard reboot libvirt remove the interface form ovs and deletes teh tap | |
| 19:21:54 | mnaser | ok i understand now | |
| 19:21:56 | mnaser | thank youy | |
| 19:22:17 | cfriesen | my organization has a local patch to make plug_vifs() optionally wait until the vifs are actually plugged in order to ensure that things work reliably during a live migration...we added that a long time ago and I was wondering if it's still needed, but it sounds like it is. | |
| 19:22:44 | sean-k-mooney | cfriesen: when you say actully plugged what do you wait for | |
| 19:23:06 | dansmith | we can't know that the plumbing is done behind the neutron curtain without the event right? | |
| 19:23:39 | sean-k-mooney | cfriesen: if its external-ids:iface-status=active in the ovs db we hard code that here https://github.com/openstack/os-vif/blob/master/vif_plug_ovs/linux_net.py#L71 | |
| 19:24:35 | sean-k-mooney | dansmith: technically for ovs and ovs agents only we shoudl be able to use external-ids:iface-status=active but i dont know for odl | |
| 19:24:54 | dansmith | but that's a very specific case | |
| 19:24:55 | sean-k-mooney | dansmith: for lb or vpp or whatever else we have no way to tell | |
| 19:24:57 | dansmith | for that one type | |
| 19:24:59 | dansmith | yeah | |
| 19:25:36 | mnaser | mriedem: https://bugs.launchpad.net/nova/+bug/1755890 i tried to write a basic description and referenced this conversation but yeah feel free to update it with any lacking details | |
| 19:25:37 | openstack | Launchpad bug 1755890 in OpenStack Compute (nova) "Instances fail to hard reboot when using OpenDaylight" [Undecided,New] | |
| 19:25:38 | cfriesen | sean-k-mooney: actually, never mind. the only one with a useful query is for our custom networking thing, the others are assumed to always be up. My bad. | |
| 19:25:47 | melwitt | I have to run for now but I'm cool with reverting those patches if that's the best option we have. bbl | |
| 19:25:59 | arvindn05 | mriedem: jaypipes: updated the spec based on comments Patch set 9 should address the issues https://review.openstack.org/#/c/541507/9 | |