| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-14 | |||
| 19:02:29 | mnaser | (i can imagine that you'll have other closed source sdns showing up soon too) | |
| 19:02:30 | mriedem | sean-k-mooney: and that would be a bigger change, to re-bind the port | |
| 19:02:57 | sean-k-mooney | mriedem: yes it would. we could explore that at some point but prehaps not now | |
| 19:03:16 | melwitt | yeah. I was wondering, is there a call to neutron we could make that would be mostly a no-op, not actually rebind it but update it to the same binding profile or something like that? | |
| 19:03:35 | melwitt | I don't know if neutron would emit an event if nothing changed | |
| 19:03:51 | mnaser | melwitt: it would not, i looked into the logic, its a db hook that checks if states changed | |
| 19:03:54 | sean-k-mooney | as context the odl ml2 works the way it does today because odl does not have acess to rabitmq to be able to notify nova/neutron when its finished wireing things up | |
| 19:03:59 | melwitt | mnaser: ah, rats | |
| 19:04:03 | mnaser | https://github.com/openstack/neutron/blob/master/neutron/db/db_base_plugin_v2.py#L155-L169 | |
| 19:04:28 | mnaser | https://github.com/openstack/neutron/blob/master/neutron/notifiers/nova.py | |
| 19:04:31 | mnaser | and it explicitely ignores noops | |
| 19:05:03 | mriedem | nova would have to unbind/bind or change the state of the binding (i think) | |
| 19:05:08 | mriedem | and that all seems scary and hacky | |
| 19:05:18 | mnaser | mriedem: and not something you'd want to backport all the way to stable/ocata either | |
| 19:05:26 | melwitt | yeah, I don't want to do that | |
| 19:05:38 | sean-k-mooney | mriedem: ya we would have to set the host_id in the binding profile to None then back to the hostname to get it to work | |
| 19:06:24 | mriedem | well the good news is the original change didn't make it to newton | |
| 19:07:01 | sean-k-mooney | mriedem: maybe we can use the multiple port bindings neutron are adding to adress this in the future | |
| 19:07:05 | melwitt | I was thinking if we could not actually change it, update it to the same, and it would emit a notification, that might be cool. but I definitely don't want to unbind and rebind it just for this. guess it all depends on what's the impact of not waiting for the event. instance says ACTIVE when it might not actually have networking yet? | |
| 19:07:24 | mriedem | melwitt: yes | |
| 19:07:57 | mriedem | doing that for hard reboot is less shitty than initial create, at least from our CI system | |
| 19:08:13 | sean-k-mooney | mriedem: how would you feel about nova calling activate on the port binding again on a hard reboot? | |
| 19:08:15 | openstackgerrit | Arvind Nadendla proposed openstack/nova-specs master: Support traits in Glance https://review.openstack.org/541507 | |
| 19:08:21 | mriedem | any places in tempest that fallout as a result could wait on the port being ACTIVE in addition to the server being ACTIVE | |
| 19:08:38 | mriedem | sean-k-mooney: calling activate? | |
| 19:08:48 | melwitt | yeah, what's that do? | |
| 19:09:07 | sean-k-mooney | in the multiple port binding spec there is a new api call to activate a binding as part of live migration | |
| 19:09:16 | mriedem | if it relies on the new porting binding api, that doesn't exist yet, and isn't backportable :) | |
| 19:09:33 | mriedem | sean-k-mooney: you just mean changing the binding status to 'active' right? | |
| 19:09:46 | mriedem | sean-k-mooney: for a hard-reboot with a single host binding, it would already be active | |
| 19:09:53 | sean-k-mooney | well my taught would be allow that to tell neutron to validate its wired up again because did something(reboot) you man not have seen | |
| 19:10:47 | melwitt | that sounds like what we want, but as mriedem said, not backportable | |
| 19:10:58 | mriedem | mnaser: other good news is ocata isn't going anywhere :) https://review.openstack.org/#/c/548916/ | |
| 19:11:02 | sean-k-mooney | mriedem: yes it would. but since that api is not in a release yet we could ammend it to say calling activate on an active binding forces it to wire it up again or at least check its in the active state | |
| 19:11:28 | mriedem | sean-k-mooney: that's pretty hacky, | |
| 19:11:37 | melwitt | and validating it would cause an event to be sent? | |
| 19:11:39 | mriedem | i think you'd have to deactivate the binding, and then activate it | |
| 19:12:11 | mriedem | /ports/{id}/binding/{host}/action :) | |
| 19:12:12 | sean-k-mooney | melwitt: validating would sent the event if it actully changed something | |
| 19:12:13 | mriedem | i can see it now | |
| 19:12:33 | mriedem | so for present day options, | |
| 19:12:33 | melwitt | sean-k-mooney: yeah, I guess that seems similar to just unbinding it and binding it again | |
| 19:12:44 | mriedem | i think we're talking revert right? | |
| 19:12:48 | sean-k-mooney | melwitt: yep it is | |
| 19:12:50 | mriedem | back through queens and pike | |
| 19:13:01 | melwitt | I guess so. doesn't seem like we have any choice | |
| 19:13:10 | sean-k-mooney | mriedem: revert waiting. yes i think so | |
| 19:13:33 | mriedem | dansmith: jaypipes: how are you feeling about a revert | |
| 19:13:37 | mriedem | i don't see other very good options | |
| 19:13:44 | dansmith | yeah I mean, are there other options? | |
| 19:13:54 | mriedem | not outside of changing the port binding on reboot | |
| 19:13:55 | sean-k-mooney | i think we can/shoudl fix it for rocky so we can reenable waiting | |
| 19:13:57 | mriedem | which is a bigger change | |
| 19:14:00 | melwitt | unbinding the port and binding it again | |
| 19:14:30 | mriedem | sean-k-mooney: i think that would also require talking to the neutron team about whether or not that would do what we need | |
| 19:14:40 | mriedem | i can only talk to miguel once per day, he said | |
| 19:14:46 | melwitt | haha | |
| 19:14:56 | cfriesen | are there any races that come from not waiting for vifs to actually be plugged? | |
| 19:15:05 | mriedem | mnaser: is there a bug report for the ODL issue? | |
| 19:15:08 | dansmith | yes | |
| 19:15:08 | mriedem | cfriesen: yes | |
| 19:15:09 | sean-k-mooney | "in general" un binding and binding the port "should" be safe but it can fail | |
| 19:15:16 | mriedem | tempest races assuming it can ssh into the guest | |
| 19:15:18 | melwitt | cfriesen: instance can go to ACTIVE state without having networking yet | |
| 19:15:30 | mnaser | mriedem: in ODL world yes, but not in nova world, but i can make one | |
| 19:15:37 | mriedem | mnaser: yes please for tracking | |
| 19:15:43 | mnaser | ok | |
| 19:15:47 | melwitt | well, for ssh I know from using devstack that the instance says ACTIVE before ssh daemon is ready | |
| 19:15:52 | sean-k-mooney | cfriesen: odl has no mechanisium to notify nova today that they have been plugged | |
| 19:15:54 | melwitt | I thought | |
| 19:16:01 | sean-k-mooney | * notify neutron | |
| 19:16:19 | mriedem | melwitt: if that were true, | |
| 19:16:27 | mriedem | our CI would shit its pants daily | |
| 19:16:47 | sean-k-mooney | mriedem: we get around that by a retry loop | |
| 19:16:50 | dansmith | mriedem: we retry waiting for ssh | |
| 19:16:57 | mriedem | ah yes | |
| 19:17:01 | dansmith | but that's a totally different problem than not having networking and DHCP ready during boot | |
| 19:17:03 | mriedem | so we hide the poo | |
| 19:17:13 | dansmith | it's totally legit to not expect the instance is ready when the state goes to active, | |
| 19:17:21 | dansmith | it's not legit for the instance to boot up and expect networking but have none | |
| 19:17:31 | sean-k-mooney | tempest waits for active then wiats for ping to work then ssh's i think | |
| 19:17:41 | dansmith | and then tries ssh a few times with long timeouts | |
| 19:17:43 | mriedem | sean-k-mooney: yes i forgot about the latter | |
| 19:17:58 | mnaser | i mean, dhcp could be slow too, so it doesnt matter if internet is wired up.. | |
| 19:18:02 | mnaser | start up of the OS could be slow | |
| 19:18:14 | mnaser | unless you setup something inside the OS that tells nova the vm is ready, we'll always have ACTIVE vms that arent accessible | |
| 19:18:22 | melwitt | yeah, +1 dansmith. could this screw boot up entirely? if DHCP not ready while it's coming up? or will it always recover | |
| 19:18:33 | sean-k-mooney | dansmith: active and ready in an ironic case is partcallarly different things. vms tend to be ready soon after they are active | |
| 19:18:38 | dansmith | melwitt: yes if networking isn't running | |
| 19:18:43 | melwitt | urgh | |
| 19:18:52 | dansmith | melwitt: some OSes may try DHCP briefly, and if it gets none, then it never retries | |
| 19:19:05 | dansmith | if you are a cloud image and don't know what kind of cloud you're on, | |
| 19:19:13 | melwitt | that would majorly suck to just have a junk instance if this is hit | |
| 19:19:26 | dansmith | you really have to just poke around until you figure it out because it could be a variety of things | |
| 19:19:31 | mnaser | ...but then again this is something that has been in nova for 3-4 releases, so a fix would be nice, but i dont think its a issue thats happening often enough? | |
| 19:19:36 | sean-k-mooney | melwitt: also oftend cloud images will wait for cloud init to finish before starting ssh so that can take a while too | |
| 19:19:45 | mnaser | considering the port is never really unplugged in ovs, most things are already wired up | |
| 19:19:48 | melwitt | I see | |
| 19:20:03 | dansmith | mnaser: yeah less of a concern on reboot for that reason | |
| 19:20:05 | sean-k-mooney | mnaser: on hard reboot it is unpluged | |