| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-14 | |||
| 19:18:14 | mnaser | unless you setup something inside the OS that tells nova the vm is ready, we'll always have ACTIVE vms that arent accessible | |
| 19:18:22 | melwitt | yeah, +1 dansmith. could this screw boot up entirely? if DHCP not ready while it's coming up? or will it always recover | |
| 19:18:33 | sean-k-mooney | dansmith: active and ready in an ironic case is partcallarly different things. vms tend to be ready soon after they are active | |
| 19:18:38 | dansmith | melwitt: yes if networking isn't running | |
| 19:18:43 | melwitt | urgh | |
| 19:18:52 | dansmith | melwitt: some OSes may try DHCP briefly, and if it gets none, then it never retries | |
| 19:19:05 | dansmith | if you are a cloud image and don't know what kind of cloud you're on, | |
| 19:19:13 | melwitt | that would majorly suck to just have a junk instance if this is hit | |
| 19:19:26 | dansmith | you really have to just poke around until you figure it out because it could be a variety of things | |
| 19:19:31 | mnaser | ...but then again this is something that has been in nova for 3-4 releases, so a fix would be nice, but i dont think its a issue thats happening often enough? | |
| 19:19:36 | sean-k-mooney | melwitt: also oftend cloud images will wait for cloud init to finish before starting ssh so that can take a while too | |
| 19:19:45 | mnaser | considering the port is never really unplugged in ovs, most things are already wired up | |
| 19:19:48 | melwitt | I see | |
| 19:20:03 | dansmith | mnaser: yeah less of a concern on reboot for that reason | |
| 19:20:05 | sean-k-mooney | mnaser: on hard reboot it is unpluged | |
| 19:20:15 | sean-k-mooney | mnaser: on soft reboot it is not | |
| 19:20:26 | mnaser | sean-k-mooney: but the unplug in os_vif seems to be noop from what i see (or maybe im misunderstanding things) | |
| 19:20:36 | mnaser | sean-k-mooney: https://github.com/openstack/os-vif/blob/master/vif_plug_ovs/ovs.py#L266-L268 nothing happens here... | |
| 19:20:45 | sean-k-mooney | mnaser: that only because in that code path the plugin is done by libvirt | |
| 19:20:59 | mnaser | oh okay sorry i'm following you now | |
| 19:21:06 | mnaser | the actual unplug happens by libvirt which is then picked up by n-ovs-agent | |
| 19:21:34 | sean-k-mooney | yep when we do the domain destroy on hard reboot libvirt remove the interface form ovs and deletes teh tap | |
| 19:21:54 | mnaser | ok i understand now | |
| 19:21:56 | mnaser | thank youy | |
| 19:22:17 | cfriesen | my organization has a local patch to make plug_vifs() optionally wait until the vifs are actually plugged in order to ensure that things work reliably during a live migration...we added that a long time ago and I was wondering if it's still needed, but it sounds like it is. | |
| 19:22:44 | sean-k-mooney | cfriesen: when you say actully plugged what do you wait for | |
| 19:23:06 | dansmith | we can't know that the plumbing is done behind the neutron curtain without the event right? | |
| 19:23:39 | sean-k-mooney | cfriesen: if its external-ids:iface-status=active in the ovs db we hard code that here https://github.com/openstack/os-vif/blob/master/vif_plug_ovs/linux_net.py#L71 | |
| 19:24:35 | sean-k-mooney | dansmith: technically for ovs and ovs agents only we shoudl be able to use external-ids:iface-status=active but i dont know for odl | |
| 19:24:54 | dansmith | but that's a very specific case | |
| 19:24:55 | sean-k-mooney | dansmith: for lb or vpp or whatever else we have no way to tell | |
| 19:24:57 | dansmith | for that one type | |
| 19:24:59 | dansmith | yeah | |
| 19:25:36 | mnaser | mriedem: https://bugs.launchpad.net/nova/+bug/1755890 i tried to write a basic description and referenced this conversation but yeah feel free to update it with any lacking details | |
| 19:25:37 | openstack | Launchpad bug 1755890 in OpenStack Compute (nova) "Instances fail to hard reboot when using OpenDaylight" [Undecided,New] | |
| 19:25:38 | cfriesen | sean-k-mooney: actually, never mind. the only one with a useful query is for our custom networking thing, the others are assumed to always be up. My bad. | |
| 19:25:47 | melwitt | I have to run for now but I'm cool with reverting those patches if that's the best option we have. bbl | |
| 19:25:59 | arvindn05 | mriedem: jaypipes: updated the spec based on comments Patch set 9 should address the issues https://review.openstack.org/#/c/541507/9 | |
| 19:27:09 | sean-k-mooney | cfriesen: the dpdk based titainium server vswitch i assuem | |
| 19:27:17 | cfriesen | yep | |
| 19:28:07 | cfriesen | it'll query the vswitch API | |
| 19:29:03 | sean-k-mooney | cfriesen: ya the issue is that bar seeting external-ids:iface-status=active which is an ovs agent specif thing there is no way to tell for ovs in general. | |
| 19:31:51 | sean-k-mooney | when odl first intregrated support into the ml2 framework this gap was discussed back in icehouse but they had no way syncronis state in odl with neutron without a intoducing an sdn constrolller specific api, b having odl emit notifcation on the rabbitmq bus or be haveing the neutron server poll odl for state change | |
| 19:32:27 | sean-k-mooney | the chose to do non of the above an just report vif_plugged on port binding instead and its been that way ever since | |
| 19:32:59 | mnaser | sean-k-mooney: maybe this is a networking-odl bug? i know that right now it creates a websocket that listens for state updates | |
| 19:33:01 | mnaser | https://github.com/openstack/networking-odl/blob/master/networking_odl/ml2/port_status_update.py | |
| 19:33:28 | mnaser | https://bugs.launchpad.net/networking-odl/+bug/1686023 | |
| 19:33:29 | openstack | Launchpad bug 1686023 in networking-odl "networking-odl dynamic port status update full support missing" [Low,In progress] | |
| 19:33:32 | mriedem | jaypipes: replied in https://review.openstack.org/#/c/539605/ - need bauzas to probably elaborate at this point since i don't remember this being talked about at the ptg | |
| 19:33:42 | mnaser | i guess they support DOWN => ACTIVE but not ACTIVE=>DOWN | |
| 19:34:44 | mnaser | i wonder if there's an easy way to let neutron know that the port has gone down in there | |
| 19:34:45 | mnaser | and avoid the revert | |
| 19:34:48 | sean-k-mooney | mnaser: well i think its a ml2 framework feature request. e.g. allow agentless backend notify neutron of state changes | |
| 19:35:15 | mriedem | if we can blame this all on an incomplete neutron backend then that works for me | |
| 19:35:19 | mnaser | lolol | |
| 19:35:19 | sean-k-mooney | mnaser: that bug seams to focus on admin state which is different | |
| 19:36:01 | mriedem | https://review.openstack.org/#/c/465463/ | |
| 19:36:04 | mnaser | sean-k-mooney: https://github.com/openstack/networking-odl/blob/master/networking_odl/ml2/port_status_update.py#L91-L95 ever that little bit? the bug talks about admin state but it seems to get the actual port state | |
| 19:37:20 | sean-k-mooney | mnaser: yest that should adress this but it seams that odl is not detecting the removal of the port and readding it | |
| 19:38:22 | sean-k-mooney | mnaser: perhaps the networking-odl ml2 driver is just not sending the notification to nova when it recives the notification from odl | |
| 19:38:41 | mriedem | shouldn't the driver go through the normal notificatoin code in neutron? | |
| 19:39:05 | mnaser | sean-k-mooney: its certainly not sending it when its being unplugged, but it looks like the code to watch for state doesnt even update it to 'down' or 'unplugged' or whatever state it should be in neutron | |
| 19:39:17 | sean-k-mooney | mriedem: that notification code is triggered by port status updates on the rpc bus | |
| 19:39:28 | sean-k-mooney | mriedem: these update from odl are from the websocket | |
| 19:41:19 | sean-k-mooney | mriedem: since we are not storing the state changes from https://github.com/openstack/networking-odl/blob/master/networking_odl/ml2/port_status_update.py#L91-L95 in the db i dont think https://github.com/openstack/neutron/blob/master/neutron/db/db_base_plugin_v2.py#L155-L169 will be invoked | |
| 19:41:52 | mnaser | sean-k-mooney: is there an example of a plugin that is storing/updating state changes in db which might be good to reference | |
| 19:42:58 | sean-k-mooney | maybe look at the ovs agent code. | |
| 19:43:05 | mnaser | https://github.com/openstack/networking-ovn/blob/master/networking_ovn/ml2/mech_driver.py#L713-L729 | |
| 19:43:13 | mnaser | i guess i can use that as a reference and try submitting something to networking-odl | |
| 19:44:27 | sean-k-mooney | i dont think we actully need to store it in the db. we just need to send the notification to nova | |
| 19:45:07 | mnaser | i guess one thing though is talking about the revert is... if this is going to affect other drivers too | |
| 19:45:16 | mnaser | aka we fix odl but find 4 other vif types that are affected by that issue | |
| 19:46:17 | sean-k-mooney | mnaser: we are already special casing lb and we need to skip waithing i think for sriov too so basically we can only wait for ovs with agents today | |
| 19:46:54 | sean-k-mooney | maybe we can wait for ovn? but not sure. | |
| 19:49:47 | mriedem | unless the port profile/details add some specific flag that says we can expect events on replug, i don't think it's safe to just have the blanket wait that we have today | |
| 19:50:12 | openstackgerrit | Ed Leafe proposed openstack/nova master: Add 'member_of' param to GET /allocation_candidates https://review.openstack.org/552098 | |
| 19:50:15 | edleafe | jaypipes: cdent: ^^ Addressed your comments | |
| 19:50:42 | sean-k-mooney | mriedem: that is true. i think this is something that first needs to made a requirement for all neutron backend before nova can start to depend on it. | |
| 19:51:13 | sean-k-mooney | mriedem: one option is add a nova config opetion to the compute node? we wait there correct? but not ideal | |
| 19:52:42 | mriedem | it's an option yes but not a good one imo | |
| 19:52:53 | mriedem | we already have vif_plugging_timeout and vif_plugging_is_fatal | |
| 19:53:11 | mriedem | we'd have to add a vif_plugging_reboot_if_hard_and_libvirt_and_thursday_timeout | |
| 19:54:46 | sean-k-mooney | hehe well it could be vif_wait_for_network but ya only ovs + reference agents could set it to true | |
| 19:55:11 | sean-k-mooney | it would have to default to false and proably should be deprecated from the start | |
| 19:56:01 | mnaser | i guess a revert seems like the cleanest possible choice? :x | |
| 19:56:10 | jaypipes | edleafe: +2 from me. | |
| 19:56:10 | sean-k-mooney | mnaser: ya | |
| 19:56:30 | mnaser | i'm not sure how a revert would happen because this seems to be 2 patches. a patch that reverts both referencing both commits? | |
| 19:56:36 | mnaser | unless someone else volunteers to do this | |
| 19:57:34 | sean-k-mooney | what were the 2 commits again. it will be next week if i do it at the earliest. | |
| 19:57:56 | mriedem | i think it's just this one https://review.openstack.org/#/q/Ib08afad3822f2ca95cfeea18d7f4fc4cb407b4d6 | |
| 19:58:21 | mriedem | that's the one that adding the wait back in | |
| 19:58:23 | mnaser | what about https://review.openstack.org/#/q/Ib0cf5d55750f13d0499a570f14024dca551ed4d4 ? | |
| 19:58:39 | mriedem | that's what you want to happen, not wait | |
| 19:58:56 | mriedem | there were 3 changes: | |
| 19:59:21 | mriedem | 1. full hard reboot blasting everything away and re-plug; that caused issues because we were waiting for something that wouldn't happen | |
| 19:59:29 | mriedem | 2. to workaround ^, we stopped waiting | |
| 19:59:46 | mriedem | 3. because not waiting when it will work for some (ovs but not lb), the final patch was added which added the bridge conditional | |
| 19:59:53 | mriedem | but as we've now seen, that doesn't work for ovs (odl) | |
| 20:00:04 | mriedem | so revert #3 to go back to not waiting | |