| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-11-15 | |||
| 22:59:33 | mriedem | what do you actually need cleaned up before the reschedule? | |
| 22:59:48 | mriedem | the ports should not be bound to that host at that point | |
| 22:59:56 | eandersson | Well the problem we are hitting is that each time it gets re-scheduled, a new ip is assigned | |
| 23:00:03 | mriedem | is a new port created? | |
| 23:00:08 | eandersson | Yea | |
| 23:00:15 | mriedem | ok, that's a problem, and i think an old one | |
| 23:00:29 | mriedem | i'm sure i could find some bugs if i dug for a bit | |
| 23:00:49 | eandersson | I am reproducing it with this http://paste.openstack.org/show/626450/ | |
| 23:00:55 | openstack | Launchpad bug 1703540 in OpenStack Compute (nova) "Reschedule with libvirt exception leaves dangling neutron ports" [High,Triaged] | |
| 23:00:55 | mriedem | https://bugs.launchpad.net/nova/+bug/1703540 | |
| 23:01:08 | mriedem | https://bugs.launchpad.net/nova/+bug/1531022 | |
| 23:01:09 | openstack | Launchpad bug 1531022 in OpenStack Compute (nova) "libvirt driver doesn't cleanup the tap interface on vm re-schedule" [Low,Confirmed] | |
| 23:01:24 | eandersson | oh not sure why I didn't find that one | |
| 23:01:59 | mriedem | yeah see comment #3 on https://bugs.launchpad.net/nova/+bug/1703540 | |
| 23:02:00 | openstack | Launchpad bug 1703540 in OpenStack Compute (nova) "Reschedule with libvirt exception leaves dangling neutron ports" [High,Triaged] | |
| 23:02:23 | eandersson | Yep - perfect | |
| 23:02:31 | eandersson | that is the exact same conclusion I came to | |
| 23:02:37 | mriedem | it would be as easy as calling cleanup_instance_network_on_host to remove any ports we created | |
| 23:03:03 | mriedem | however, doing that for neutron would basically be the same as for _cleanup_allocated_networks i think | |
| 23:03:15 | mriedem | because nova has to be smart about not deleting ports that the user supplied vs ones that nova created | |
| 23:03:44 | eandersson | Is there a reason why we can't just call _cleanup_allocated_networks ? | |
| 23:03:53 | eandersson | I tested that out in my lab and it worked 100% of the time | |
| 23:04:38 | eandersson | but obviously my test case is rather limited | |
| 23:05:16 | mriedem | honestly i'm not sure, and i worry that doing that will break something else, which is probably why we've never just done that | |
| 23:05:24 | mriedem | there was this patch for a related bug long ago https://review.openstack.org/#/c/367316/ | |
| 23:06:37 | mriedem | oh that was a backport | |
| 23:07:11 | mriedem | ok so https://review.openstack.org/#/c/243477/ is a different fix, | |
| 23:07:29 | mriedem | ^ is if all reschedules fail and we've run out of retries i think, so the instance is set to ERROR | |
| 23:07:32 | mriedem | which is different than, | |
| 23:07:37 | mriedem | host A fails, host B works | |
| 23:07:41 | mriedem | you'd have 2 ports | |
| 23:07:48 | openstackgerrit | Michael Still proposed openstack/nova master: Convert ext filesystem resizes to privsep. https://review.openstack.org/517516 | |
| 23:07:49 | openstackgerrit | Michael Still proposed openstack/nova master: Start moving users of parted to privsep. https://review.openstack.org/519011 | |
| 23:07:49 | openstackgerrit | Michael Still proposed openstack/nova master: Move flushing block devices to privsep. https://review.openstack.org/519010 | |
| 23:07:50 | openstackgerrit | Michael Still proposed openstack/nova master: Convert users of tune2fs to privsep. https://review.openstack.org/519484 | |
| 23:07:50 | openstackgerrit | Michael Still proposed openstack/nova master: Move remaining uses of parted to privsep. https://review.openstack.org/519483 | |
| 23:08:20 | eandersson | I see | |
| 23:10:53 | mriedem | i'm also not entirely sure if _cleanup_allocated_networks will do the entire job since it doesn't explicitly unplug vifs via the virt driver, | |
| 23:11:08 | mriedem | i'm not sure if deleting the port will take care of that out of band? | |
| 23:11:28 | openstack | Launchpad bug 1531022 in OpenStack Compute (nova) "libvirt driver doesn't cleanup the tap interface on vm re-schedule" [Low,Confirmed] | |
| 23:11:28 | mriedem | https://bugs.launchpad.net/nova/+bug/1531022/comments/2 | |
| 23:12:21 | efried | jaypipes More sanity check: In order to pass the entire provider tree to update_inventory, I'm going to need to *get* the entire tree. Which entails a new method in report client that calls GET with the new ?tree= deal, then builds up the ProviderTree object accordingly. And then actually returns that guy, which is currently a private _provider_tree attribute and not actually returned anywhere yet. | |
| 23:12:27 | eandersson | I don't see the tap at least on the compute I tested it on | |
| 23:12:52 | eandersson | but this is mitaka, who knows how it would work on other versions of Openstack | |
| 23:13:08 | mriedem | well, does it maybe depend on the type of interface? | |
| 23:13:12 | mriedem | vif type i mean | |
| 23:13:23 | efried | jaypipes Then when I get it back from update_inventory, the driver may have added children and whatnot, so I'm going to have to walk the tree to rearrange the placement db accordingly. | |
| 23:14:01 | mriedem | eandersson: here is another one https://review.openstack.org/#/c/335788 | |
| 23:14:13 | efried | jaypipes And also call _normalize_inventory_from_cn_obj on every provider in the tree, in case they decided to shove their VCPU/MEMORY_MB/DISK_GB in children instead of leaving them in the root. | |
| 23:14:52 | mriedem | "If I understand the code correctly, the intention of not deleting the port when rescheduling is that it could be reused on the second compute node. But that reuse does not seem to happen, instead nova allocates another port, leaving the first one pending." | |
| 23:15:11 | mriedem | eandersson: it could also totally be some legacy nova-network thing | |
| 23:15:20 | efried | jaypipes That first thing sort of a tree-ified version of _ensure_resource_provider | |
| 23:15:28 | mriedem | like maybe we didn't deallocate networks on reschedule if you were using nova-net? i'm not sure | |
| 23:17:32 | eandersson | Yea - I have done a lot of testing, but I am hesitant to change this without knowing all potential side-effects (even if only for internal usage) | |
| 23:17:45 | eandersson | but it being a legacy thing would make a lot of sense | |
| 23:20:50 | openstack | Launchpad bug 1597596 in OpenStack Compute (nova) "network not always cleaned up when spawning VMs" [Medium,Confirmed] | |
| 23:20:50 | mriedem | eandersson: ok i'm going to duplicate some of the bugs to just this one https://bugs.launchpad.net/nova/+bug/1597596 | |
| 23:20:56 | mriedem | which had a patch and did a lot of the same investigation | |
| 23:24:55 | mriedem | eandersson: so https://review.openstack.org/#/c/335788/ isn't going to fly, because it's totally relying on 2 out of tree scheduler hints :/ | |
| 23:25:45 | mriedem | eandersson: part of me just wants to add an "or utils.is_neutron()" to that conditional check on reschedule | |
| 23:25:53 | mriedem | so if there is a legacy case for nova-net, it's not affected here | |
| 23:26:04 | mriedem | and we'll just always remove ports that nova created on reschedule | |
| 23:26:15 | mriedem | dansmith: do you recall any reasons why we don't just always remove ports we created when we reschedule? | |
| 23:26:39 | eandersson | Yea - I don't like 335788 | |
| 23:26:47 | eandersson | but utils.is_neutron would be awesome for this | |
| 23:27:16 | eandersson | that way if there really is some nova-net edge case we would be covered | |
| 23:27:25 | dansmith | um, I don't remember the exact reason, but I think there is one... like setup time or resources for certain backends maybe? | |
| 23:27:52 | mriedem | the bug reports i've been digging through seem to indicate that certain ML2 backends are cool with us not deallocating ports between reschedules | |
| 23:28:00 | mriedem | however, i think from the nova side, | |
| 23:28:09 | mriedem | we're still creating a port each time we call build_and_run_instance | |
| 23:28:20 | mriedem | so even if the underlying network is tying those together somehow, or ignoring them, | |
| 23:28:25 | mriedem | we're at worst eating up port quota for the project | |
| 23:28:28 | mriedem | for ports that we don't use | |
| 23:28:32 | mriedem | s/use/bind/ | |
| 23:28:53 | dansmith | tbh, it's been a long time since I looked at that path, probably before the great compute/conductor split for build | |
| 23:29:06 | mriedem | yeah, so, | |
| 23:29:20 | mriedem | in the interest of our "damn the torpedos" pledge from the ptg | |
| 23:29:23 | mriedem | and just break everything | |
| 23:29:37 | mriedem | i say we add an "is_neutron" check in this case and just always deallocate before reschedluing | |
| 23:29:46 | mriedem | if that breaks some unicorn ML2 backend, let them speak up | |
| 23:29:47 | dansmith | we delete ones we create unceremoniously on delete now? | |
| 23:30:03 | dansmith | and what about offload? | |
| 23:30:04 | mriedem | when the instance is deleted yes we cleanup and delete anything nova created | |
| 23:30:34 | gmann | alex_xu: hi, returned back to tokyo today | |
| 23:30:36 | mriedem | well, as you can guess, | |
| 23:30:44 | mriedem | dansmith: shelve offload won't delete or cleanup the ports at all, | |
| 23:30:52 | mriedem | it only unplugs the vifs via driver.destroy() | |
| 23:31:06 | mriedem | the actual port will still show as being bound to the original host, | |
| 23:31:15 | mriedem | until we unshelve the instance and bind it to a new host | |
| 23:31:15 | dansmith | and unshelve can continue using those ports? | |
| 23:31:23 | mriedem | i assume so... | |
| 23:31:40 | dansmith | I guess I would expect reschedule to do the same thing.. unbind/rebind | |
| 23:31:48 | mriedem | i only have a single node devstack atm so can't really test unshelve to another host and ssh into the guest after it's unshelved | |
| 23:31:54 | dansmith | but using shelve as the poster child is probably a weaker argument than delete | |
| 23:32:14 | mriedem | yeah, anything to do with ports and volumes + shelve is a roll of the dice | |
| 23:32:25 | dansmith | yeah | |
| 23:32:25 | mriedem | given we didn't actually terminate connetions for the last 4 years for volumes on unshelve | |
| 23:32:41 | dansmith | well, doing the same thing as delete is defensible so if that's the plan it seems reasonable | |
| 23:32:52 | mriedem | alright | |
| 23:33:12 | mriedem | i'll see if can wordsmith a reasonable commit message | |