Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-11-15
23:00:55 openstack Launchpad bug 1703540 in OpenStack Compute (nova) "Reschedule with libvirt exception leaves dangling neutron ports" [High,Triaged]
23:00:55 mriedem https://bugs.launchpad.net/nova/+bug/1703540
23:01:08 mriedem https://bugs.launchpad.net/nova/+bug/1531022
23:01:09 openstack Launchpad bug 1531022 in OpenStack Compute (nova) "libvirt driver doesn't cleanup the tap interface on vm re-schedule" [Low,Confirmed]
23:01:24 eandersson oh not sure why I didn't find that one
23:01:59 mriedem yeah see comment #3 on https://bugs.launchpad.net/nova/+bug/1703540
23:02:00 openstack Launchpad bug 1703540 in OpenStack Compute (nova) "Reschedule with libvirt exception leaves dangling neutron ports" [High,Triaged]
23:02:23 eandersson Yep - perfect
23:02:31 eandersson that is the exact same conclusion I came to
23:02:37 mriedem it would be as easy as calling cleanup_instance_network_on_host to remove any ports we created
23:03:03 mriedem however, doing that for neutron would basically be the same as for _cleanup_allocated_networks i think
23:03:15 mriedem because nova has to be smart about not deleting ports that the user supplied vs ones that nova created
23:03:44 eandersson Is there a reason why we can't just call _cleanup_allocated_networks ?
23:03:53 eandersson I tested that out in my lab and it worked 100% of the time
23:04:38 eandersson but obviously my test case is rather limited
23:05:16 mriedem honestly i'm not sure, and i worry that doing that will break something else, which is probably why we've never just done that
23:05:24 mriedem there was this patch for a related bug long ago https://review.openstack.org/#/c/367316/
23:06:37 mriedem oh that was a backport
23:07:11 mriedem ok so https://review.openstack.org/#/c/243477/ is a different fix,
23:07:29 mriedem ^ is if all reschedules fail and we've run out of retries i think, so the instance is set to ERROR
23:07:32 mriedem which is different than,
23:07:37 mriedem host A fails, host B works
23:07:41 mriedem you'd have 2 ports
23:07:48 openstackgerrit Michael Still proposed openstack/nova master: Convert ext filesystem resizes to privsep. https://review.openstack.org/517516
23:07:49 openstackgerrit Michael Still proposed openstack/nova master: Start moving users of parted to privsep. https://review.openstack.org/519011
23:07:49 openstackgerrit Michael Still proposed openstack/nova master: Move flushing block devices to privsep. https://review.openstack.org/519010
23:07:50 openstackgerrit Michael Still proposed openstack/nova master: Convert users of tune2fs to privsep. https://review.openstack.org/519484
23:07:50 openstackgerrit Michael Still proposed openstack/nova master: Move remaining uses of parted to privsep. https://review.openstack.org/519483
23:08:20 eandersson I see
23:10:53 mriedem i'm also not entirely sure if _cleanup_allocated_networks will do the entire job since it doesn't explicitly unplug vifs via the virt driver,
23:11:08 mriedem i'm not sure if deleting the port will take care of that out of band?
23:11:28 openstack Launchpad bug 1531022 in OpenStack Compute (nova) "libvirt driver doesn't cleanup the tap interface on vm re-schedule" [Low,Confirmed]
23:11:28 mriedem https://bugs.launchpad.net/nova/+bug/1531022/comments/2
23:12:21 efried jaypipes More sanity check: In order to pass the entire provider tree to update_inventory, I'm going to need to *get* the entire tree. Which entails a new method in report client that calls GET with the new ?tree= deal, then builds up the ProviderTree object accordingly. And then actually returns that guy, which is currently a private _provider_tree attribute and not actually returned anywhere yet.
23:12:27 eandersson I don't see the tap at least on the compute I tested it on
23:12:52 eandersson but this is mitaka, who knows how it would work on other versions of Openstack
23:13:08 mriedem well, does it maybe depend on the type of interface?
23:13:12 mriedem vif type i mean
23:13:23 efried jaypipes Then when I get it back from update_inventory, the driver may have added children and whatnot, so I'm going to have to walk the tree to rearrange the placement db accordingly.
23:14:01 mriedem eandersson: here is another one https://review.openstack.org/#/c/335788
23:14:13 efried jaypipes And also call _normalize_inventory_from_cn_obj on every provider in the tree, in case they decided to shove their VCPU/MEMORY_MB/DISK_GB in children instead of leaving them in the root.
23:14:52 mriedem "If I understand the code correctly, the intention of not deleting the port when rescheduling is that it could be reused on the second compute node. But that reuse does not seem to happen, instead nova allocates another port, leaving the first one pending."
23:15:11 mriedem eandersson: it could also totally be some legacy nova-network thing
23:15:20 efried jaypipes That first thing sort of a tree-ified version of _ensure_resource_provider
23:15:28 mriedem like maybe we didn't deallocate networks on reschedule if you were using nova-net? i'm not sure
23:17:32 eandersson Yea - I have done a lot of testing, but I am hesitant to change this without knowing all potential side-effects (even if only for internal usage)
23:17:45 eandersson but it being a legacy thing would make a lot of sense
23:20:50 openstack Launchpad bug 1597596 in OpenStack Compute (nova) "network not always cleaned up when spawning VMs" [Medium,Confirmed]
23:20:50 mriedem eandersson: ok i'm going to duplicate some of the bugs to just this one https://bugs.launchpad.net/nova/+bug/1597596
23:20:56 mriedem which had a patch and did a lot of the same investigation
23:24:55 mriedem eandersson: so https://review.openstack.org/#/c/335788/ isn't going to fly, because it's totally relying on 2 out of tree scheduler hints :/
23:25:45 mriedem eandersson: part of me just wants to add an "or utils.is_neutron()" to that conditional check on reschedule
23:25:53 mriedem so if there is a legacy case for nova-net, it's not affected here
23:26:04 mriedem and we'll just always remove ports that nova created on reschedule
23:26:15 mriedem dansmith: do you recall any reasons why we don't just always remove ports we created when we reschedule?
23:26:39 eandersson Yea - I don't like 335788
23:26:47 eandersson but utils.is_neutron would be awesome for this
23:27:16 eandersson that way if there really is some nova-net edge case we would be covered
23:27:25 dansmith um, I don't remember the exact reason, but I think there is one... like setup time or resources for certain backends maybe?
23:27:52 mriedem the bug reports i've been digging through seem to indicate that certain ML2 backends are cool with us not deallocating ports between reschedules
23:28:00 mriedem however, i think from the nova side,
23:28:09 mriedem we're still creating a port each time we call build_and_run_instance
23:28:20 mriedem so even if the underlying network is tying those together somehow, or ignoring them,
23:28:25 mriedem we're at worst eating up port quota for the project
23:28:28 mriedem for ports that we don't use
23:28:32 mriedem s/use/bind/
23:28:53 dansmith tbh, it's been a long time since I looked at that path, probably before the great compute/conductor split for build
23:29:06 mriedem yeah, so,
23:29:20 mriedem in the interest of our "damn the torpedos" pledge from the ptg
23:29:23 mriedem and just break everything
23:29:37 mriedem i say we add an "is_neutron" check in this case and just always deallocate before reschedluing
23:29:46 mriedem if that breaks some unicorn ML2 backend, let them speak up
23:29:47 dansmith we delete ones we create unceremoniously on delete now?
23:30:03 dansmith and what about offload?
23:30:04 mriedem when the instance is deleted yes we cleanup and delete anything nova created
23:30:34 gmann alex_xu: hi, returned back to tokyo today
23:30:36 mriedem well, as you can guess,
23:30:44 mriedem dansmith: shelve offload won't delete or cleanup the ports at all,
23:30:52 mriedem it only unplugs the vifs via driver.destroy()
23:31:06 mriedem the actual port will still show as being bound to the original host,
23:31:15 mriedem until we unshelve the instance and bind it to a new host
23:31:15 dansmith and unshelve can continue using those ports?
23:31:23 mriedem i assume so...
23:31:40 dansmith I guess I would expect reschedule to do the same thing.. unbind/rebind
23:31:48 mriedem i only have a single node devstack atm so can't really test unshelve to another host and ssh into the guest after it's unshelved
23:31:54 dansmith but using shelve as the poster child is probably a weaker argument than delete
23:32:14 mriedem yeah, anything to do with ports and volumes + shelve is a roll of the dice
23:32:25 dansmith yeah
23:32:25 mriedem given we didn't actually terminate connetions for the last 4 years for volumes on unshelve
23:32:41 dansmith well, doing the same thing as delete is defensible so if that's the plan it seems reasonable
23:32:52 mriedem alright
23:33:12 mriedem i'll see if can wordsmith a reasonable commit message
23:33:24 mriedem because there is also prior art in this area which always confuses me
23:35:11 gmann yikun: re: on https://review.openstack.org/#/c/518644/11
23:36:04 gmann yikun: all param in query are as string so we use type as string, if any param is integer then we can use non_integer or positive_integer schema
23:37:13 gmann yikun: and on additionalProperties=True we kept that for backward compatibility but at validation layer all the additional properties will be ignored and stripout
23:38:17 gmann yikun: if you are doing migration pagination with microversion then you can make it False. anyways i will check the patch series today
23:38:39 mriedem gmann: i updated that one today
23:38:47 mriedem the schema thing anyway
23:39:04 openstackgerrit Matt Riedemann proposed openstack/python-novaclient master: Microversion 2.54 - Enable reset keypair while rebuild https://review.openstack.org/519572

Earlier   Later