Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-11-15
22:10:40 openstackgerrit Matt Riedemann proposed openstack/nova master: Add microversion to allow setting flavor description https://review.openstack.org/515213
22:10:59 mriedem sdague: ^ i rebased to kick it into the gate, it didn't go through for some reason
22:17:23 openstackgerrit Eric Berglund proposed openstack/nova master: WIP(5): PowerVM driver: ovs vif https://review.openstack.org/422512
22:54:55 eandersson Is there a reason why this flag is always set to False? https://github.com/openstack/nova/blob/mitaka-eol/nova/virt/driver.py#L1249
22:56:14 eandersson In certain scenarios when nova-compute fails to schedule it does not clean up the network ports
22:56:35 eandersson e.g. when hitting > qemu-kvm: cannot set up guest memory 'pc.ram': Cannot allocate memory
22:57:10 mriedem eandersson: it's apparently specific to baremetal
22:57:15 mriedem in how it's used anyway
22:58:19 mriedem otherwise i think ports created for an instance will go with the instance and be bound on the next host
22:58:34 eandersson https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1880
22:58:50 eandersson It's used in this code path
22:59:10 mriedem yeah i'm looking at it locally
22:59:13 eandersson and basically if this function is called self._cleanup_allocated_networks
22:59:17 eandersson it will clean it up properly
22:59:33 mriedem what do you actually need cleaned up before the reschedule?
22:59:48 mriedem the ports should not be bound to that host at that point
22:59:56 eandersson Well the problem we are hitting is that each time it gets re-scheduled, a new ip is assigned
23:00:03 mriedem is a new port created?
23:00:08 eandersson Yea
23:00:15 mriedem ok, that's a problem, and i think an old one
23:00:29 mriedem i'm sure i could find some bugs if i dug for a bit
23:00:49 eandersson I am reproducing it with this http://paste.openstack.org/show/626450/
23:00:55 mriedem https://bugs.launchpad.net/nova/+bug/1703540
23:00:55 openstack Launchpad bug 1703540 in OpenStack Compute (nova) "Reschedule with libvirt exception leaves dangling neutron ports" [High,Triaged]
23:01:08 mriedem https://bugs.launchpad.net/nova/+bug/1531022
23:01:09 openstack Launchpad bug 1531022 in OpenStack Compute (nova) "libvirt driver doesn't cleanup the tap interface on vm re-schedule" [Low,Confirmed]
23:01:24 eandersson oh not sure why I didn't find that one
23:01:59 mriedem yeah see comment #3 on https://bugs.launchpad.net/nova/+bug/1703540
23:02:00 openstack Launchpad bug 1703540 in OpenStack Compute (nova) "Reschedule with libvirt exception leaves dangling neutron ports" [High,Triaged]
23:02:23 eandersson Yep - perfect
23:02:31 eandersson that is the exact same conclusion I came to
23:02:37 mriedem it would be as easy as calling cleanup_instance_network_on_host to remove any ports we created
23:03:03 mriedem however, doing that for neutron would basically be the same as for _cleanup_allocated_networks i think
23:03:15 mriedem because nova has to be smart about not deleting ports that the user supplied vs ones that nova created
23:03:44 eandersson Is there a reason why we can't just call _cleanup_allocated_networks ?
23:03:53 eandersson I tested that out in my lab and it worked 100% of the time
23:04:38 eandersson but obviously my test case is rather limited
23:05:16 mriedem honestly i'm not sure, and i worry that doing that will break something else, which is probably why we've never just done that
23:05:24 mriedem there was this patch for a related bug long ago https://review.openstack.org/#/c/367316/
23:06:37 mriedem oh that was a backport
23:07:11 mriedem ok so https://review.openstack.org/#/c/243477/ is a different fix,
23:07:29 mriedem ^ is if all reschedules fail and we've run out of retries i think, so the instance is set to ERROR
23:07:32 mriedem which is different than,
23:07:37 mriedem host A fails, host B works
23:07:41 mriedem you'd have 2 ports
23:07:48 openstackgerrit Michael Still proposed openstack/nova master: Convert ext filesystem resizes to privsep. https://review.openstack.org/517516
23:07:49 openstackgerrit Michael Still proposed openstack/nova master: Move flushing block devices to privsep. https://review.openstack.org/519010
23:07:49 openstackgerrit Michael Still proposed openstack/nova master: Start moving users of parted to privsep. https://review.openstack.org/519011
23:07:50 openstackgerrit Michael Still proposed openstack/nova master: Move remaining uses of parted to privsep. https://review.openstack.org/519483
23:07:50 openstackgerrit Michael Still proposed openstack/nova master: Convert users of tune2fs to privsep. https://review.openstack.org/519484
23:08:20 eandersson I see
23:10:53 mriedem i'm also not entirely sure if _cleanup_allocated_networks will do the entire job since it doesn't explicitly unplug vifs via the virt driver,
23:11:08 mriedem i'm not sure if deleting the port will take care of that out of band?
23:11:28 mriedem https://bugs.launchpad.net/nova/+bug/1531022/comments/2
23:11:28 openstack Launchpad bug 1531022 in OpenStack Compute (nova) "libvirt driver doesn't cleanup the tap interface on vm re-schedule" [Low,Confirmed]
23:12:21 efried jaypipes More sanity check: In order to pass the entire provider tree to update_inventory, I'm going to need to *get* the entire tree. Which entails a new method in report client that calls GET with the new ?tree= deal, then builds up the ProviderTree object accordingly. And then actually returns that guy, which is currently a private _provider_tree attribute and not actually returned anywhere yet.
23:12:27 eandersson I don't see the tap at least on the compute I tested it on
23:12:52 eandersson but this is mitaka, who knows how it would work on other versions of Openstack
23:13:08 mriedem well, does it maybe depend on the type of interface?
23:13:12 mriedem vif type i mean
23:13:23 efried jaypipes Then when I get it back from update_inventory, the driver may have added children and whatnot, so I'm going to have to walk the tree to rearrange the placement db accordingly.
23:14:01 mriedem eandersson: here is another one https://review.openstack.org/#/c/335788
23:14:13 efried jaypipes And also call _normalize_inventory_from_cn_obj on every provider in the tree, in case they decided to shove their VCPU/MEMORY_MB/DISK_GB in children instead of leaving them in the root.
23:14:52 mriedem "If I understand the code correctly, the intention of not deleting the port when rescheduling is that it could be reused on the second compute node. But that reuse does not seem to happen, instead nova allocates another port, leaving the first one pending."
23:15:11 mriedem eandersson: it could also totally be some legacy nova-network thing
23:15:20 efried jaypipes That first thing sort of a tree-ified version of _ensure_resource_provider
23:15:28 mriedem like maybe we didn't deallocate networks on reschedule if you were using nova-net? i'm not sure
23:17:32 eandersson Yea - I have done a lot of testing, but I am hesitant to change this without knowing all potential side-effects (even if only for internal usage)
23:17:45 eandersson but it being a legacy thing would make a lot of sense
23:20:50 mriedem eandersson: ok i'm going to duplicate some of the bugs to just this one https://bugs.launchpad.net/nova/+bug/1597596
23:20:50 openstack Launchpad bug 1597596 in OpenStack Compute (nova) "network not always cleaned up when spawning VMs" [Medium,Confirmed]
23:20:56 mriedem which had a patch and did a lot of the same investigation
23:24:55 mriedem eandersson: so https://review.openstack.org/#/c/335788/ isn't going to fly, because it's totally relying on 2 out of tree scheduler hints :/
23:25:45 mriedem eandersson: part of me just wants to add an "or utils.is_neutron()" to that conditional check on reschedule
23:25:53 mriedem so if there is a legacy case for nova-net, it's not affected here
23:26:04 mriedem and we'll just always remove ports that nova created on reschedule
23:26:15 mriedem dansmith: do you recall any reasons why we don't just always remove ports we created when we reschedule?
23:26:39 eandersson Yea - I don't like 335788
23:26:47 eandersson but utils.is_neutron would be awesome for this
23:27:16 eandersson that way if there really is some nova-net edge case we would be covered
23:27:25 dansmith um, I don't remember the exact reason, but I think there is one... like setup time or resources for certain backends maybe?
23:27:52 mriedem the bug reports i've been digging through seem to indicate that certain ML2 backends are cool with us not deallocating ports between reschedules
23:28:00 mriedem however, i think from the nova side,
23:28:09 mriedem we're still creating a port each time we call build_and_run_instance
23:28:20 mriedem so even if the underlying network is tying those together somehow, or ignoring them,
23:28:25 mriedem we're at worst eating up port quota for the project
23:28:28 mriedem for ports that we don't use
23:28:32 mriedem s/use/bind/
23:28:53 dansmith tbh, it's been a long time since I looked at that path, probably before the great compute/conductor split for build
23:29:06 mriedem yeah, so,
23:29:20 mriedem in the interest of our "damn the torpedos" pledge from the ptg
23:29:23 mriedem and just break everything
23:29:37 mriedem i say we add an "is_neutron" check in this case and just always deallocate before reschedluing
23:29:46 mriedem if that breaks some unicorn ML2 backend, let them speak up
23:29:47 dansmith we delete ones we create unceremoniously on delete now?
23:30:03 dansmith and what about offload?
23:30:04 mriedem when the instance is deleted yes we cleanup and delete anything nova created
23:30:34 gmann alex_xu: hi, returned back to tokyo today
23:30:36 mriedem well, as you can guess,
23:30:44 mriedem dansmith: shelve offload won't delete or cleanup the ports at all,

Earlier   Later