Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-05-04
16:19:56 efried rm_work: It's really "always" been bad that you could do this (delete a port without considering what it's attached to), but it never mattered enough to do anything about. Now that the port is tied to real allocations in placement, it constitutes a leak and that kinda pushes it over the edge to where we should really do something.
16:19:59 rm_work the reason we do this directly is that there are several cases where a port detach in nova will simply fail to complete
16:20:14 rm_work at which point, we can either orphan the entire port, or we can just delete it
16:20:41 melwitt I guess if a port detach in nova fails to complete, there's a bug that needs to be fixed
16:21:11 efried When the detach fails to complete, is the instance still deleted?
16:21:11 rm_work example A: nova compute node is down
16:21:40 efried If the failure path still removes the allocations from placement, then it would be fine for you to force-delete the port.
16:21:54 melwitt if compute is down it would get cleaned up at the next run of the 'reap' periodic task on compute
16:22:03 rm_work but we can't wait that long
16:22:12 rm_work this is usually during time-sensitive failover operations
16:22:18 melwitt but this is another case where it would help to add start_immediately=True to the reap periodic
16:22:27 rm_work i think in all cases that we do a port delete the VM that it was attached to WILL be deleted anyway
16:22:45 efried actually, gibi_cape: doesn't the allocation go away whenever the instance is deleted? So the actual problem scenario is when we're detaching a port without deleting the instance?
16:22:46 melwitt that way the first run would happen when compute first comes up
16:22:49 rm_work because we only delete these ports when we are also deleting a VM
16:23:02 melwitt gotcha
16:23:06 gibi_cape efried, rm_work: the leaked allocation is reclaimed when the instance is deleted
16:23:07 rm_work unless I am missing a flow <_<
16:23:18 efried rm_work: If you're deleting the VM, and that succeeds, then we're good.
16:23:31 rm_work ok then so long as that is our workflow, we should not be hard-prevented from doing the port deletion
16:23:34 efried rm_work: So the way the proposed change would affect you is you would have to change the way you do that port detach.
16:23:37 rm_work it comes down to a timing issue
16:24:04 rm_work on our side, we're not going to wait for the detach to succeed in nova before we delete the port
16:24:10 rm_work we're going to move on with our cleanup
16:24:28 rm_work waiting would be problematic when we're in the middle of a time sensitive flow
16:24:30 efried rm_work: "not be hard-prevented" - right; but in order to do it after this change, you would have to change either code or policy (depending on how it's decided to implement)
16:24:38 rm_work yeah, if it's policy, fine
16:24:48 rm_work that's doable (we're already a service admin account)
16:25:08 rm_work if it's a code change, what does that mean -- changing to doing a detach first? because that's unlikely to happen
16:25:18 openstackgerrit garyk proposed openstack/nova master: VMware: populate datastore refs at init https://review.opendev.org/574688
16:25:40 efried Changing code to do two calls: 1) null the "owner" field, 2) delete op as you have it today
16:25:42 rm_work especially if you're telling us that it's really not a problem anyway given what we're doing in the big picture
16:26:10 rm_work ok, so just an owner null -- assuming there's no real delay in that, then that seems doable, one extra call isn't the end of the world
16:26:49 efried If code change is the direction we go, nova will have to do the same thing. mriedem expressed not feeling great about that. If it's possible to do this with policy, I'm guessing that will be preferred all around.
16:27:45 gibi_cape do we want to fix this-> < rm_work> the reason we do this directly is that there are several cases where a port detach in
16:27:48 gibi_cape nova will simply fail to complete
16:28:05 gibi_cape do we have some way to reproduce that?
16:28:06 rm_work see johnsom's just-sent email for a more complete example
16:28:12 rm_work i believe it's reproducable yes
16:28:14 gibi_cape rm_work: looking
16:30:58 rm_work the biggest one we can definitively reproduce would be a powered off compute host
16:31:17 rm_work this is a problem whether we are about to delete the port post-detach, or if we are going to re-use it
16:31:27 rm_work (it's ESPECIALLY a problem if we want to re-use it)
16:31:39 rm_work we've seen scenarios where a compute host was powered off for 12-24 hours
16:31:56 rm_work it's theoretically possible that it might NEVER come up -- and what would happen to our port in that case?
16:32:19 efried rm_work: Does the instance get deleted from the database?
16:32:39 openstackgerrit melanie witt proposed openstack/nova master: Use run_immediately=True for _cleanup_running_deleted_instances https://review.opendev.org/657132
16:32:57 rm_work it sticks in "deleting" vm state
16:33:55 efried rm_work: But if the point is that this whole thing is centered around you wanting to delete the VM, and you're going to delete it as soon as that's possible, then you're fine.
16:34:13 efried The proc/mem/disk resources are just as leaked as the bandwidth resources in this case where the VM is undeletable.
16:34:25 rm_work though what happens if we *do* need to delete the port?
16:34:35 rm_work err sorry, if we do need to re-use the port?
16:34:44 rm_work it's the same bug, just unrelated to the placement-leakage issue
16:34:57 gibi_cape rm_work: how do you re-use the port when the compute is down?
16:34:59 rm_work (since you were asking about the specific nova bug)
16:35:07 rm_work move it to another compute
16:35:25 rm_work so the case would be if we are using one port as our main ingress (instead of AAP)
16:35:42 rm_work so if a compute host is powered off, we need to move that port over to another working VM
16:35:53 efried gibi_cape: Do we reclaim the allocations when a port is detached?
16:35:55 gibi_cape rm_work: if nova is not involved in this move then nova cannot make sure that the resource needs of the port also moved
16:35:56 rm_work but a detach will just fail to complete
16:36:14 gibi_cape efried: if the port is detached in nova then yes we delete the port allocation in placement
16:36:18 rm_work so we can't reattach the port to a different VM
16:36:26 efried rm_work: Right, it sounds like this scenario is not possible.
16:36:33 efried You can't detach it if you can't talk to the compute
16:36:38 efried did I understand correcctly?
16:36:46 rm_work ok, so that's not a solvable issue in nova?
16:37:24 rm_work what if that compute host never comes back up? in case of hardware failure, this is possible
16:37:28 rm_work is that port gone forever?
16:38:15 gibi_cape rm_work: if the compute never comes up then the resoruce leak in placement is not relevant as nobody else could you the leaked resource
16:38:17 mriedem zzzeek: sort of a random mysql uniqueness / primary key question for you when you have a sec
16:38:20 cfriesen new etherpad: https://etherpad.openstack.org/p/nova-ptg-train-2
16:38:27 rm_work right, not really talking about the placement-specific issue anymore
16:38:34 rm_work this is about the bug that was behind our reasoning
16:38:44 rm_work there's multiple impacts on our side from the same bug
16:39:02 rm_work one of them was that we just force-delete the port to handle cleanup on our side
16:39:12 rm_work but the other is that we necessarily lose the port
16:41:39 gibi_cape rm_work: I'm not fully understand what you really want to achieve with moving the bound port from one VM / compute to another. For me a neutron port feels like just a logical entity that easy to create / re-create
16:41:59 melwitt gibi_cape: they can't ever change IPs :)
16:42:06 gibi_cape melwitt: thanks
16:42:13 rm_work correct
16:42:25 rm_work we NEED to maintain the IP or we're sunk
16:44:33 kashyap stephenfin: I saw your comment scroll-by, on removing the needless libvirt version constants: I have a half-baked branch with some WIP. Or have you already posted something?
16:44:47 kashyap stephenfin: Trying to ensure we're doing duplicating work :-)
16:45:07 stephenfin kashyap: Nah, I'm not writing anything. Just reviewing your and mriedem's patches
16:45:16 kashyap stephenfin: Ah, like that
16:45:39 gibi_cape rm_work: if the VM on the down compute recovers later how that VM will be recovered from networking perspective? On the compute side the vif is still attached, but the neutron port is moved and attached to another VM.
16:45:41 kashyap stephenfin: All clear. Thanks, more clean-ups coming to a Firefox tab near you
16:45:53 rm_work gibi_cape: we don't care, that VM is going to be deleted anyway
16:46:01 rm_work once we move the port, we consider the old VM "dead" to us
16:46:18 rm_work we'll always be issuing a VM delete on it
16:46:34 rm_work very much cattle
16:47:11 gibi_cape rm_work: OK. then the policy thing could work. Octavia can do the port delete in neutron with the extra policy as Octavia makes sure the VM is deleted so the leaked resource is recovered by the VM delete
16:47:24 rm_work I think that would be the best approach
16:47:39 rm_work though it WOULD be awesome if we could figure out a solution to the underlying nova "bug"
16:47:55 rm_work I know you folks may not consider it a bug, but from our perspective, it sure seems that way
16:48:17 rm_work if a detach would just ... "complete" even if the compute host is unreachable, it'd make us very happy :)
16:48:29 gibi_cape rm_work: would be nice to have a bug report on that or / and talk about this in the nova room
16:49:09 rm_work ok -- though i believe we've brought it up before and were told that was expected behaviour
16:49:28 rm_work but I suppose filing it officially would at least provide history for a decision even if it's still the same

Earlier   Later