Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-06-27
17:22:12 tssurya oh just saw mriedem's comment
17:23:29 melwitt tssurya: http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2018-06-26.log.html#t2018-06-26T21:23:37
17:23:35 mriedem i haven't gone over the spec updates yet
17:23:38 tssurya okay then I don't have anything else :) thanks
17:23:55 mriedem i would appreciate reviews on this regression bug fix which is cells related https://review.openstack.org/#/c/576161/
17:24:05 mriedem since that regression was backported and released in queens as well
17:24:46 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove unused DB API instance_group_delete method https://review.openstack.org/578223
17:24:47 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove unused DB API instance_group_member* methods https://review.openstack.org/578488
17:24:48 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove remaining legacy DB API instance_group* methods https://review.openstack.org/578497
17:27:02 mriedem dansmith: this was the queens backport for tssurya's fix to ignore down cells while listing https://review.openstack.org/#/c/578152/ - you might want to hit that before heading out
17:27:19 mriedem tssurya: i assume you'll also want that in pike, although cern is at queens now so maybe you personally don't care
17:27:19 dansmith ah cool thanks
17:27:30 mriedem but others might
17:27:54 mriedem i think mgagne is going to queens as well
17:27:58 mriedem not sure about godaddy
17:28:01 mriedem or nectar
17:28:02 tssurya mriedem: true we only want things in queens now :D
17:28:14 mriedem SpamapS: are you guys going to queens for cells v2?
17:28:18 mriedem multi-cell i mean
17:28:37 mgagne mriedem: we are far from starting the process but that's the goal yes
17:28:46 mriedem ok
17:29:07 dansmith mriedem: ah yeah I had looked at that patch of yours and forgot to circle back after the typo
17:29:08 tssurya mriedem: I will put a backport out to pike as well , once the queens one gets merged just in case
17:29:09 dansmith mah bad
17:30:18 mriedem while i'm begging for stable reviews, this is a simple latent bug fix, hitting people in the ops list (that's how it came up) https://review.openstack.org/#/q/Iaaf7f68d6874fd5d6e737e7d2bc589ea4a048fee
17:30:44 mriedem we screw up reschedules for a failed resize if you're using the multitenant aggregates filter
17:30:53 mriedem tenant isolation filter i mean
17:37:04 dansmith I guess I should highlight this to the cells people too: https://review.openstack.org/#/c/578163/
17:37:06 dansmith tssurya: ^
18:31:35 openstackgerrit Merged openstack/os-vif master: Add release note link in README https://review.openstack.org/578446
18:36:16 mriedem mordred: melwitt: dansmith: i've gone over the 'handling a down cell' spec again, comments inline but i think it's mostly good, just asking for more detail in a few spots - also wondering about handling queued_for_delete instances when listing/showing and when the cell comes back up for those
18:36:21 mriedem https://review.openstack.org/#/c/557369/
18:38:49 mriedem tl;dr i think for queued_for_delete instances we want to list/show those as UNKNOWN,
18:39:09 mriedem and then once the cell is back up, the status on those goes back to whatever it's reported in the cell db, could be ACTIVE, SHUTDOWN, etc
18:39:27 mriedem oh which reminds me, when do we clear the queued_for_delete flag once it's set....
18:42:10 mriedem uh, i think we might just not want to do queued_for_delete at all,
18:42:31 mriedem and 500 or 503 if you try to delete an instance in a down cell, like any other action on a server that's in a down cell like PUT /servers/{id}
18:49:04 mriedem random aside - can we delete this compat code from RT now? https://github.com/openstack/nova/blob/c8b93fa2493dce82ef4c0b1e7a503ba9b81c2e86/nova/compute/resource_tracker.py#L1206
18:49:16 mriedem if you're in rocky, you shouldn't have ocata computes
18:49:57 melwitt mriedem: well, blocking a delete is new and undesirable behavior that we haven't had before. that's my first thought
18:50:13 mriedem it's not new if your database is down
18:50:16 mriedem which is the case here
18:51:14 mriedem and it's not like once the cell computes come back up they can up-call to the instance mappings table to see, oh while i was dead the user wanted to delete this instance
18:51:23 melwitt so the pre cells v2 behavior when nova-api is up but the database is down is what, a 500?
18:51:31 mriedem melwitt: i'm sure
18:52:39 mriedem queued_for_delete means, to me anyway, that we have some kind of periodic in the control plane services that is trying to hit the cells for instance mappings that are queued for delete and once it reaches a cell that's back up, it does the actual delete
18:52:42 mriedem that's not being proposed
18:53:36 mriedem sort of like how the _reclaim_queued_deletes periodic works in the nova-compute service for SOFT_DELETED instances
18:53:41 melwitt yeah, I get where you're coming from on that. and I'm trying to remember why then we wanted queued_for_delete in the first place. the ability to list/show "deleted" instance shells? since 'novs list --deleted' is a thing you can do
18:54:01 melwitt *nova list --deleted
18:54:11 mriedem maybe, but the spec calls it out as being needed for quota calculation
18:54:51 melwitt okay, I don't think it was supposed to, that is, we had thought we could piggyback the quota stuff on that column. the column itself had other purpose prior to that
18:55:05 mriedem well i guess it's not explicitly linking that in, but it says it's needed for nova list, nova show and nova boot
18:55:43 mriedem calculation is implemented through `quotas using placement`_."
18:55:43 mriedem down cell until an all-cell-iteration independent solution for quota
18:55:43 mriedem "``nova boot`` should not succeed if that project has any living VMs in the
18:55:53 melwitt yeah. my guess is that it's for the case where an instance was deleted in the past from some cell, then that cell goes down, then someone does 'nova list --deleted', we'd have no way of knowing deleted or not without access to the cell database
18:56:02 mriedem *living* VMs means, i assume, that queued_for_delete would be considered there during the quota calculation
18:57:25 melwitt because if an instance was deleted from a cell in the past, if the cell goes down, without queued_for_delete the instance would reappear in the nova list
18:57:33 melwitt if we're going off instance mappings and nothing else
18:58:52 mriedem so by that logic, we'd always need to set InstanceMapping.queued_for_delete=True when deleting an instance even if the cell is up and we actually do delete the instance from the cell
18:59:43 melwitt yeah, I thought that was how it's supposed to work. mark queued_for_delete=True when a delete request comes in, and that just means "the user intended to delete this" and does not indicate success or not
19:02:23 mriedem ok, and we never set it back to queued_for_delete=False?
19:02:28 mriedem that's what is bothering me
19:02:29 melwitt and that "queued" was a misnomer in that there's no queuing system at the API level for deletes. maybe better named "delete_requested" or something. guess we should double check with tssurya and dansmith.
19:02:35 melwitt I don't think so, no
19:03:13 mriedem we've said in the spec that we're just not going to include instances from down cells if there are filter parameters,
19:03:30 mriedem we could just consider 'nova list --deleted' as a filter parameter that we're not going to include results from down cells
19:03:34 mriedem b/c we don'tknow
19:04:54 melwitt yeah, that's true. but as we talk I'm realizing I think the case we are trying to avoid is, deleted instance reappearing in a plain 'nova list'
19:05:07 melwitt or wait,
19:05:20 melwitt you're saying omit that one because it's in a down cell
19:05:32 melwitt yeah, I dunno.
19:05:57 melwitt the original thinking was show instance "shells" for instances in down cells
19:06:18 mriedem yes which we still plan on doing
19:06:21 melwitt we need to know when not to show a shell if it's deleted or was intended to be deleted
19:06:32 melwitt that's where queued_for_delete comes in, I think
19:08:29 mriedem https://review.openstack.org/#/c/557369/5/specs/rocky/approved/handling-down-cell.rst@103
19:08:49 mriedem "But then it's weird if I issue DELETE /servers/{id}, get a 202, and then my server status is always UNKNOWN and eventually goes to ACTIVE once the cell is back up - we aren't really doing any queuing in that case."
19:09:14 mriedem so maybe we need to:
19:09:32 mriedem actually nvm
19:10:13 mriedem was going to say, always set queued_for_delete on any delete request, but if the cell is down and we don't delete the instance, the api is going to start hiding it
19:10:25 mriedem and then it's going to show up again once the cell is back
19:10:45 mriedem i want to say:
19:10:58 mriedem 1. delete fails if the cell is down as it would today, don't touch queued_for_delete
19:11:07 melwitt yeah, I thought that's what the deal was going to be though. queued_for_delete was supposed to be a minimal effort to show something correct when cells are down, AFAIK
19:11:14 mriedem 2. if we actually get the instance and queue a delete of the instance and the cell is up, flip queued_for_delete=True
19:11:45 melwitt and to not have deleted instances reappear in nova list once a cell goes down
19:12:08 mriedem i think you can only do that if you actually issued an actual working delete request
19:12:15 mriedem either local in the api or cast to the compute
19:12:16 mriedem but the cell is up
19:12:46 mriedem otherwise we're lying if we say it's queued for delete while the cell is down and never actually delete it once the cell comes back up
19:13:07 mriedem i think DELETE /servers/{id} has to 500 if the cell is down
19:13:08 melwitt okay, yeah I agree with that. so you're just saying wait to set queued_for_delete until a delete is actually initiated with an up cell
19:13:17 mriedem yeah i think so
19:13:25 melwitt that's fair enough
19:13:40 mriedem because in that case we should have actually deleted the instance in the cell
19:13:54 melwitt right
19:14:08 mriedem and then queued_for_delete is only read while listing/showing over down cells
19:14:17 mriedem and never goes from True back to False

Earlier   Later