Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-09-29
15:25:45 lyarwood awesome
16:12:53 bauzas artom: honestly, I'm torn with https://review.opendev.org/c/openstack/nova/+/808474
16:13:00 bauzas that's an behavioural change
16:13:10 opendevreview Balazs Gibizer proposed openstack/nova master: Enable min pps tempest testing in nova-next https://review.opendev.org/c/openstack/nova/+/811748
16:13:18 bauzas operators suppose a plugoff when delete
16:13:38 bauzas now, we'll first try to shutdown the guest for every instance
16:13:46 bauzas including other drivers but libvirt
16:14:20 melwitt that is my concern as well. maybe it could be conditional on bfv that is not "delete on termination"?
16:15:18 melwitt is that the only case where this would be desired?
16:15:22 melwitt or rather, is there any gain in doing it for non bfv non delete on termination?
16:15:58 dansmith generally you don't want that :)
16:16:07 melwitt maybe also shared storage is another case
16:16:08 dansmith the only real case for non-bfv volumes is for precious data
16:16:53 melwitt but precious data on a local disk that's going to be deleted anyway? I must be missing something
16:17:15 dansmith sorry, thought you were talking about delete-on-termination for non-bfv cinder volumes
16:17:23 artom melwitt, "is there any gain in doing it for non bfv non delete on termination?" None that I can see
16:17:33 artom Well, no, any volume, really
16:17:41 artom Doesn't have to be bfv
16:17:50 melwitt oh, yeah attached volumes. I wasn't thinking of that. yeah
16:18:02 dansmith that's my point, delete-on-termination should only be useful for bfv volumes we created with non-precious data from an image
16:18:05 artom It's still attached and mounted in the guest, and would ideally be flushed correctly if it's not delete_on_termination=True
16:18:38 artom I need to run an errand quickly, can this be carried over to the gerrit review?
16:18:44 artom And thanks for looking into it :)
16:19:22 artom And yeah, so bauzas's point, the compute manager/driver division of labour here is pretty muddy
16:19:32 melwitt my bad for not looking at the change yet. if it's targeted to only instance with volume(s) cases I think that makes a lot more sense
16:19:38 gibi is it really a gracefull shutdown via openstack server stop and then a openstack server delete?
16:19:56 dansmith gibi: stop + delete should be graceful
16:20:20 gibi so my point is this can already done with our APIs
16:20:23 melwitt just saying I don't think we should be doing it for everything, for things where the data is going to be blown away anyway
16:21:01 dansmith gibi: for sure. I assume the goal is to make nova do the graceful behavior if volumes are attached, but to do it properly really requires some higher-level orch, like a stop...timeout...destroy kind of thing
16:21:24 dansmith "do the graceful behavior *automatically*" I should have said
16:21:36 gibi OK I see
16:21:49 melwitt yeah that is my understanding as well
16:22:10 dansmith I'm a bit torn, because unless you're running with unsafe cache, I would think that fast destroy is fine.. might have a journal to replay when you use the volume later, but...
16:22:11 gibi it make sense for data consistency but it also makes delete slower so I think this should be opt in
16:23:16 melwitt dansmith: yeah it's weird, the user is experiencing volume gets corrupted and no longer usable when they delete without stopping first
16:23:23 melwitt we had thought just deleting should be fine but it's behaving in a way we didn't expect
16:23:34 melwitt not sure why
16:23:37 dansmith destroy of a running vm is the same as pulling the plug.. if you're using a precious volume, you wouldn't do that to a physical server, so...
16:24:27 artom dansmith, so the "real" problem is https://bugzilla.redhat.com/show_bug.cgi?id=1965081
16:24:29 melwitt yeah but re: "I would think that fast destroy is fine"?
16:24:39 artom Apparently in some cases stop+delete causes races
16:25:01 melwitt yeah I think you have to poll and wait for it to be stopped no?
16:25:32 dansmith melwitt: to do the graceful shutdown, you'd need some long-running task, yeah, what I said above
16:25:38 artom So we can either make delete safer, or require that any orchestration/automation on top of Nova does stop + delete, but then we would need to fix that race
16:25:41 dansmith artom: so this has nothing to do with volume safety?
16:26:10 artom dansmith, it does, because the reason for doing stop + delete (which can cause this deadlock) is volume safety
16:26:18 melwitt it does. it began with the racing problem and then we said "try a delete without the stop" and then their volumes got messed up
16:26:38 dansmith oh the dbdeadlock came from stop?
16:26:54 lyarwood doesn't delete take an instance.uuid lock on the compute?
16:26:57 artom dansmith, stop immediately followed by delete, apparently
16:27:18 melwitt it came from doing a delete right after a stop
16:27:20 melwitt without waiting for the stop to be stopped
16:27:24 melwitt lyarwood: good question
16:27:25 dansmith okay, so, fix that
16:27:42 dansmith don't engineer an orchestrated graceful delete, IMHO
16:27:59 dansmith I'm guessing task_state doesn't protect the delete from running,
16:28:12 artom I dunno, I don't necessarily think expecting an attached volume to not be corrupted after a delete is invalid
16:28:18 dansmith and the api might start the delete process while the stop is still running on the compute host or something
16:28:29 melwitt hm, stop and delete are both locked with instance.uuid
16:28:46 melwitt I just checked
16:28:46 dansmith artom: I agree, delete could leave a volume unhappy, but I'm thinking if you issued a stop and then delete you're assuming they're queued
16:28:47 lyarwood https://github.com/openstack/nova/blob/e07bb310b674fb471a92edf3258e564f05534595/nova/compute/manager.py#L3237-L3255 looks like soft delete doesnt
16:29:38 dansmith artom: in reality it's probably about like hitting shutdown on your server and then pulling the plug before it finishes, but...
16:29:42 melwitt dansmith: that's exactly what they want, the queuing
16:30:21 dansmith I think relying on the instance lock is probably too fragile here,
16:30:35 dansmith since it's just on the compute node, but I'd have to go look at the (many) delete path(s) we have
16:31:16 lyarwood I wouldn't say so for this case where the instance was running on a host
16:31:25 dansmith delete is and always has been pretty much "I want this to complete and stop charging me immediately", so... we're really not wrong here, IMHO
16:31:28 lyarwood that makes perfect sense
16:31:30 opendevreview Balazs Gibizer proposed openstack/nova master: Enable min pps tempest testing in nova-next https://review.opendev.org/c/openstack/nova/+/811748
16:32:26 dansmith lyarwood: wouldn't say what, that relying on the lock is unsafe? we can do stuff in the api to delete things (in the local case) which has nothing to do with the instance lock,
16:32:46 dansmith and I'm not sure that relying on the ordering of two calls is really safe either unless we have perfectly fair locks
16:32:50 bauzas sorry, I was afk
16:33:13 lyarwood dansmith: we can't force the delete in the API of an instance still associated with a host can we?
16:33:18 dansmith like, if you issue three calls on that instance and the stop is #2, delete is #3, the lock has to be fair to make sure we don't happen to choose the waiting #3 thread when we stop doing #1
16:33:32 dansmith lyarwood: of course we can
16:33:43 dansmith if we think the host is down, or we missed the last service update, etc
16:34:12 lyarwood right I'm stuck in the happy path here
16:34:38 melwitt we have the ability to add fair=True to the locks, I wonder if that would work?
16:34:48 bauzas my biggest concern here is that we would make destroy holding for a graceful shutdown
16:35:01 dansmith bauzas: yeah I don't think we should do that for sure
16:35:23 dansmith bauzas: I think at most, we should try to make sure a delete doesn't preempt a stop operation in progress
16:35:31 bauzas yup, agreed
16:35:41 bauzas if a stop is occuring, destroy should wait
16:35:58 melwitt how can we make it wait without rejecting it?
16:36:08 bauzas excellent questioin
16:36:21 melwitt I do wonder if the fair=True would help
16:36:28 bauzas I'd say we would reject synchronously by looking at the VM state
16:36:39 melwitt I'll try it in a devstack if I can repro the problem
16:37:09 bauzas destroy is synchronous, right?
16:37:20 bauzas even if the host is down, destroy will occur
16:37:43 dansmith melwitt: I'm just suggesting that lock inversion is *one* possible way that we might not be able to depend on the ordering
16:37:48 bauzas that's the guest destroy which is async, right?
16:38:19 dansmith anything else that happens on the compute host before we block on the lock could cause it, as well as if conductor is involved or something like that
16:38:31 bauzas dansmith: I missed your proposal
16:38:43 bauzas you're telling we could lock on stop ?
16:38:58 bauzas hence preventing the delete ?
16:39:15 melwitt we do lock on stop on the compute node
16:39:21 melwitt but lock waiters would not be in order because they are not fair locks

Earlier   Later