| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-09-29 | |||
| 16:25:01 | melwitt | yeah I think you have to poll and wait for it to be stopped no? | |
| 16:25:32 | dansmith | melwitt: to do the graceful shutdown, you'd need some long-running task, yeah, what I said above | |
| 16:25:38 | artom | So we can either make delete safer, or require that any orchestration/automation on top of Nova does stop + delete, but then we would need to fix that race | |
| 16:25:41 | dansmith | artom: so this has nothing to do with volume safety? | |
| 16:26:10 | artom | dansmith, it does, because the reason for doing stop + delete (which can cause this deadlock) is volume safety | |
| 16:26:18 | melwitt | it does. it began with the racing problem and then we said "try a delete without the stop" and then their volumes got messed up | |
| 16:26:38 | dansmith | oh the dbdeadlock came from stop? | |
| 16:26:54 | lyarwood | doesn't delete take an instance.uuid lock on the compute? | |
| 16:26:57 | artom | dansmith, stop immediately followed by delete, apparently | |
| 16:27:18 | melwitt | it came from doing a delete right after a stop | |
| 16:27:20 | melwitt | without waiting for the stop to be stopped | |
| 16:27:24 | melwitt | lyarwood: good question | |
| 16:27:25 | dansmith | okay, so, fix that | |
| 16:27:42 | dansmith | don't engineer an orchestrated graceful delete, IMHO | |
| 16:27:59 | dansmith | I'm guessing task_state doesn't protect the delete from running, | |
| 16:28:12 | artom | I dunno, I don't necessarily think expecting an attached volume to not be corrupted after a delete is invalid | |
| 16:28:18 | dansmith | and the api might start the delete process while the stop is still running on the compute host or something | |
| 16:28:29 | melwitt | hm, stop and delete are both locked with instance.uuid | |
| 16:28:46 | melwitt | I just checked | |
| 16:28:46 | dansmith | artom: I agree, delete could leave a volume unhappy, but I'm thinking if you issued a stop and then delete you're assuming they're queued | |
| 16:28:47 | lyarwood | https://github.com/openstack/nova/blob/e07bb310b674fb471a92edf3258e564f05534595/nova/compute/manager.py#L3237-L3255 looks like soft delete doesnt | |
| 16:29:38 | dansmith | artom: in reality it's probably about like hitting shutdown on your server and then pulling the plug before it finishes, but... | |
| 16:29:42 | melwitt | dansmith: that's exactly what they want, the queuing | |
| 16:30:21 | dansmith | I think relying on the instance lock is probably too fragile here, | |
| 16:30:35 | dansmith | since it's just on the compute node, but I'd have to go look at the (many) delete path(s) we have | |
| 16:31:16 | lyarwood | I wouldn't say so for this case where the instance was running on a host | |
| 16:31:25 | dansmith | delete is and always has been pretty much "I want this to complete and stop charging me immediately", so... we're really not wrong here, IMHO | |
| 16:31:28 | lyarwood | that makes perfect sense | |
| 16:31:30 | opendevreview | Balazs Gibizer proposed openstack/nova master: Enable min pps tempest testing in nova-next https://review.opendev.org/c/openstack/nova/+/811748 | |
| 16:32:26 | dansmith | lyarwood: wouldn't say what, that relying on the lock is unsafe? we can do stuff in the api to delete things (in the local case) which has nothing to do with the instance lock, | |
| 16:32:46 | dansmith | and I'm not sure that relying on the ordering of two calls is really safe either unless we have perfectly fair locks | |
| 16:32:50 | bauzas | sorry, I was afk | |
| 16:33:13 | lyarwood | dansmith: we can't force the delete in the API of an instance still associated with a host can we? | |
| 16:33:18 | dansmith | like, if you issue three calls on that instance and the stop is #2, delete is #3, the lock has to be fair to make sure we don't happen to choose the waiting #3 thread when we stop doing #1 | |
| 16:33:32 | dansmith | lyarwood: of course we can | |
| 16:33:43 | dansmith | if we think the host is down, or we missed the last service update, etc | |
| 16:34:12 | lyarwood | right I'm stuck in the happy path here | |
| 16:34:38 | melwitt | we have the ability to add fair=True to the locks, I wonder if that would work? | |
| 16:34:48 | bauzas | my biggest concern here is that we would make destroy holding for a graceful shutdown | |
| 16:35:01 | dansmith | bauzas: yeah I don't think we should do that for sure | |
| 16:35:23 | dansmith | bauzas: I think at most, we should try to make sure a delete doesn't preempt a stop operation in progress | |
| 16:35:31 | bauzas | yup, agreed | |
| 16:35:41 | bauzas | if a stop is occuring, destroy should wait | |
| 16:35:58 | melwitt | how can we make it wait without rejecting it? | |
| 16:36:08 | bauzas | excellent questioin | |
| 16:36:21 | melwitt | I do wonder if the fair=True would help | |
| 16:36:28 | bauzas | I'd say we would reject synchronously by looking at the VM state | |
| 16:36:39 | melwitt | I'll try it in a devstack if I can repro the problem | |
| 16:37:09 | bauzas | destroy is synchronous, right? | |
| 16:37:20 | bauzas | even if the host is down, destroy will occur | |
| 16:37:43 | dansmith | melwitt: I'm just suggesting that lock inversion is *one* possible way that we might not be able to depend on the ordering | |
| 16:37:48 | bauzas | that's the guest destroy which is async, right? | |
| 16:38:19 | dansmith | anything else that happens on the compute host before we block on the lock could cause it, as well as if conductor is involved or something like that | |
| 16:38:31 | bauzas | dansmith: I missed your proposal | |
| 16:38:43 | bauzas | you're telling we could lock on stop ? | |
| 16:38:58 | bauzas | hence preventing the delete ? | |
| 16:39:15 | melwitt | we do lock on stop on the compute node | |
| 16:39:21 | melwitt | but lock waiters would not be in order because they are not fair locks | |
| 16:39:42 | melwitt | *would not necessarily be in order | |
| 16:39:59 | dansmith | bauzas: I haven't proposed anything | |
| 16:40:14 | bauzas | ok, melwitt explained | |
| 16:41:03 | dansmith | yeah, so when we go to do delete, | |
| 16:41:06 | melwitt | dansmith: ack, I was just thinking that would be such a small and simple way to fix it if the fair locks would do that. but like you said if we have an issue with two async things arriving on the compute node in the "wrong order" then it wouldn't help | |
| 16:41:12 | dansmith | we create an instance event before we run terminate_instance and grab the lock, | |
| 16:41:22 | dansmith | which means we're doing network IO to conductor | |
| 16:41:32 | dansmith | so even fair locks won't prevent order inversion there | |
| 16:41:53 | melwitt | yeah... | |
| 16:42:13 | dansmith | so if stop and delete arrive in the right order, but each call out to conductor, we hit any conductor in the cluster, each one tries to create records in the db, who knows which one will finish first, grab the lock, etc | |
| 16:42:35 | dansmith | that's what I mean by assuming this sort of super tight ordering is unsafe, even though we *think* we're on the compute and largely single-threaded | |
| 16:42:46 | bauzas | I see | |
| 16:43:03 | bauzas | stupid idea, can't we rely on the state of the instance ? | |
| 16:43:13 | dansmith | delete is anything-goes I think | |
| 16:43:27 | bauzas | yeah, that's the original problem | |
| 16:43:32 | dansmith | because that's what we want... | |
| 16:43:41 | bauzas | yup... | |
| 16:44:15 | dansmith | and anything else we build into there is going to be pretty obscure.. "delete always deletes, except stop but not shelve... and either waits or refuses or ..." | |
| 16:44:24 | dansmith | makes the "delete always works" contract a little less clear | |
| 16:44:27 | bauzas | honestly I don't know how to move on with this ask | |
| 16:44:38 | dansmith | tell them that delete means pulling the plug, | |
| 16:44:41 | lyarwood | how about making stop more graceful just for the libvirt driver? | |
| 16:44:41 | bauzas | "ask your orchestration to be smarter ?" | |
| 16:44:46 | gibi | can we add a flag to the delete api saying I-want-a-shutdown-first? | |
| 16:45:00 | dansmith | and you wouldn't do that while waiting for start->shutdown on a physical server, so they should wait for vm_state=STOPPED before delete | |
| 16:45:24 | dansmith | gibi: that makes it a little less obscure, but doesn't eliminate the need to make that orchestration bit work, of course | |
| 16:45:37 | bauzas | yeah | |
| 16:45:39 | lyarwood | artom: there main issue with stop was that it eventually destroys the instance right? | |
| 16:45:47 | bauzas | that doesn't solve the ordering problem | |
| 16:46:07 | lyarwood | artom: and in your change you've suggested that we call shutdown on the domain as an initial step to make this more graceful | |
| 16:46:35 | dansmith | lyarwood: can we call quiesce or something else constant-time before libvirt destroy? | |
| 16:46:43 | dansmith | instead of shutdown, which the guest can block or ignore? | |
| 16:47:31 | bauzas | the root problem is the I/O flushes, right? | |
| 16:47:40 | dansmith | novaclient has a --poll option for some things. "nova stop server --poll; nova delete server" would solve this pretty easy :) | |
| 16:49:01 | melwitt | they're using tripleo/heat but I'm pretty sure heat has dependency or waiting ability | |
| 16:49:09 | dansmith | surely | |
| 16:49:33 | lyarwood | dansmith: quiesce before $domain.shutdown() might help flush things if that's where the guestOS is getting hung up | |
| 16:49:52 | melwitt | their argument has been that stop + insta delete should not break the volume, IIUC | |
| 16:50:00 | lyarwood | waiting assumes we don't kill it before it's finished shutting down | |
| 16:50:14 | dansmith | lyarwood: I meant to make sure journal buffers are written before we nuke the guest | |
| 16:50:36 | dansmith | melwitt: that argument is fine as long as you wait between stop and delete :) | |
| 16:51:01 | dansmith | melwitt: because again, start->shutdown and then pulling the plug before it finishes yields a corrupted disk :) | |