| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-06-17 | |||
| 14:08:46 | noonedeadpunk | and maybe you have some guess where in code worth looking for this? | |
| 14:10:15 | lyarwood | so the compute manager that gets the cast in this case isn't doing any checks to ensure the instance is still on that host | |
| 14:10:41 | lyarwood | I guess it's a valid thing to do for any operations using casts | |
| 14:10:50 | noonedeadpunk | yeah, I expect smth like that is happening. But not super familiar with codebase :( | |
| 14:11:02 | sean-k-mooney | noonedeadpunk: why are you doing reset-state in your evacuate workflow | |
| 14:11:11 | sean-k-mooney | noonedeadpunk: you should not be doing reset state first | |
| 14:11:35 | noonedeadpunk | well, otherwise it can't be evacuated with `ERROR (Conflict): Cannot 'evacuate' instance e46404b1-e6e1-4d22-9f8f-12d6f51b55ae while it is in task_state powering-off` | |
| 14:11:57 | sean-k-mooney | hum | |
| 14:12:01 | sean-k-mooney | i see | |
| 14:12:07 | noonedeadpunk | Is there any other proper way to do evacuate? | |
| 14:12:21 | gibi | lyarwood, sean-k-mooney thanks. I'm happy to see that this week a lot of us focused on stabilizing the gate. | |
| 14:12:29 | noonedeadpunk | I mean technicaly we could wait until node goes up, but it might be days theoretically? | |
| 14:12:49 | lyarwood | tbh I think we should allow evacuate if the instance is powering-off | |
| 14:12:55 | lyarwood | either way the src compute is dead | |
| 14:13:07 | sean-k-mooney | yep i ws thinking the same | |
| 14:13:10 | noonedeadpunk | but it won't resolve original issue though | |
| 14:13:29 | noonedeadpunk | as then evacuated instance would be shot anyway | |
| 14:13:42 | lyarwood | well it shouldn't kill the instance on the dest | |
| 14:13:46 | noonedeadpunk | (but agree it's super valid to allow evacuate) | |
| 14:13:56 | lyarwood | the cast to shutdown the original instance on the original host should fail | |
| 14:14:12 | noonedeadpunk | fwiw it's on Victoria | |
| 14:14:22 | lyarwood | but that has the potential of moving the instance into an ERROR state | |
| 14:14:54 | lyarwood | a simple decorator to check that instance.host points at the current host would work here | |
| 14:15:00 | lyarwood | it might not work everywhere we cast | |
| 14:15:07 | lyarwood | but in this example it's fine | |
| 14:15:26 | lyarwood | noonedeadpunk: did you have a bug for this? | |
| 14:15:53 | noonedeadpunk | nope, not yet, but will submit one :) | |
| 14:16:10 | lyarwood | awesome thanks | |
| 14:16:26 | noonedeadpunk | or maybe even two... | |
| 14:17:22 | gibi | there is a recent bug asking for evacuating in soft-delete state https://bugs.launchpad.net/nova/+bug/1932126 | |
| 14:17:59 | sean-k-mooney | we spoke about allowing it in other state at the ptg | |
| 14:18:11 | sean-k-mooney | like paused/suspended | |
| 14:18:22 | noonedeadpunk | well... soft delete is really corner case imo... | |
| 14:18:27 | lyarwood | I'm not sure that soft-delete makes sense | |
| 14:18:28 | bauzas | soft-deleted in the Nova API or in the database ? | |
| 14:18:29 | sean-k-mooney | im not sure soft delete makes much sense | |
| 14:18:38 | noonedeadpunk | once you will evacuate it it would be already time to delete instance... | |
| 14:18:38 | sean-k-mooney | lyarwood: :) | |
| 14:19:00 | bauzas | ah, this | |
| 14:19:06 | sean-k-mooney | i guess the use case is to undelete it | |
| 14:19:11 | bauzas | (nova api soft delete, that's it) | |
| 14:19:21 | bauzas | well, i do understand the concern from an operator pov | |
| 14:19:42 | lyarwood | TIL we can do that | |
| 14:19:43 | bauzas | if you wanna evacuate, you're in a rush | |
| 14:19:43 | sean-k-mooney | i would be ok with making undelete work when the host is down and then allow evacuate | |
| 14:19:47 | lyarwood | for some reason I didn't think we had a way back | |
| 14:19:58 | sean-k-mooney | btu eveauate on a soft-deleted instace form me would have to undelete it | |
| 14:20:08 | bauzas | sean-k-mooney: or we could just not rebuild the instance | |
| 14:20:24 | bauzas | it's soft deleted in the source host | |
| 14:20:45 | bauzas | so the target should just not rebuild the instance | |
| 14:20:48 | sean-k-mooney | bauzas: so rebuild when the hosts is down | |
| 14:20:57 | bauzas | nah | |
| 14:21:09 | bauzas | not rebuild the soft-deleted instance | |
| 14:21:19 | sean-k-mooney | but why would we keep it deleted | |
| 14:21:22 | bauzas | but the evacuate API should woirk | |
| 14:21:34 | sean-k-mooney | not for soft deleted | |
| 14:21:41 | bauzas | because it's already deleted | |
| 14:21:57 | bauzas | someone asked the instance to be deleted | |
| 14:22:05 | bauzas | then the host got an issue | |
| 14:22:14 | sean-k-mooney | yep | |
| 14:22:16 | bauzas | so the operator would recreate the instances in a target | |
| 14:22:26 | sean-k-mooney | at which point i think we should jsut treat it as if it has been deelted fully | |
| 14:22:34 | bauzas | sean-k-mooney: that's my point | |
| 14:22:53 | bauzas | when saying to not rebuild it on the target | |
| 14:23:05 | bauzas | but here the API doesn't work | |
| 14:23:28 | bauzas | so, we should provide a HTTP200 for an soft-delete evacuate | |
| 14:23:35 | bauzas | but not recreating it | |
| 14:24:13 | bauzas | anyway, needs to get my kids from the school | |
| 14:27:21 | gibi | I think the use case could be | |
| 14:27:32 | gibi | 1) user deletes the VM | |
| 14:27:46 | gibi | 2) soft deleting is enabled so the VM is just soft-deleted | |
| 14:27:51 | gibi | 3) host goes down | |
| 14:28:05 | gibi | 4) user realizes that there was a mistake deleting the VM and calls restore | |
| 14:28:47 | gibi | 5) restore fails as the host is down | |
| 14:28:52 | gibi | what to do nwo | |
| 14:28:53 | gibi | now | |
| 14:28:57 | sean-k-mooney | so the user cannot know if soft delete is avaiable or the time to restore | |
| 14:30:30 | sean-k-mooney | to me we can make the restorce call work when the hsot is down but we should not change the evacuate beahvior IMO | |
| 14:30:53 | sean-k-mooney | so have restore jsut undelete it in the db | |
| 14:31:06 | gibi | OK so then the restore + evacuate would work | |
| 14:31:11 | sean-k-mooney | yep | |
| 14:31:17 | gibi | that is acceptable to me | |
| 14:31:30 | gibi | but | |
| 14:31:56 | gibi | soft-delete, host down, restore (undelete in db), host up sequence would lead to inconsistency | |
| 14:32:27 | sean-k-mooney | well when the host comes up it will need to check the db state | |
| 14:32:41 | sean-k-mooney | before completing the soft delete action correct | |
| 14:33:16 | sean-k-mooney | e.g. when the compute comes back up it shoudl see the vm was evacuated | |
| 14:33:21 | sean-k-mooney | or if it was jsut restored | |
| 14:33:32 | gibi | there was no evacuation in this sequence | |
| 14:33:34 | sean-k-mooney | tehn it would see that its been restored in the db | |
| 14:33:46 | sean-k-mooney | so it would need to hanel that | |
| 14:34:48 | gibi | today restore sets the power state back to running. the db only restore would set it to shutoff? | |
| 14:36:13 | gibi | anyhow I have to run | |
| 14:36:35 | gibi | my days are soo random, I don't feel productive | |
| 14:37:35 | noonedeadpunk | https://bugs.launchpad.net/nova/+bug/1932326 | |
| 14:46:50 | sean-k-mooney | noonedeadpunk: thanks | |
| 14:47:21 | sean-k-mooney | noonedeadpunk: just going to triage this quickly how impactful is to you production wise | |
| 14:47:47 | sean-k-mooney | im leaning towards medium or low since there is no data lose but there is a workload outage | |
| 14:48:02 | sean-k-mooney | noonedeadpunk: i.e. you can just fix it by starting the vm again | |
| 14:48:45 | noonedeadpunk | I'd say it's closer to medium I guess, because as for public cloud provider it's hard to explain why customer VM wents down in a day after previous outage | |