| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-09-07 | |||
| 16:21:48 | Uggla | #topic Open discussion | |
| 16:22:02 | Uggla | on topic from gibi | |
| 16:22:02 | Uggla | (gibi): I need technical guidance what to do with https://bugs.launchpad.net/nova/+bug/2166474 (Uggla: I will describe the case on the meeting) | |
| 16:24:05 | gibi | o/ | |
| 16:24:35 | gibi | so we have that bug, it describe a nasty race where nova allows two evacuation action to overlap on the same VM at the same time | |
| 16:24:48 | gibi | then today I realized that we started allowing that couple of years ago | |
| 16:24:58 | gibi | by allowing evacuation from a non empty task_state | |
| 16:25:03 | gibi | to support things like | |
| 16:25:12 | gibi | 1. compute with a VM dies | |
| 16:25:32 | gibi | 2. owner try to restart the VM, VM goes to powering_off task_state that never progresses | |
| 16:25:51 | gibi | 3. operator realizes that the compute is dead an wants to evacuate the VM | |
| 16:26:19 | gibi | to support this nova allows triggering evacuation with task_state not empty | |
| 16:26:39 | gibi | but task_state is our lock to avoid two operation to overlap on the same VM | |
| 16:27:25 | gibi | obviously I can develop a patch that allow evacuation to start only from non rebuilding* task_state | |
| 16:27:36 | gibi | but that only solves the currently proven overlap | |
| 16:28:01 | gibi | I'm a bit hesitant about all the other task states what should be allowed and what should not | |
| 16:28:12 | gibi | so ideas folks? | |
| 16:28:24 | gibi | (here or offline in the bug) | |
| 16:28:27 | bauzas | I see | |
| 16:28:36 | bauzas | took a bit of time to load the context | |
| 16:28:58 | bauzas | we relied on task_state for knowing whether we were having an user instance action | |
| 16:29:19 | bauzas | but given it was an issue if we want to evacuate, we removed the conditional | |
| 16:29:28 | gibi | yeah | |
| 16:29:46 | bauzas | but now we dunno whether a current task state is because of an user action or because an evacuation is starting, right? | |
| 16:30:53 | gibi | right, basically we don't know if a non empty task_state at the start of evacuation means an action that cannot progress because it is against the dead compute and therefore allowed to evacuate, or it measn an action that is progressing already on another compute for this VM like a previous evac | |
| 16:31:13 | bauzas | hmmm OK | |
| 16:31:18 | gibi | but also a previous evac can be against the dead compute | |
| 16:31:19 | bauzas | (thinking) | |
| 16:31:48 | gibi | so in theory you might want to re-try evacuation as the previous evac is not progressing as that compute died during evac | |
| 16:31:50 | bauzas | I guess we can't look at the VM state thn ? | |
| 16:32:26 | gibi | not fully mapped the states, but I think you want to be able to evac from ACTIVE, STOPPED, ERROR | |
| 16:32:30 | bauzas | and we don't have a Migration object when evacuating ? | |
| 16:32:42 | bauzas | because we could look at it | |
| 16:32:51 | gibi | we have but we can have multiple unfinished migration object with type evac | |
| 16:33:06 | gibi | having a migration object does not mean the action of that migration can progress | |
| 16:33:18 | bauzas | hmmmm | |
| 16:33:21 | gibi | maybe the dest or the source of the migration is dead and you want to evac :) | |
| 16:33:53 | gibi | so a perfect solution is impossible I think | |
| 16:33:58 | gibi | we need to compromise some requirements | |
| 16:34:15 | tkajinam | the most safe approach might be add new api microversion to require task_state=None and require operators to do reset-state | |
| 16:34:21 | tkajinam | though I know that's not something operators may like | |
| 16:34:45 | gibi | I will never ask an operator in our do to do a reset-state | |
| 16:34:58 | gibi | that is like asking for a bigger problem because you had a small one :) | |
| 16:35:19 | gibi | s/in our do/ in our doc/ | |
| 16:35:33 | tkajinam | hmm ok. I agree | |
| 16:36:01 | gibi | I mean I spent so many time in customer cases where the deployer used reset-state and created a big mess | |
| 16:36:12 | gibi | maybe, a smarter reset-state... | |
| 16:36:23 | tkajinam | I know :-) | |
| 16:36:24 | gibi | but that is probably the same problem of deciding what action can be reset safely | |
| 16:36:50 | gibi | the same task_state can be ignored in evac precondition as well | |
| 16:37:20 | bauzas | are we already checking in the API evacuation API whether an evacuation is already run ? | |
| 16:38:22 | gibi | we checked in the past by looking at the task_state, but we removed that check effectively when we started allowing to evacuate from task_state=powering_off to support the above use cae | |
| 16:38:26 | gibi | case | |
| 16:39:54 | bauzas | https://github.com/openstack/nova/blob/master/nova/compute/api.py#L5747 | |
| 16:40:08 | gibi | btw task_state=power_off can happen *during* an evac, as we started evacuating to STOPed state in microvesrion 2.95 | |
| 16:40:32 | bauzas | here, we ask for a task state to be None, right? | |
| 16:40:59 | gibi | bauzas: that task_state=None means do not check task_state, the code that expects an empty task_state is expected_task_state=(None,) | |
| 16:41:03 | gibi | or [None] | |
| 16:41:05 | bauzas | sorry, yeah | |
| 16:41:16 | bauzas | so we could say that we expect no task state | |
| 16:41:31 | gibi | no we cannot as that breaks the evac from task_state=powering_off case | |
| 16:41:33 | bauzas | but if we have an exception, we could look at the task state | |
| 16:41:36 | gibi | we support since 2022 | |
| 16:42:14 | bauzas | I mean, if we get a task state here, we can look whether this is due to an already existing rebuild task | |
| 16:42:23 | gibi | the problem is basically what task_state is allowed and what does not | |
| 16:42:24 | bauzas | if so, we can say "mmmh, nope" | |
| 16:43:02 | gibi | I don't have a list of allowable task_states | |
| 16:43:19 | gibi | I don't even have a list of definietly rejectable task_states | |
| 16:43:42 | gibi | as rebuild* task_state could mean an ongoing evac, or a failed evac that you want to retry | |
| 16:43:55 | gibi | where failed, is failed do the dest compute is dead | |
| 16:44:07 | gibi | s/do the/ do to/ | |
| 16:44:17 | gibi | s/do the/ due to/ | |
| 16:44:59 | bauzas | oh right the problem is not only when concurrent evacuation requests, this is also about ensuring we can ask for evac again if it fails | |
| 16:45:34 | bauzas | but if the evac failed, then the instance is not in ACTIVE vm state, right ? | |
| 16:45:40 | bauzas | I have to look at the code | |
| 16:46:02 | bauzas | I don't think we move the instance back to ACTIVE if we fail on evac | |
| 16:46:14 | bauzas | (compared to migrations for example) | |
| 16:46:29 | gibi | I guess I can propose PTG topic :) | |
| 16:46:29 | gibi | or write a book about distributes systems :) | |
| 16:46:58 | tkajinam | if evacuation "fails" normally it may be easy though if compute node dies in the middle then evacuation may get stuck | |
| 16:47:07 | tkajinam | can we check state of the dest compute node to determine whether the current evac can progress ? | |
| 16:47:40 | opendevreview | Ashish Gupta proposed openstack/nova master: Fix init_host crash migration test under native threading https://review.opendev.org/c/openstack/nova/+/1001369 | |
| 16:48:04 | gibi | bauzas: evac is possible from ERROR state :) | |
| 16:48:23 | bauzas | doh | |
| 16:48:39 | gibi | tkajinam: the dest compute is not well defined unfortunately | |
| 16:48:55 | bauzas | gibi: if you write a book about distributed systems, you wouldn't be the first :) | |
| 16:49:18 | gibi | there is instance.host that evacuate sets pretty late. There is the migration object dest host but we can have multiple unfinished evacuation migration object with different dests | |
| 16:49:25 | bauzas | I guess this sounds a PTG discussion indeed | |
| 16:49:33 | tkajinam | yeah | |
| 16:49:44 | gibi | bauzas: mine will solve it! | |
| 16:49:45 | gibi | :) | |
| 16:49:52 | bauzas | hah then I can review | |
| 16:50:07 | bauzas | provided you're not proposing a state machine :) | |
| 16:50:07 | gibi | you will get a dedicate pre-release copy of it for sure | |
| 16:50:25 | gibi | "a" state machine that is short sighted :) | |
| 16:50:51 | bauzas | if someone says "etcd will solve that", I swear I will find you | |
| 16:50:54 | gibi | OK I think I will propose a ptg topic | |
| 16:51:06 | Uggla | etcd will solve that :) | |
| 16:51:12 | bauzas | I'll find you | |
| 16:51:17 | bauzas | and I know your home | |