| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-09-07 | |||
| 16:28:36 | bauzas | took a bit of time to load the context | |
| 16:28:58 | bauzas | we relied on task_state for knowing whether we were having an user instance action | |
| 16:29:19 | bauzas | but given it was an issue if we want to evacuate, we removed the conditional | |
| 16:29:28 | gibi | yeah | |
| 16:29:46 | bauzas | but now we dunno whether a current task state is because of an user action or because an evacuation is starting, right? | |
| 16:30:53 | gibi | right, basically we don't know if a non empty task_state at the start of evacuation means an action that cannot progress because it is against the dead compute and therefore allowed to evacuate, or it measn an action that is progressing already on another compute for this VM like a previous evac | |
| 16:31:13 | bauzas | hmmm OK | |
| 16:31:18 | gibi | but also a previous evac can be against the dead compute | |
| 16:31:19 | bauzas | (thinking) | |
| 16:31:48 | gibi | so in theory you might want to re-try evacuation as the previous evac is not progressing as that compute died during evac | |
| 16:31:50 | bauzas | I guess we can't look at the VM state thn ? | |
| 16:32:26 | gibi | not fully mapped the states, but I think you want to be able to evac from ACTIVE, STOPPED, ERROR | |
| 16:32:30 | bauzas | and we don't have a Migration object when evacuating ? | |
| 16:32:42 | bauzas | because we could look at it | |
| 16:32:51 | gibi | we have but we can have multiple unfinished migration object with type evac | |
| 16:33:06 | gibi | having a migration object does not mean the action of that migration can progress | |
| 16:33:18 | bauzas | hmmmm | |
| 16:33:21 | gibi | maybe the dest or the source of the migration is dead and you want to evac :) | |
| 16:33:53 | gibi | so a perfect solution is impossible I think | |
| 16:33:58 | gibi | we need to compromise some requirements | |
| 16:34:15 | tkajinam | the most safe approach might be add new api microversion to require task_state=None and require operators to do reset-state | |
| 16:34:21 | tkajinam | though I know that's not something operators may like | |
| 16:34:45 | gibi | I will never ask an operator in our do to do a reset-state | |
| 16:34:58 | gibi | that is like asking for a bigger problem because you had a small one :) | |
| 16:35:19 | gibi | s/in our do/ in our doc/ | |
| 16:35:33 | tkajinam | hmm ok. I agree | |
| 16:36:01 | gibi | I mean I spent so many time in customer cases where the deployer used reset-state and created a big mess | |
| 16:36:12 | gibi | maybe, a smarter reset-state... | |
| 16:36:23 | tkajinam | I know :-) | |
| 16:36:24 | gibi | but that is probably the same problem of deciding what action can be reset safely | |
| 16:36:50 | gibi | the same task_state can be ignored in evac precondition as well | |
| 16:37:20 | bauzas | are we already checking in the API evacuation API whether an evacuation is already run ? | |
| 16:38:22 | gibi | we checked in the past by looking at the task_state, but we removed that check effectively when we started allowing to evacuate from task_state=powering_off to support the above use cae | |
| 16:38:26 | gibi | case | |
| 16:39:54 | bauzas | https://github.com/openstack/nova/blob/master/nova/compute/api.py#L5747 | |
| 16:40:08 | gibi | btw task_state=power_off can happen *during* an evac, as we started evacuating to STOPed state in microvesrion 2.95 | |
| 16:40:32 | bauzas | here, we ask for a task state to be None, right? | |
| 16:40:59 | gibi | bauzas: that task_state=None means do not check task_state, the code that expects an empty task_state is expected_task_state=(None,) | |
| 16:41:03 | gibi | or [None] | |
| 16:41:05 | bauzas | sorry, yeah | |
| 16:41:16 | bauzas | so we could say that we expect no task state | |
| 16:41:31 | gibi | no we cannot as that breaks the evac from task_state=powering_off case | |
| 16:41:33 | bauzas | but if we have an exception, we could look at the task state | |
| 16:41:36 | gibi | we support since 2022 | |
| 16:42:14 | bauzas | I mean, if we get a task state here, we can look whether this is due to an already existing rebuild task | |
| 16:42:23 | gibi | the problem is basically what task_state is allowed and what does not | |
| 16:42:24 | bauzas | if so, we can say "mmmh, nope" | |
| 16:43:02 | gibi | I don't have a list of allowable task_states | |
| 16:43:19 | gibi | I don't even have a list of definietly rejectable task_states | |
| 16:43:42 | gibi | as rebuild* task_state could mean an ongoing evac, or a failed evac that you want to retry | |
| 16:43:55 | gibi | where failed, is failed do the dest compute is dead | |
| 16:44:07 | gibi | s/do the/ do to/ | |
| 16:44:17 | gibi | s/do the/ due to/ | |
| 16:44:59 | bauzas | oh right the problem is not only when concurrent evacuation requests, this is also about ensuring we can ask for evac again if it fails | |
| 16:45:34 | bauzas | but if the evac failed, then the instance is not in ACTIVE vm state, right ? | |
| 16:45:40 | bauzas | I have to look at the code | |
| 16:46:02 | bauzas | I don't think we move the instance back to ACTIVE if we fail on evac | |
| 16:46:14 | bauzas | (compared to migrations for example) | |
| 16:46:29 | gibi | I guess I can propose PTG topic :) | |
| 16:46:29 | gibi | or write a book about distributes systems :) | |
| 16:46:58 | tkajinam | if evacuation "fails" normally it may be easy though if compute node dies in the middle then evacuation may get stuck | |
| 16:47:07 | tkajinam | can we check state of the dest compute node to determine whether the current evac can progress ? | |
| 16:47:40 | opendevreview | Ashish Gupta proposed openstack/nova master: Fix init_host crash migration test under native threading https://review.opendev.org/c/openstack/nova/+/1001369 | |
| 16:48:04 | gibi | bauzas: evac is possible from ERROR state :) | |
| 16:48:23 | bauzas | doh | |
| 16:48:39 | gibi | tkajinam: the dest compute is not well defined unfortunately | |
| 16:48:55 | bauzas | gibi: if you write a book about distributed systems, you wouldn't be the first :) | |
| 16:49:18 | gibi | there is instance.host that evacuate sets pretty late. There is the migration object dest host but we can have multiple unfinished evacuation migration object with different dests | |
| 16:49:25 | bauzas | I guess this sounds a PTG discussion indeed | |
| 16:49:33 | tkajinam | yeah | |
| 16:49:44 | gibi | bauzas: mine will solve it! | |
| 16:49:45 | gibi | :) | |
| 16:49:52 | bauzas | hah then I can review | |
| 16:50:07 | bauzas | provided you're not proposing a state machine :) | |
| 16:50:07 | gibi | you will get a dedicate pre-release copy of it for sure | |
| 16:50:25 | gibi | "a" state machine that is short sighted :) | |
| 16:50:51 | bauzas | if someone says "etcd will solve that", I swear I will find you | |
| 16:50:54 | gibi | OK I think I will propose a ptg topic | |
| 16:51:06 | Uggla | etcd will solve that :) | |
| 16:51:12 | bauzas | I'll find you | |
| 16:51:17 | bauzas | and I know your home | |
| 16:51:17 | gibi | there are "better" (mean newer) trials than etcd :) | |
| 16:51:17 | Uggla | ;) | |
| 16:51:45 | gibi | anyhow I think I'm done here. mic drop | |
| 16:51:52 | bauzas | gibi: so, are you asking for reviews about your WIP or do we punt that to PTG? | |
| 16:52:21 | gibi | bauzas: feel free to read the commit message of my WIP and my last comment on the bug | |
| 16:52:25 | bauzas | it will require a substantial amount of context loading, but that sounds a nasty bug tho | |
| 16:52:39 | gibi | but we won't have solution on RC1 week | |
| 16:52:42 | bauzas | gibi: OK'll try | |
| 16:52:47 | Uggla | To my mind that sounds complex enough to discuss to the PTG. | |
| 16:52:48 | gibi | so we can continue after H release | |
| 16:52:57 | gibi | probably on the PTG (depending of downstream push) | |
| 16:53:13 | Uggla | ok | |
| 16:53:28 | bauzas | my concern is that a PTG discussion will probably require some prerequisite effort for context loading | |
| 16:53:45 | gibi | I will create more context if this will end up being a PTG topic | |
| 16:53:53 | bauzas | cool | |
| 16:53:55 | gibi | I cannot help loading it though | |
| 16:54:07 | Uggla | anything else to discuss in the last 5mn, if not I think we are done. | |
| 16:54:14 | gibi | I don't yet have a remote exploit in your head | |
| 16:54:35 | bauzas | well, I think I know how we manage our VM and task states, but I'm more concerned by the quirks we have in code (all the nasty assumptions we have as code conditionals) | |