| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-06-17 | |||
| 12:18:14 | sean-k-mooney | i forgot you cant return sting in bash | |
| 12:18:19 | sean-k-mooney | you echo them | |
| 12:19:06 | opendevreview | Merged openstack/nova master: db: Remove dead code https://review.opendev.org/c/openstack/nova/+/786291 | |
| 12:19:33 | opendevreview | Merged openstack/nova master: gate: Remove test_evacuate.sh https://review.opendev.org/c/openstack/nova/+/796523 | |
| 12:50:01 | opendevreview | Rodrigo Barbieri proposed openstack/nova stable/ussuri: Error anti-affinity violation on migrations https://review.opendev.org/c/openstack/nova/+/796719 | |
| 13:06:38 | opendevreview | Merged openstack/nova stable/stein: Improve error log when snapshot fails https://review.opendev.org/c/openstack/nova/+/782962 | |
| 13:06:59 | opendevreview | Merged openstack/nova stable/ussuri: Reproduce bug 1897528 https://review.opendev.org/c/openstack/nova/+/791770 | |
| 13:13:00 | opendevreview | Lee Yarwood proposed openstack/nova master: zuul: Add nova-tox-functional-centos8-py36 job https://review.opendev.org/c/openstack/nova/+/796684 | |
| 13:18:27 | opendevreview | Lee Yarwood proposed openstack/nova master: zuul: Add nova-tox-functional-centos8-py36 job https://review.opendev.org/c/openstack/nova/+/796684 | |
| 13:19:13 | lyarwood | gah! | |
| 13:19:23 | opendevreview | Lee Yarwood proposed openstack/nova master: zuul: Add nova-tox-functional-centos8-py36 job https://review.opendev.org/c/openstack/nova/+/796684 | |
| 13:50:38 | lyarwood | gibi: sean-k-mooney is working on https://review.opendev.org/c/openstack/devstack/+/796826 to hopefully resolve lots of instability | |
| 13:51:25 | sean-k-mooney | i wonder why we are hitting this so much more often recently | |
| 13:53:21 | lyarwood | maybe we are just noticing it more recently, it's an awkward one. | |
| 13:53:54 | sean-k-mooney | ya we also kind of mentally filter out those lines in the log | |
| 13:54:01 | sean-k-mooney | at least i do most of the time | |
| 13:59:16 | lyarwood | right takes some processing of timestamps to even see the issue but most of the time the ultimate test failure is miles away from that | |
| 13:59:40 | lyarwood | sometimes I wish I worked on an easier stack :) | |
| 13:59:45 | sean-k-mooney | lyarwood: gibi its almost finished the check run by the way the current version seams to be working | |
| 13:59:53 | lyarwood | ack yeah I've been watching | |
| 13:59:59 | lyarwood | looking good thus far | |
| 14:04:09 | noonedeadpunk | o/ | |
| 14:04:39 | noonedeadpunk | folks we noticed weird behaviour that you're probably aware about | |
| 14:05:25 | opendevreview | Mohammed Naser proposed openstack/nova stable/wallaby: Allow X-OpenStack-Nova-API-Version header in CORS https://review.opendev.org/c/openstack/nova/+/796860 | |
| 14:06:08 | opendevreview | Mohammed Naser proposed openstack/nova stable/victoria: Allow X-OpenStack-Nova-API-Version header in CORS https://review.opendev.org/c/openstack/nova/+/796861 | |
| 14:06:41 | opendevreview | Mohammed Naser proposed openstack/nova stable/ussuri: Allow X-OpenStack-Nova-API-Version header in CORS https://review.opendev.org/c/openstack/nova/+/796862 | |
| 14:07:19 | opendevreview | Mohammed Naser proposed openstack/nova stable/train: Allow X-OpenStack-Nova-API-Version header in CORS https://review.opendev.org/c/openstack/nova/+/796863 | |
| 14:07:38 | noonedeadpunk | So, algorithm is kind of the following: 1. HV goes down. 2. VM is sent Shutdown (or any other request). 3. Then VM is in `powering-off` state, but it needs to be evacuated. So reset-state is issued and evacuate is processed. Now VM is running on another HV. 4 When original HV goes up it process messages that were issued while it was down and powers off VM that was evacuated and owned by another HV atm | |
| 14:08:22 | noonedeadpunk | I have a feeling that if node is not owning VM it should not have ability to influence it even if it has some commands in queue? | |
| 14:08:46 | noonedeadpunk | and maybe you have some guess where in code worth looking for this? | |
| 14:10:15 | lyarwood | so the compute manager that gets the cast in this case isn't doing any checks to ensure the instance is still on that host | |
| 14:10:41 | lyarwood | I guess it's a valid thing to do for any operations using casts | |
| 14:10:50 | noonedeadpunk | yeah, I expect smth like that is happening. But not super familiar with codebase :( | |
| 14:11:02 | sean-k-mooney | noonedeadpunk: why are you doing reset-state in your evacuate workflow | |
| 14:11:11 | sean-k-mooney | noonedeadpunk: you should not be doing reset state first | |
| 14:11:35 | noonedeadpunk | well, otherwise it can't be evacuated with `ERROR (Conflict): Cannot 'evacuate' instance e46404b1-e6e1-4d22-9f8f-12d6f51b55ae while it is in task_state powering-off` | |
| 14:11:57 | sean-k-mooney | hum | |
| 14:12:01 | sean-k-mooney | i see | |
| 14:12:07 | noonedeadpunk | Is there any other proper way to do evacuate? | |
| 14:12:21 | gibi | lyarwood, sean-k-mooney thanks. I'm happy to see that this week a lot of us focused on stabilizing the gate. | |
| 14:12:29 | noonedeadpunk | I mean technicaly we could wait until node goes up, but it might be days theoretically? | |
| 14:12:49 | lyarwood | tbh I think we should allow evacuate if the instance is powering-off | |
| 14:12:55 | lyarwood | either way the src compute is dead | |
| 14:13:07 | sean-k-mooney | yep i ws thinking the same | |
| 14:13:10 | noonedeadpunk | but it won't resolve original issue though | |
| 14:13:29 | noonedeadpunk | as then evacuated instance would be shot anyway | |
| 14:13:42 | lyarwood | well it shouldn't kill the instance on the dest | |
| 14:13:46 | noonedeadpunk | (but agree it's super valid to allow evacuate) | |
| 14:13:56 | lyarwood | the cast to shutdown the original instance on the original host should fail | |
| 14:14:12 | noonedeadpunk | fwiw it's on Victoria | |
| 14:14:22 | lyarwood | but that has the potential of moving the instance into an ERROR state | |
| 14:14:54 | lyarwood | a simple decorator to check that instance.host points at the current host would work here | |
| 14:15:00 | lyarwood | it might not work everywhere we cast | |
| 14:15:07 | lyarwood | but in this example it's fine | |
| 14:15:26 | lyarwood | noonedeadpunk: did you have a bug for this? | |
| 14:15:53 | noonedeadpunk | nope, not yet, but will submit one :) | |
| 14:16:10 | lyarwood | awesome thanks | |
| 14:16:26 | noonedeadpunk | or maybe even two... | |
| 14:17:22 | gibi | there is a recent bug asking for evacuating in soft-delete state https://bugs.launchpad.net/nova/+bug/1932126 | |
| 14:17:59 | sean-k-mooney | we spoke about allowing it in other state at the ptg | |
| 14:18:11 | sean-k-mooney | like paused/suspended | |
| 14:18:22 | noonedeadpunk | well... soft delete is really corner case imo... | |
| 14:18:27 | lyarwood | I'm not sure that soft-delete makes sense | |
| 14:18:28 | bauzas | soft-deleted in the Nova API or in the database ? | |
| 14:18:29 | sean-k-mooney | im not sure soft delete makes much sense | |
| 14:18:38 | noonedeadpunk | once you will evacuate it it would be already time to delete instance... | |
| 14:18:38 | sean-k-mooney | lyarwood: :) | |
| 14:19:00 | bauzas | ah, this | |
| 14:19:06 | sean-k-mooney | i guess the use case is to undelete it | |
| 14:19:11 | bauzas | (nova api soft delete, that's it) | |
| 14:19:21 | bauzas | well, i do understand the concern from an operator pov | |
| 14:19:42 | lyarwood | TIL we can do that | |
| 14:19:43 | bauzas | if you wanna evacuate, you're in a rush | |
| 14:19:43 | sean-k-mooney | i would be ok with making undelete work when the host is down and then allow evacuate | |
| 14:19:47 | lyarwood | for some reason I didn't think we had a way back | |
| 14:19:58 | sean-k-mooney | btu eveauate on a soft-deleted instace form me would have to undelete it | |
| 14:20:08 | bauzas | sean-k-mooney: or we could just not rebuild the instance | |
| 14:20:24 | bauzas | it's soft deleted in the source host | |
| 14:20:45 | bauzas | so the target should just not rebuild the instance | |
| 14:20:48 | sean-k-mooney | bauzas: so rebuild when the hosts is down | |
| 14:20:57 | bauzas | nah | |
| 14:21:09 | bauzas | not rebuild the soft-deleted instance | |
| 14:21:19 | sean-k-mooney | but why would we keep it deleted | |
| 14:21:22 | bauzas | but the evacuate API should woirk | |
| 14:21:34 | sean-k-mooney | not for soft deleted | |
| 14:21:41 | bauzas | because it's already deleted | |
| 14:21:57 | bauzas | someone asked the instance to be deleted | |
| 14:22:05 | bauzas | then the host got an issue | |
| 14:22:14 | sean-k-mooney | yep | |
| 14:22:16 | bauzas | so the operator would recreate the instances in a target | |
| 14:22:26 | sean-k-mooney | at which point i think we should jsut treat it as if it has been deelted fully | |
| 14:22:34 | bauzas | sean-k-mooney: that's my point | |
| 14:22:53 | bauzas | when saying to not rebuild it on the target | |
| 14:23:05 | bauzas | but here the API doesn't work | |
| 14:23:28 | bauzas | so, we should provide a HTTP200 for an soft-delete evacuate | |
| 14:23:35 | bauzas | but not recreating it | |
| 14:24:13 | bauzas | anyway, needs to get my kids from the school | |
| 14:27:21 | gibi | I think the use case could be | |
| 14:27:32 | gibi | 1) user deletes the VM | |
| 14:27:46 | gibi | 2) soft deleting is enabled so the VM is just soft-deleted | |