| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-06-17 | |||
| 14:19:00 | bauzas | ah, this | |
| 14:19:06 | sean-k-mooney | i guess the use case is to undelete it | |
| 14:19:11 | bauzas | (nova api soft delete, that's it) | |
| 14:19:21 | bauzas | well, i do understand the concern from an operator pov | |
| 14:19:42 | lyarwood | TIL we can do that | |
| 14:19:43 | bauzas | if you wanna evacuate, you're in a rush | |
| 14:19:43 | sean-k-mooney | i would be ok with making undelete work when the host is down and then allow evacuate | |
| 14:19:47 | lyarwood | for some reason I didn't think we had a way back | |
| 14:19:58 | sean-k-mooney | btu eveauate on a soft-deleted instace form me would have to undelete it | |
| 14:20:08 | bauzas | sean-k-mooney: or we could just not rebuild the instance | |
| 14:20:24 | bauzas | it's soft deleted in the source host | |
| 14:20:45 | bauzas | so the target should just not rebuild the instance | |
| 14:20:48 | sean-k-mooney | bauzas: so rebuild when the hosts is down | |
| 14:20:57 | bauzas | nah | |
| 14:21:09 | bauzas | not rebuild the soft-deleted instance | |
| 14:21:19 | sean-k-mooney | but why would we keep it deleted | |
| 14:21:22 | bauzas | but the evacuate API should woirk | |
| 14:21:34 | sean-k-mooney | not for soft deleted | |
| 14:21:41 | bauzas | because it's already deleted | |
| 14:21:57 | bauzas | someone asked the instance to be deleted | |
| 14:22:05 | bauzas | then the host got an issue | |
| 14:22:14 | sean-k-mooney | yep | |
| 14:22:16 | bauzas | so the operator would recreate the instances in a target | |
| 14:22:26 | sean-k-mooney | at which point i think we should jsut treat it as if it has been deelted fully | |
| 14:22:34 | bauzas | sean-k-mooney: that's my point | |
| 14:22:53 | bauzas | when saying to not rebuild it on the target | |
| 14:23:05 | bauzas | but here the API doesn't work | |
| 14:23:28 | bauzas | so, we should provide a HTTP200 for an soft-delete evacuate | |
| 14:23:35 | bauzas | but not recreating it | |
| 14:24:13 | bauzas | anyway, needs to get my kids from the school | |
| 14:27:21 | gibi | I think the use case could be | |
| 14:27:32 | gibi | 1) user deletes the VM | |
| 14:27:46 | gibi | 2) soft deleting is enabled so the VM is just soft-deleted | |
| 14:27:51 | gibi | 3) host goes down | |
| 14:28:05 | gibi | 4) user realizes that there was a mistake deleting the VM and calls restore | |
| 14:28:47 | gibi | 5) restore fails as the host is down | |
| 14:28:52 | gibi | what to do nwo | |
| 14:28:53 | gibi | now | |
| 14:28:57 | sean-k-mooney | so the user cannot know if soft delete is avaiable or the time to restore | |
| 14:30:30 | sean-k-mooney | to me we can make the restorce call work when the hsot is down but we should not change the evacuate beahvior IMO | |
| 14:30:53 | sean-k-mooney | so have restore jsut undelete it in the db | |
| 14:31:06 | gibi | OK so then the restore + evacuate would work | |
| 14:31:11 | sean-k-mooney | yep | |
| 14:31:17 | gibi | that is acceptable to me | |
| 14:31:30 | gibi | but | |
| 14:31:56 | gibi | soft-delete, host down, restore (undelete in db), host up sequence would lead to inconsistency | |
| 14:32:27 | sean-k-mooney | well when the host comes up it will need to check the db state | |
| 14:32:41 | sean-k-mooney | before completing the soft delete action correct | |
| 14:33:16 | sean-k-mooney | e.g. when the compute comes back up it shoudl see the vm was evacuated | |
| 14:33:21 | sean-k-mooney | or if it was jsut restored | |
| 14:33:32 | gibi | there was no evacuation in this sequence | |
| 14:33:34 | sean-k-mooney | tehn it would see that its been restored in the db | |
| 14:33:46 | sean-k-mooney | so it would need to hanel that | |
| 14:34:48 | gibi | today restore sets the power state back to running. the db only restore would set it to shutoff? | |
| 14:36:13 | gibi | anyhow I have to run | |
| 14:36:35 | gibi | my days are soo random, I don't feel productive | |
| 14:37:35 | noonedeadpunk | https://bugs.launchpad.net/nova/+bug/1932326 | |
| 14:46:50 | sean-k-mooney | noonedeadpunk: thanks | |
| 14:47:21 | sean-k-mooney | noonedeadpunk: just going to triage this quickly how impactful is to you production wise | |
| 14:47:47 | sean-k-mooney | im leaning towards medium or low since there is no data lose but there is a workload outage | |
| 14:48:02 | sean-k-mooney | noonedeadpunk: i.e. you can just fix it by starting the vm again | |
| 14:48:45 | noonedeadpunk | I'd say it's closer to medium I guess, because as for public cloud provider it's hard to explain why customer VM wents down in a day after previous outage | |
| 14:49:00 | noonedeadpunk | and you can start it when you own vm or monitor it | |
| 14:49:52 | noonedeadpunk | but if it's not your VM it's hard to even know that it went down | |
| 14:50:17 | sean-k-mooney | yep | |
| 14:50:19 | noonedeadpunk | As current workaround we will probably attempt to flush queue for compute that went down... | |
| 14:50:33 | noonedeadpunk | but it's so nasty imo | |
| 14:50:35 | sean-k-mooney | more then likely it will cause the custoemr to notice a failure of the vm restore it and file a ticket | |
| 14:50:48 | noonedeadpunk | that's exactly what has happened :) | |
| 14:51:37 | noonedeadpunk | that's pretty much a corner case though as well | |
| 14:51:55 | sean-k-mooney | noonedeadpunk: so if we allow evac in the powering-off state restoring it to shutdown would make sense right | |
| 14:51:59 | sean-k-mooney | rather then active | |
| 14:52:11 | noonedeadpunk | yes, totally | |
| 14:52:29 | noonedeadpunk | and powering-on to active :) | |
| 14:52:54 | sean-k-mooney | we had disscced doing that for vms in suspend and pause so including powering-off in that list i think is consitent | |
| 14:52:57 | noonedeadpunk | (but it's harder to imagine happening) | |
| 14:53:24 | sean-k-mooney | powering-on to active would also make sense | |
| 14:53:32 | sean-k-mooney | well i dont know | |
| 14:53:55 | sean-k-mooney | if i was a custoemr and my vm sudenly stoped working i might do a start to see if that fixes it | |
| 14:54:12 | noonedeadpunk | I think it will appear as active | |
| 14:54:19 | sean-k-mooney | it will yes | |
| 14:54:27 | noonedeadpunk | so you are able only to reboot or shutdown? | |
| 14:54:37 | sean-k-mooney | but i might not check and just do a start but ya i normlally woud do hard-reboot | |
| 14:55:14 | sean-k-mooney | its less likely but if we are adressing this we proably should go throug all the statees and just make them consitent/intuitive | |
| 14:55:28 | noonedeadpunk | I mean that you can't start already active instance - you will get same Conflict exception iirc | |
| 14:55:44 | sean-k-mooney | ah | |
| 14:55:51 | sean-k-mooney | yes proably since its in the state you want | |
| 14:55:56 | noonedeadpunk | yeah | |
| 14:57:13 | noonedeadpunk | so powering-on would be really unfortunate co-incidence that will affect most likely only CI toolings or dunno... | |
| 14:58:05 | sean-k-mooney | ya its much less likely | |
| 14:58:20 | melwitt | gibi, stephenfin: heya, I've updated the --task-log archive patch to address gibi's comments https://review.opendev.org/c/openstack/nova/+/780395 | |
| 14:58:39 | sean-k-mooney | we can reason about it though and come to a logic conclution for what it shoudl do so we proably shoudl just cover it | |
| 15:15:10 | stephenfin | melwitt: trying to backport https://review.opendev.org/c/openstack/nova/+/602432 at the moment (it's hell) but I'll hit that again before EOD, hopefully | |
| 15:16:16 | melwitt | stephenfin: ok np, and good luck | |
| 15:24:15 | kashyap | stephenfin: sean-k-mooney: NUMA-related: you might find it interesting - libvirt upstream is wiring up "HMAT" - which defines the different latencies and bandwidths b/n NUMA nodes: | |
| 15:24:19 | kashyap | [quote] | |
| 15:24:21 | kashyap | "Links between NUMA nodes can have different latencies and bandwidths. This info is newly defined in ACPI 6.2 under Heterogeneous Memory Attribute Table (HMAT) table. Linux kernel learned how to report these values under sysfs and thus we can expose them in our capabilities XML. The sysfs interface is documented in kernel's Documentation/admin-guide/mm/numaperf.rst." | |
| 15:24:27 | kashyap | [/quote] | |
| 15:24:58 | kashyap | This is called "NUMA interconnects", apparently: https://listman.redhat.com/archives/libvir-list/2021-June/msg00268.html | |
| 15:28:12 | sean-k-mooney | ill take a look | |