| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-06-03 | |||
| 08:35:07 | bauzas | I honestly don't have the answer straight out of my mind | |
| 08:35:22 | gibi | anyhow if it save something to disk and that disk is on shared storage then we might even recover that saves state on the destination host | |
| 08:35:30 | bauzas | the crucial bit to remember with instance HA is that the host is already done | |
| 08:35:32 | gibi | but as a first step I would loose that too | |
| 08:35:33 | bauzas | gone* | |
| 08:35:47 | gibi | bauzas: yepp | |
| 08:35:56 | bauzas | so, yeah, ephemeral storage can't be somehow persisted | |
| 08:36:17 | gibi | except if it is on shared storage ^^ ;) | |
| 08:36:19 | bauzas | you need to have either shared storage or volumes | |
| 08:36:23 | gibi | yepp | |
| 08:36:38 | bauzas | yeah, but in general, you need to assume a crash | |
| 08:36:52 | bauzas | so any memory that's not synced is lost | |
| 08:36:58 | gibi | bauzas: yeah, this is why first I would assume that the suspended state is lost as well | |
| 08:37:16 | gibi | to avoid an inconsistent suspended state to be loaded | |
| 08:37:18 | bauzas | if suspend stores on disk, we're ok | |
| 08:37:32 | bauzas | kashyap: around ? | |
| 08:37:40 | kashyap | bauzas: Mornin, yes | |
| 08:37:53 | kashyap | How can I be useful? :) | |
| 08:37:57 | bauzas | kashyap: we are in office hour and we have a question about suspended instances | |
| 08:38:25 | bauzas | with the libvirt driver and qemu, what happens to the memory state when suspending ? | |
| 08:38:32 | bauzas | do we suspend on disk ? | |
| 08:38:41 | bauzas | I'd be inclined to say so | |
| 08:38:52 | kashyap | bauzas: gibi: Yes: suspend usually indeed means save-the-state-to-a-file-on-disk | |
| 08:39:02 | kashyap | Your inclination is correct :) | |
| 08:39:33 | bauzas | the problem is that we can't tell whether the instance is on shared storage or not | |
| 08:39:43 | gibi | kashyap: and what do you think, moves such suspended state between compute host make sense? | |
| 08:40:25 | bauzas | gibi: keep in mind evacuate is a rebuild | |
| 08:40:26 | kashyap | gibi: You mean moving such suspended state between different compute hosts make sense? | |
| 08:40:40 | gibi | kashyap: yeaht that is my queston | |
| 08:41:05 | gibi | bauzas: in case of suspend we see the vm_state on the dest being suspended so we can look for the state file on the disk. if it is there then we know that it was on shared storage | |
| 08:41:27 | bauzas | gibi: sure but then we leak the state of the host | |
| 08:41:40 | bauzas | this isn't predictable | |
| 08:42:28 | gibi | OK, I agree this can be a can of worm | |
| 08:42:54 | gibi | suzhengwei: in case of evacuating a suspended VM, is it OK to you to loose the suspended state? | |
| 08:44:03 | kashyap | gibi: I need to think a bit more about it. (libvirt has managedSave() API that does the suspend thingie, which already Nova uses. So we have the primitives...) | |
| 08:44:06 | suzhengwei | If host down, active suspend instance both loose their memory. | |
| 08:44:53 | kashyap | gibi: suzhengwei: What is the main use-case here? The ability to start suspended instances on any compute host from a given pool? | |
| 08:44:59 | gibi | suzhengwei: if you don't want to recover the suspended state that is saved to disk, then I think your proposal is pretty simple and straight forward | |
| 08:45:08 | suzhengwei | Instance Ha, try best to recover the workload as much as possible. | |
| 08:45:17 | bauzas | interestingly, I found some nova admin docs https://docs.openstack.org/nova/latest/admin/node-down.html | |
| 08:45:38 | gibi | kashyap: we looked at it from evacuation perspective. VM is suspended to disk (on shared storage), the host dies, user evacuates VM | |
| 08:46:04 | kashyap | gibi: I see; that makes sense | |
| 08:46:51 | bauzas | gibi: I honestly feel we can just support recreating a new instance | |
| 08:47:07 | kashyap | bauzas: Isn't that what already 'rebuild' is? | |
| 08:47:16 | kashyap | Ah, you said that already above :) | |
| 08:47:18 | bauzas | kashyap: yup, the question was about the memory state | |
| 08:47:51 | suzhengwei | If host down, the suspend instance can be active agian on the origin node. So I think it makes sence to evacuate suspend instances. | |
| 08:48:01 | suzhengwei | can not | |
| 08:48:06 | bauzas | I guess here suzhengwei's concern is that we limitate evacuate on active instances | |
| 08:48:14 | bauzas | right? | |
| 08:48:22 | bauzas | that's the problem we're trying to solve ? | |
| 08:48:35 | gibi | I think so | |
| 08:48:39 | suzhengwei | yes | |
| 08:49:09 | gibi | and I'm totally supportive to extend evac to support paused and suspended instances. It is simple if we allow loosing the running state | |
| 08:49:11 | bauzas | I just remembered we have a --on-shared-storage flag https://docs.openstack.org/nova/latest/admin/evacuate.html#evacuate-a-single-instance | |
| 08:49:36 | bauzas | since evacuate is an admin action, op can use it | |
| 08:49:39 | bauzas | on purpose | |
| 08:50:51 | bauzas | so we already do the check automatically | |
| 08:50:52 | gibi | bauzas: onSharedStorage is deprecated in 2.13 | |
| 08:51:00 | gibi | bauzas: today we automatically detect it I guess | |
| 08:51:01 | bauzas | gibi: because we detect this ? | |
| 08:51:04 | bauzas | yeah | |
| 08:51:09 | gibi | "Starting since version 2.14, Nova automatically detects whether the server is on shared storage or not. Therefore this parameter was removed." | |
| 08:51:12 | gibi | yepp | |
| 08:51:20 | bauzas | ok, so I guess we can consider adding suspend | |
| 08:51:31 | gibi | suzhengwei: I suggest to propose a small spec about this. I'm happy to review it | |
| 08:51:44 | bauzas | if the target host is on shared storage, we could just try to boot with the suspended state | |
| 08:52:02 | bauzas | for paused, the implication would be that the evacuated instance would become active | |
| 08:52:15 | gibi | bauzas: active, or stopped | |
| 08:52:15 | bauzas | for suspend, too | |
| 08:52:20 | gibi | bauzas: we can decide | |
| 08:52:27 | bauzas | yup, that's the point | |
| 08:52:31 | gibi | bauzas: but true, it cannot be pasued any more | |
| 08:52:36 | gibi | paused | |
| 08:53:05 | gibi | I don't want to stop the discussion, but we have 8 minutes left. If there any other topic to discuss? | |
| 08:53:14 | gibi | /If/Is/ | |
| 08:53:24 | kashyap | bauzas: gibi: One last: | |
| 08:53:27 | gibi | kashyap: go | |
| 08:53:30 | suzhengwei | I think stopped is better. No matter pause or suspend, users can not acess the instance directly. | |
| 08:53:43 | gibi | suzhengwei: I can accept that | |
| 08:54:07 | kashyap | gibi: suzhengwei: On whether it makes sense of moving suspended instances between compute hosts, a thumb-rule can be: "follow the same rules for hardware matching as for a live migration between the hosts" | |
| 08:54:19 | kashyap | (I mean, to uncover any "gotchas") | |
| 08:54:47 | gibi | kashyap: ahh you have a point, this state can be hw dependent | |
| 08:54:49 | bauzas | kashyap: since evacuate is a rebuild, we can't predict this | |
| 08:54:49 | kashyap | FWIW, I also just checked the above w/ a QEMU migration developer; and he agrees. | |
| 08:55:25 | bauzas | gibi: sorry, I wasn't explicit but when I said 'we're gonna try to unsuspend from disk", I was thinking of hardware capabilities | |
| 08:55:28 | gibi | OK, then I propose not to try to recover the suspended state during evac. At list not in the first step | |
| 08:55:40 | kashyap | bauzas: gibi: Hm, so looks like this needs to be fleshed out in a design document | |
| 08:55:41 | gibi | /list/least/ | |
| 08:56:06 | bauzas | kashyap: the evacuate workflow is waaaaay different from live-migrate | |
| 08:56:16 | bauzas | you can't just check the source host at first ;) | |
| 08:56:27 | bauzas | and compare both | |
| 08:56:45 | bauzas | the scheduler is just giving you a target and then good luck with it | |
| 08:56:57 | kashyap | bauzas: I see; fair enough | |
| 08:57:03 | gibi | so in summary | |
| 08:57:04 | bauzas | but yeah, we're 4 mins | |
| 08:57:07 | gibi | so in summary | |
| 08:57:07 | bauzas | left | |
| 08:57:49 | gibi | suzhengwei: please propose a spec. I don't see any problem supporting evac for paused and suspended VMs. But they will lose the in memory or suspended state. They will be fresh VMs on the dest host in stopped state | |
| 08:58:18 | kashyap | Yeah; makes sense. | |