| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-06-03 | |||
| 08:25:25 | bauzas | so, sometimes we directly discuss on IRC for a spec | |
| 08:25:31 | suzhengwei | cool | |
| 08:25:35 | bauzas | but like gibi said, we're both on the same TZ | |
| 08:25:45 | bauzas | so it's simple | |
| 08:26:09 | gibi | suzhengwei: I keep my client up during the night so I will see pings next day | |
| 08:26:14 | bauzas | me too | |
| 08:28:37 | XinxinShen | cool, We should ensure that the spec author and reviewer can communicate in time. | |
| 08:28:46 | gibi | suzhengwei: if you ping us, one thing to help is not just pinging but stating your problem right away | |
| 08:29:50 | suzhengwei | yes | |
| 08:30:30 | suzhengwei | I am doing some work about instance HA, and interested in the topic "Support vm evacuation while server status is suspended, paused". | |
| 08:31:00 | suzhengwei | I wonder if some one has processed it. | |
| 08:31:27 | bauzas | I still have open specs to look at | |
| 08:31:49 | suzhengwei | If not, I would like to do it. | |
| 08:31:57 | gibi | suzhengwei: interesting ideas | |
| 08:32:34 | gibi | suzhengwei: I think we don't support these today, but if with some compromise we could | |
| 08:33:18 | gibi | we will loose the in memory state of the paused instance but we could rebuild it still on another compute host | |
| 08:33:48 | gibi | and if the source compute is dead already then the in memory state is lost anyhow | |
| 08:34:13 | suzhengwei | host failure triggers evacuation. And active instance loose the in memory state too. | |
| 08:34:32 | gibi | suzhengwei: yes, so I think it is OK to lose that for the pause instance too | |
| 08:34:41 | gibi | suzhengwei, bauzas: does suspend saves some state to the disk? | |
| 08:34:49 | bauzas | good question | |
| 08:35:07 | bauzas | I honestly don't have the answer straight out of my mind | |
| 08:35:22 | gibi | anyhow if it save something to disk and that disk is on shared storage then we might even recover that saves state on the destination host | |
| 08:35:30 | bauzas | the crucial bit to remember with instance HA is that the host is already done | |
| 08:35:32 | gibi | but as a first step I would loose that too | |
| 08:35:33 | bauzas | gone* | |
| 08:35:47 | gibi | bauzas: yepp | |
| 08:35:56 | bauzas | so, yeah, ephemeral storage can't be somehow persisted | |
| 08:36:17 | gibi | except if it is on shared storage ^^ ;) | |
| 08:36:19 | bauzas | you need to have either shared storage or volumes | |
| 08:36:23 | gibi | yepp | |
| 08:36:38 | bauzas | yeah, but in general, you need to assume a crash | |
| 08:36:52 | bauzas | so any memory that's not synced is lost | |
| 08:36:58 | gibi | bauzas: yeah, this is why first I would assume that the suspended state is lost as well | |
| 08:37:16 | gibi | to avoid an inconsistent suspended state to be loaded | |
| 08:37:18 | bauzas | if suspend stores on disk, we're ok | |
| 08:37:32 | bauzas | kashyap: around ? | |
| 08:37:40 | kashyap | bauzas: Mornin, yes | |
| 08:37:53 | kashyap | How can I be useful? :) | |
| 08:37:57 | bauzas | kashyap: we are in office hour and we have a question about suspended instances | |
| 08:38:25 | bauzas | with the libvirt driver and qemu, what happens to the memory state when suspending ? | |
| 08:38:32 | bauzas | do we suspend on disk ? | |
| 08:38:41 | bauzas | I'd be inclined to say so | |
| 08:38:52 | kashyap | bauzas: gibi: Yes: suspend usually indeed means save-the-state-to-a-file-on-disk | |
| 08:39:02 | kashyap | Your inclination is correct :) | |
| 08:39:33 | bauzas | the problem is that we can't tell whether the instance is on shared storage or not | |
| 08:39:43 | gibi | kashyap: and what do you think, moves such suspended state between compute host make sense? | |
| 08:40:25 | bauzas | gibi: keep in mind evacuate is a rebuild | |
| 08:40:26 | kashyap | gibi: You mean moving such suspended state between different compute hosts make sense? | |
| 08:40:40 | gibi | kashyap: yeaht that is my queston | |
| 08:41:05 | gibi | bauzas: in case of suspend we see the vm_state on the dest being suspended so we can look for the state file on the disk. if it is there then we know that it was on shared storage | |
| 08:41:27 | bauzas | gibi: sure but then we leak the state of the host | |
| 08:41:40 | bauzas | this isn't predictable | |
| 08:42:28 | gibi | OK, I agree this can be a can of worm | |
| 08:42:54 | gibi | suzhengwei: in case of evacuating a suspended VM, is it OK to you to loose the suspended state? | |
| 08:44:03 | kashyap | gibi: I need to think a bit more about it. (libvirt has managedSave() API that does the suspend thingie, which already Nova uses. So we have the primitives...) | |
| 08:44:06 | suzhengwei | If host down, active suspend instance both loose their memory. | |
| 08:44:53 | kashyap | gibi: suzhengwei: What is the main use-case here? The ability to start suspended instances on any compute host from a given pool? | |
| 08:44:59 | gibi | suzhengwei: if you don't want to recover the suspended state that is saved to disk, then I think your proposal is pretty simple and straight forward | |
| 08:45:08 | suzhengwei | Instance Ha, try best to recover the workload as much as possible. | |
| 08:45:17 | bauzas | interestingly, I found some nova admin docs https://docs.openstack.org/nova/latest/admin/node-down.html | |
| 08:45:38 | gibi | kashyap: we looked at it from evacuation perspective. VM is suspended to disk (on shared storage), the host dies, user evacuates VM | |
| 08:46:04 | kashyap | gibi: I see; that makes sense | |
| 08:46:51 | bauzas | gibi: I honestly feel we can just support recreating a new instance | |
| 08:47:07 | kashyap | bauzas: Isn't that what already 'rebuild' is? | |
| 08:47:16 | kashyap | Ah, you said that already above :) | |
| 08:47:18 | bauzas | kashyap: yup, the question was about the memory state | |
| 08:47:51 | suzhengwei | If host down, the suspend instance can be active agian on the origin node. So I think it makes sence to evacuate suspend instances. | |
| 08:48:01 | suzhengwei | can not | |
| 08:48:06 | bauzas | I guess here suzhengwei's concern is that we limitate evacuate on active instances | |
| 08:48:14 | bauzas | right? | |
| 08:48:22 | bauzas | that's the problem we're trying to solve ? | |
| 08:48:35 | gibi | I think so | |
| 08:48:39 | suzhengwei | yes | |
| 08:49:09 | gibi | and I'm totally supportive to extend evac to support paused and suspended instances. It is simple if we allow loosing the running state | |
| 08:49:11 | bauzas | I just remembered we have a --on-shared-storage flag https://docs.openstack.org/nova/latest/admin/evacuate.html#evacuate-a-single-instance | |
| 08:49:36 | bauzas | since evacuate is an admin action, op can use it | |
| 08:49:39 | bauzas | on purpose | |
| 08:50:51 | bauzas | so we already do the check automatically | |
| 08:50:52 | gibi | bauzas: onSharedStorage is deprecated in 2.13 | |
| 08:51:00 | gibi | bauzas: today we automatically detect it I guess | |
| 08:51:01 | bauzas | gibi: because we detect this ? | |
| 08:51:04 | bauzas | yeah | |
| 08:51:09 | gibi | "Starting since version 2.14, Nova automatically detects whether the server is on shared storage or not. Therefore this parameter was removed." | |
| 08:51:12 | gibi | yepp | |
| 08:51:20 | bauzas | ok, so I guess we can consider adding suspend | |
| 08:51:31 | gibi | suzhengwei: I suggest to propose a small spec about this. I'm happy to review it | |
| 08:51:44 | bauzas | if the target host is on shared storage, we could just try to boot with the suspended state | |
| 08:52:02 | bauzas | for paused, the implication would be that the evacuated instance would become active | |
| 08:52:15 | gibi | bauzas: active, or stopped | |
| 08:52:15 | bauzas | for suspend, too | |
| 08:52:20 | gibi | bauzas: we can decide | |
| 08:52:27 | bauzas | yup, that's the point | |
| 08:52:31 | gibi | bauzas: but true, it cannot be pasued any more | |
| 08:52:36 | gibi | paused | |
| 08:53:05 | gibi | I don't want to stop the discussion, but we have 8 minutes left. If there any other topic to discuss? | |
| 08:53:14 | gibi | /If/Is/ | |
| 08:53:24 | kashyap | bauzas: gibi: One last: | |
| 08:53:27 | gibi | kashyap: go | |
| 08:53:30 | suzhengwei | I think stopped is better. No matter pause or suspend, users can not acess the instance directly. | |