Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-06-03
08:37:32 bauzas kashyap: around ?
08:37:40 kashyap bauzas: Mornin, yes
08:37:53 kashyap How can I be useful? :)
08:37:57 bauzas kashyap: we are in office hour and we have a question about suspended instances
08:38:25 bauzas with the libvirt driver and qemu, what happens to the memory state when suspending ?
08:38:32 bauzas do we suspend on disk ?
08:38:41 bauzas I'd be inclined to say so
08:38:52 kashyap bauzas: gibi: Yes: suspend usually indeed means save-the-state-to-a-file-on-disk
08:39:02 kashyap Your inclination is correct :)
08:39:33 bauzas the problem is that we can't tell whether the instance is on shared storage or not
08:39:43 gibi kashyap: and what do you think, moves such suspended state between compute host make sense?
08:40:25 bauzas gibi: keep in mind evacuate is a rebuild
08:40:26 kashyap gibi: You mean moving such suspended state between different compute hosts make sense?
08:40:40 gibi kashyap: yeaht that is my queston
08:41:05 gibi bauzas: in case of suspend we see the vm_state on the dest being suspended so we can look for the state file on the disk. if it is there then we know that it was on shared storage
08:41:27 bauzas gibi: sure but then we leak the state of the host
08:41:40 bauzas this isn't predictable
08:42:28 gibi OK, I agree this can be a can of worm
08:42:54 gibi suzhengwei: in case of evacuating a suspended VM, is it OK to you to loose the suspended state?
08:44:03 kashyap gibi: I need to think a bit more about it. (libvirt has managedSave() API that does the suspend thingie, which already Nova uses. So we have the primitives...)
08:44:06 suzhengwei If host down, active suspend instance both loose their memory.
08:44:53 kashyap gibi: suzhengwei: What is the main use-case here? The ability to start suspended instances on any compute host from a given pool?
08:44:59 gibi suzhengwei: if you don't want to recover the suspended state that is saved to disk, then I think your proposal is pretty simple and straight forward
08:45:08 suzhengwei Instance Ha, try best to recover the workload as much as possible.
08:45:17 bauzas interestingly, I found some nova admin docs https://docs.openstack.org/nova/latest/admin/node-down.html
08:45:38 gibi kashyap: we looked at it from evacuation perspective. VM is suspended to disk (on shared storage), the host dies, user evacuates VM
08:46:04 kashyap gibi: I see; that makes sense
08:46:51 bauzas gibi: I honestly feel we can just support recreating a new instance
08:47:07 kashyap bauzas: Isn't that what already 'rebuild' is?
08:47:16 kashyap Ah, you said that already above :)
08:47:18 bauzas kashyap: yup, the question was about the memory state
08:47:51 suzhengwei If host down, the suspend instance can be active agian on the origin node. So I think it makes sence to evacuate suspend instances.
08:48:01 suzhengwei can not
08:48:06 bauzas I guess here suzhengwei's concern is that we limitate evacuate on active instances
08:48:14 bauzas right?
08:48:22 bauzas that's the problem we're trying to solve ?
08:48:35 gibi I think so
08:48:39 suzhengwei yes
08:49:09 gibi and I'm totally supportive to extend evac to support paused and suspended instances. It is simple if we allow loosing the running state
08:49:11 bauzas I just remembered we have a --on-shared-storage flag https://docs.openstack.org/nova/latest/admin/evacuate.html#evacuate-a-single-instance
08:49:36 bauzas since evacuate is an admin action, op can use it
08:49:39 bauzas on purpose
08:50:51 bauzas so we already do the check automatically
08:50:52 gibi bauzas: onSharedStorage is deprecated in 2.13
08:51:00 gibi bauzas: today we automatically detect it I guess
08:51:01 bauzas gibi: because we detect this ?
08:51:04 bauzas yeah
08:51:09 gibi "Starting since version 2.14, Nova automatically detects whether the server is on shared storage or not. Therefore this parameter was removed."
08:51:12 gibi yepp
08:51:20 bauzas ok, so I guess we can consider adding suspend
08:51:31 gibi suzhengwei: I suggest to propose a small spec about this. I'm happy to review it
08:51:44 bauzas if the target host is on shared storage, we could just try to boot with the suspended state
08:52:02 bauzas for paused, the implication would be that the evacuated instance would become active
08:52:15 bauzas for suspend, too
08:52:15 gibi bauzas: active, or stopped
08:52:20 gibi bauzas: we can decide
08:52:27 bauzas yup, that's the point
08:52:31 gibi bauzas: but true, it cannot be pasued any more
08:52:36 gibi paused
08:53:05 gibi I don't want to stop the discussion, but we have 8 minutes left. If there any other topic to discuss?
08:53:14 gibi /If/Is/
08:53:24 kashyap bauzas: gibi: One last:
08:53:27 gibi kashyap: go
08:53:30 suzhengwei I think stopped is better. No matter pause or suspend, users can not acess the instance directly.
08:53:43 gibi suzhengwei: I can accept that
08:54:07 kashyap gibi: suzhengwei: On whether it makes sense of moving suspended instances between compute hosts, a thumb-rule can be: "follow the same rules for hardware matching as for a live migration between the hosts"
08:54:19 kashyap (I mean, to uncover any "gotchas")
08:54:47 gibi kashyap: ahh you have a point, this state can be hw dependent
08:54:49 kashyap FWIW, I also just checked the above w/ a QEMU migration developer; and he agrees.
08:54:49 bauzas kashyap: since evacuate is a rebuild, we can't predict this
08:55:25 bauzas gibi: sorry, I wasn't explicit but when I said 'we're gonna try to unsuspend from disk", I was thinking of hardware capabilities
08:55:28 gibi OK, then I propose not to try to recover the suspended state during evac. At list not in the first step
08:55:40 kashyap bauzas: gibi: Hm, so looks like this needs to be fleshed out in a design document
08:55:41 gibi /list/least/
08:56:06 bauzas kashyap: the evacuate workflow is waaaaay different from live-migrate
08:56:16 bauzas you can't just check the source host at first ;)
08:56:27 bauzas and compare both
08:56:45 bauzas the scheduler is just giving you a target and then good luck with it
08:56:57 kashyap bauzas: I see; fair enough
08:57:03 gibi so in summary
08:57:04 bauzas but yeah, we're 4 mins
08:57:07 bauzas left
08:57:07 gibi so in summary
08:57:49 gibi suzhengwei: please propose a spec. I don't see any problem supporting evac for paused and suspended VMs. But they will lose the in memory or suspended state. They will be fresh VMs on the dest host in stopped state
08:58:18 kashyap Yeah; makes sense.
08:58:24 suzhengwei I will.
08:58:26 kashyap (On spec)
08:58:32 gibi suzhengwei: cool, thanks
08:58:48 gibi any last words before we stop the meeting? ;)
08:59:03 suzhengwei nothing from me.
08:59:07 gibi XinxinShen: ?
08:59:37 XinxinShen nothing for me. thanks.
08:59:56 gibi then thanks for joining. please continue discussion if needed
09:00:00 gibi I just stop the meeting log here
09:00:04 gibi #endmeeting
09:00:06 opendevmeet Meeting ended Thu Jun 3 09:00:04 2021 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
09:00:07 opendevmeet Minutes: http://eavesdrop.openstack.org/meetings/nova_extra/2021/nova_extra.2021-06-03-08.00.html
09:00:08 opendevmeet Minutes (text): http://eavesdrop.openstack.org/meetings/nova_extra/2021/nova_extra.2021-06-03-08.00.txt
09:00:09 opendevmeet Log: http://eavesdrop.openstack.org/meetings/nova_extra/2021/nova_extra.2021-06-03-08.00.log.html
09:00:23 bauzas right on time

Earlier   Later