| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-09-23 | |||
| 09:27:01 | bauzas | sure, but that's why you need to stop the instance, right? | |
| 09:27:54 | bauzas | when you shelve, IIUC, you'll get a memory snapshot | |
| 09:28:36 | bauzas | sorry, a disk snapshot I mean | |
| 09:28:45 | bauzas | but the memory will be lost | |
| 09:30:08 | parasitid | bauzas: ok. couldn't the shelve process consider the VM state and also backup/restore the memory state file if ever the instance was in suspended state ? | |
| 09:32:02 | bauzas | I see your usecase tbc | |
| 09:32:42 | bauzas | but the problem here is that for the moment, AFAIK nova isn't able to save the memory | |
| 09:32:49 | parasitid | bauzas: usecase would be VDI/remote desktop | |
| 09:32:54 | bauzas | yup, I got it | |
| 09:33:18 | bauzas | that's in general *why* people want to save the memory :D | |
| 09:34:49 | parasitid | bauzas: are you sure ? i made a test this morning. when i suspend an instance and resume it, i can ssh back into my instance and tmux attach to my session and recover the vim buffer right where is was before suspending | |
| 09:34:49 | bauzas | honestly, I don't know what to tell you, except that it would need a new feature | |
| 09:35:20 | bauzas | suspend works indeed with the memory, but we don't snapshot it | |
| 09:35:31 | bauzas | that's why I said above "snapshot it" | |
| 09:35:39 | parasitid | bauzas: ok thanks. i understand perfectly. i jumped here just in case there would be a secret tip to achieve this :) | |
| 09:35:47 | bauzas | but your usecase is different : you want to /persist/ it | |
| 09:36:07 | bauzas | tbh, VMware supports that AFAIKK | |
| 09:36:18 | bauzas | (memory snapshot) | |
| 09:36:34 | bauzas | but as I said here, we need some new feature and some new API support for it | |
| 09:36:36 | bauzas | sean-k-mooney: ^ | |
| 09:38:30 | bauzas | (that and live-resize are possibly the most needed features we'd like to have for VMware migration :) ) | |
| 09:39:40 | bauzas | sean-k-mooney: I wonder, could we help parasitid with file-backed memory? | |
| 09:41:08 | bauzas | (oh I forgot, sean is on PTO this week) | |
| 09:43:31 | sean-k-mooney[m] | we have talked about this in the past, libvirt can do memory snapshots but we can because we do not guarantee a stable hardware interface | |
| 09:44:36 | sean-k-mooney[m] | when you unshleve you might shelve to a differnt host with differnent cpu, the pci device order in the guest may or may not be the same and if you have any sriov device there memory would not be captured | |
| 09:46:32 | sean-k-mooney[m] | we may be able to make it work in some cases but we would have to save and store the memory as an addtional image and also might need to recored other info about the guest like the pci device ids instead of allowing libvirt to choose them | |
| 09:47:08 | sean-k-mooney[m] | but no file backed memory will not help here. libivrt has a call to take a snapshot and include the guest memory | |
| 09:47:17 | sean-k-mooney[m] | so its not that we cant do that | |
| 09:48:55 | gibi | yeah shelve with memory would be like: hibernate your guest, write the image to disk, bring the image to another hardwared, and try to resume from hibernation. As far as I know if I change my hardware while my machine is hibernated and I try to resume I very well be in kernel panic territory | |
| 09:49:09 | sean-k-mooney[m] | for instances with no passthough devices like neurton sriov ports, or vgpus that use cpu_mode=custom and a pinned cpu model it proably would work | |
| 09:50:16 | gibi | wondering what restiction libvirt puts on resuming from memory snapshot while the hypervisor is reconfigured inbetween | |
| 09:50:37 | sean-k-mooney[m] | yep, so if there there was a strong demand for this, we could consider it but its very non cloudy, but it is an enterpirse virt feature | |
| 09:51:57 | gibi | If I have to choose what to spend time on between live-resize vs shelve with memory, then the former feels a more generally useful feature | |
| 09:52:28 | gibi | (and both is hard) | |
| 09:53:07 | sean-k-mooney[m] | the former is more cloudy | |
| 09:53:37 | sean-k-mooney[m] | shelve with memory is definetly in the enterpise virt space | |
| 09:56:24 | bauzas_ | I agree with the fact there are some technical constraints and limitations | |
| 09:56:54 | sean-k-mooney[m] | anyway yes im on pto this week so totally not going back to working on my home lab. | |
| 09:56:57 | gibi | it is more like we don't seem to know the list of constraints, and coming up with one is not simple | |
| 09:57:05 | bauzas_ | but the usecase written as "as a VDI user, I'd like my instance to be shelved with its memory so I can spin again my instance without problems" sounds a valid cloud usecase for me | |
| 09:57:18 | gibi | sean-k-mooney[m]: enjoy your PTO | |
| 09:57:22 | sean-k-mooney[m] | VDI its selef is iffy | |
| 09:58:04 | gibi | bauzas_: in a cloud we run cattle. | |
| 09:58:14 | sean-k-mooney[m] | im open to looking at this again but to be clear we have said no to this exact usecase once before and snapshot with memory once also | |
| 09:58:33 | sean-k-mooney[m] | shevle with memory i think is actully more reasonable then generic snapshot with memory | |
| 09:58:34 | bauzas_ | if you prefer, someone running a instance with virtual desktop capabilities | |
| 09:59:13 | bauzas | or there could be other workloads where saving the memory would be more than just nice | |
| 09:59:39 | bauzas | but there are security implications for sure, we'd need to keep the memory safe | |
| 09:59:49 | gibi | bauzas: I still reject the idea that this is cloudy. In a cloud you should be ready to loose a single VM any time. | |
| 09:59:52 | sean-k-mooney[m] | yep you just need to reconsile the fact that we provide no abi stablity for the guest. | |
| 10:00:10 | sean-k-mooney[m] | meaning if you add 2 volumes and remove 1 | |
| 10:00:23 | bauzas | gibi: well, that ship has already sailed for a while :( | |
| 10:00:36 | gibi | sean-k-mooney[m]: yeah we can promise that we restore your memory but we cannot promise the guest kernel wont panic | |
| 10:00:46 | sean-k-mooney[m] | the first one, then the next time we generte the xml the remaining voluems pci adress will change | |
| 10:01:04 | bauzas | while I'm one of the most feroceous guys that say 'sorry, cloud', I tend to admit that /some/ workloads require more care | |
| 10:01:07 | gibi | bauzas: we can try to stop on that slippery slope :) | |
| 10:02:08 | sean-k-mooney[m] | we can add this to the list of things people want but i would port vtpm live migration higher personally | |
| 10:02:13 | bauzas | gibi: and ask the foundation to remove the Vmware migration to OpenStack whitebook ? :) | |
| 10:02:27 | gibi | unshelve with memory is like a brain transplant while you expect not just the patient to survive but also to keep all its past memories. | |
| 10:02:28 | sean-k-mooney[m] | this is notthign to do with vmware | |
| 10:02:43 | sean-k-mooney[m] | we have a vmware direver and they never cared to enable this usecase | |
| 10:03:06 | sean-k-mooney[m] | we have disucss this in an inperson ptg before | |
| 10:03:11 | sean-k-mooney[m] | so this is an old request | |
| 10:03:17 | gibi | bauzas: just because the fundation say so brain transplant will not work out of the boz | |
| 10:03:18 | bauzas | doesn't vsphere support snapshoting the memory ? | |
| 10:03:20 | gibi | x | |
| 10:03:35 | bauzas | I like the idea of "brain transplant" | |
| 10:03:38 | sean-k-mooney[m] | it does but it does not do it as part of shelve | |
| 10:03:53 | bauzas | but this is more live neurolinks transplants | |
| 10:04:18 | bauzas | I can consider the brain as disk while the neurotransmitters are the RAM :) | |
| 10:04:29 | sean-k-mooney[m] | libvirt, hyperv and vmware all can do this (so can virtual box) but none of them ever enabled it in there virt driver | |
| 10:05:14 | gibi | the disk content goes through a full kernel boot process with hw probing etc, the resume from ram tries to continue exectuing where it left of without a boot process | |
| 10:06:35 | bauzas | then why shelve w/ memory would be more acceptable than snapshot w/ memory ? | |
| 10:06:45 | bauzas | shit | |
| 10:06:51 | gibi | I think we can ask the libvirt folks how do you feel about moving the libvirt memory snapshot, disk, and domain def to another compute and try to resume, if they say they support it then lets discuss further | |
| 10:07:24 | bauzas | sorry, I said "shit" because I was trying to understand exactly the diffs with suspend | |
| 10:08:19 | bauzas | gibi: again, I don't have the energy nor the will to push that more forward, I was just saying that the usecase may sound legit | |
| 10:08:51 | bauzas | let's not rathole on it, the answer is as of now "NOT SUPPORTED" | |
| 10:09:23 | gibi | as legit as requesting a brain transpart. legit but pointless if the tech is not there | |
| 10:10:18 | bauzas | the tech already allows us teleportation :) | |
| 10:10:41 | bauzas | but I hear ya | |
| 10:10:59 | bauzas | that would be an horrible spec to write and a terrible spec to review | |
| 10:11:28 | gibi | if teleportation you mean live-migration then yes libvirt has a bunch of tech implemented to support that, I'm not sure they have the tech to support brain transplant yet. | |
| 10:14:04 | gibi | btw that points to a direction actually. instead of shelving with a memory snapshot, do memory snapshot on the current hypervisor. The resume is supported there on the same hypevisor. I think the requesting people do not need the move aspect of shelve, they need the resumability only. So give them way to resume on same hypevisor | |
| 10:15:44 | gibi | wait we have that, they just get billed as they reseve the space to be able to resume | |
| 10:16:10 | gibi | so they don't want to get billed, but still want to make sure they can resume in place, that is contradicting | |
| 10:16:39 | gibi | or at least we are moving to interrutible instances territory to make space for resume | |
| 10:17:11 | gibi | (or what was the name of that feature that allows killing certian type of instances to make space for reservations) | |
| 10:18:43 | gibi | I guess I stop braindumping here as it is not exciting :) | |
| 11:06:56 | sean-k-mooney[m] | the reason shelve is more ligit then snapshot is it removes the snapshot once and create many copies uescase adn it also removes restore via rebuild | |
| 11:08:02 | sean-k-mooney[m] | so shelve/unshelve is much smaller in scope and has less sharp edges as a result | |
| 11:10:35 | sean-k-mooney[m] | gibi the reason sususped (which calls managed save) works today is we still have the domain so we can still restore it without worriing about the xml changing, managed save also does not allow pci passthough devices to be attached to the domain | |
| 11:12:11 | sean-k-mooney[m] | if i was to do shelve with memory my instict would say, save the xml and the memory as addtional images and reuse some of the live migraton logic to update the host side for it xml without modifying the guest visible side | |
| 11:12:51 | sean-k-mooney[m] | the same host does not really matter as long as the guest cant tell it moved | |
| 11:32:35 | gibi | sean-k-mooney[m]: I agree, move does not matter to the user if the instance works. I just pointing out that they also never requested the move, they only requests having a place to resume to but not getting billed for that place while suspended. | |
| 11:33:42 | gibi | btw having a vtpm live migration indeed seems even more important to get | |
| 11:35:06 | sean-k-mooney[m] | i mainly mentioned vtpm as another example of needing to store addtinoal data alongside the disk when shelving which we dont support today | |
| 11:35:44 | sean-k-mooney[m] | but also it a higher priority in my book but both are parralel efforts | |
| 11:37:14 | parasitid | hi gibi: i think i get you're point but there are still lots very obscure stuff to me because i'm definately not a specialist of this topic. When you say: "so they don't want to get billed, but still want to make sure they can resume in place, that is contradicting" why wouldn't it be possible to be resumed on a host supporting the same "flavor" ? why does it work in case of live migration and not unshelving ?. And yes i wouldn't | |
| 11:38:05 | parasitid | gibi: i just want to maximize the fact of being "resumable" but lowering the No Valid Host found error | |