| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-11-11 | |||
| 20:00:31 | NostawRm | efried: I really appreciate that. I also don't mind putting in the work to contribute, I just like to check here to make sure the contributions wouldn't be immediately shot down if I can't find it being discussed anywhere else. Thanks! | |
| 20:00:37 | dansmith | no way around that, but ideally it would refuse to move it so they don't accidentally break instances | |
| 20:00:57 | efried | I think the secret stays loaded up in libvirtd as long as that process is running, so admin could do pretty much anything that keeps the instance on the same host; as long as they don't restart libvirtd the instance will come back up. | |
| 20:01:23 | efried | I guess I'll have to test the secret-goes-with-live-migrate theory. | |
| 20:01:49 | sean-k-mooney | efried: ya maybe that would be correct | |
| 20:01:59 | dansmith | efried: hopefully libvirt won't show you te secret right? | |
| 20:01:59 | sean-k-mooney | however hard reboot deletes the domain | |
| 20:02:06 | sean-k-mooney | and recreates it | |
| 20:02:13 | efried | dansmith: it won't, as long as private='yes'. | |
| 20:02:19 | sean-k-mooney | so that might clear the secret | |
| 20:02:29 | efried | sean-k-mooney: I'm saying specifically that doesn't matter. | |
| 20:02:35 | dansmith | efried: yeah, so you can't use that to do an unauthorized cold migration, if that's what you're implying | |
| 20:02:55 | efried | dansmith: understood for cold, but still wondering about live. | |
| 20:03:15 | dansmith | efried: live should work I would think | |
| 20:03:28 | dansmith | efried: if not, graceful fail, otherwise you'd migrate into an unrunnable state | |
| 20:05:02 | efried | setting up the secret and creating the instance are totally different things, from what I understand. I wouldn't expect libvirt to transfer the actual secret to the destination. However, if it's transferring the "state" of the instance, which includes an already-unencrypted vtpm, that might work. But then the VM on the target is kind of in a different condition than it was on the source, in that the secret is not loaded. | |
| 20:05:32 | dansmith | efried: oh, well in that case maybe worth more thinking, | |
| 20:05:33 | efried | so like the reboot-recreate-xml scenario would *not* work in that case. | |
| 20:06:02 | sean-k-mooney | i can ask danpb tomorow when he is online if he knows what is expected to happen | |
| 20:06:03 | dansmith | efried: but since the instances can't be restarted without the user's creds anyway, even if it migrates and can't reboot | |
| 20:06:05 | dansmith | that would be the same | |
| 20:06:12 | sean-k-mooney | but testing might give you an answer quiciker | |
| 20:07:26 | sean-k-mooney | what is the state of migrations for instnace with encyption today | |
| 20:07:36 | efried | dansmith: well, what I was saying is, once the secret is loaded up in libvirtd, anybody would be able to restart, without user's creds. | |
| 20:07:44 | sean-k-mooney | the vtpm would likely have the same constraints | |
| 20:08:06 | dansmith | efried: ...and break the ability to restart the instance in place you mean right? | |
| 20:08:08 | efried | unless we do a step to undefine the secret when rebooting or something. | |
| 20:08:51 | sean-k-mooney | efried: well we proably shoudl do that as part of stop | |
| 20:09:07 | sean-k-mooney | it also depens on if its a soft or hard reboot | |
| 20:09:14 | sean-k-mooney | soft reboot shoudl be fine | |
| 20:09:20 | dansmith | a soft reboot would persist I'd think, | |
| 20:09:33 | dansmith | but yeah, hard reboot and stop should nuke the secret if you want it safe at rest | |
| 20:10:11 | efried | The dom xml contains just a pointer to the secret. The secret has to be loaded up in libvirtd's memory in order to start the guest. Once it's loaded up, it stays loaded up until/unless you undefine it (or restart the service). Clearly I would undefine the secret on instance deletion. But anything where the instance needs to start, no matter who starts it, should work as long as the secret is still loaded. | |
| 20:10:48 | sean-k-mooney | efried: well you should undeife the secrete when you stop the instance without deleting it too | |
| 20:11:00 | sean-k-mooney | to not keep it in memory | |
| 20:11:05 | efried | why? | |
| 20:11:12 | efried | I mean sure... but why? | |
| 20:11:40 | sean-k-mooney | to prevent people reading it form libvirt memory. we could make it a config option or chose not too | |
| 20:12:05 | sean-k-mooney | but if you really care about security you want the key decyped for as short a time as possible | |
| 20:12:08 | efried | that sounds in the same category as "delete the file from disk right quick" | |
| 20:12:17 | efried | i.e. "fake security" as dansmith calls it. | |
| 20:12:29 | sean-k-mooney | not really | |
| 20:13:00 | sean-k-mooney | you are say if i have not deleted the isntance but stop it keep the key in memory untill either the vm is deleted or you restart libvirt | |
| 20:13:02 | dansmith | efried: I think the user would expect that it's safe at rest when stopped | |
| 20:17:58 | dansmith | efried: this is definitely all a bit of theater of course, because none of the places where the key lives unencrypted for any amount of time are "secure", | |
| 20:18:21 | efried | dansmith: We have apparently accepted that "compromised root" menas all bets are off anyway. | |
| 20:18:43 | dansmith | but I think it's different to say "okay you stopped it, so I forgot all the secrets until you give them to me again" vs "I wasn't very careful with your secret while I had it" | |
| 20:18:43 | sean-k-mooney | i dont think that is really true | |
| 20:19:03 | efried | the important thing being, if someone walks off with the disk, they can't get at the secret, because they don't have the user's creds. | |
| 20:19:07 | sean-k-mooney | in some cases maybe but we should not make it easy even in that case | |
| 20:19:24 | dansmith | efried: compromised root of the host you mean? that's definitely game-over I think with all the other assumptions that prop up an openstack cluster | |
| 20:19:41 | dansmith | efried: yep and that's what I call 'safe at rest | |
| 20:19:48 | efried | conceivably I could delete the secret as soon as the guest starts. | |
| 20:20:04 | sean-k-mooney | efried: i have been wondering about that | |
| 20:20:14 | efried | I assume the tpm stays unlocked while the guest is up. Testing... | |
| 20:20:16 | sean-k-mooney | but that could break one usecase | |
| 20:20:20 | sean-k-mooney | maybe | |
| 20:20:30 | dansmith | efried: I don't think that buys you anything real, other than avoiding forgetting/failing to clean it up, which is good of course | |
| 20:20:31 | sean-k-mooney | what hapens when you ssh in and do a reboot in the guest | |
| 20:20:52 | efried | dansmith: yeah, that's all I meant, just simplifies some of the assumptions and operations. | |
| 20:21:04 | dansmith | sean-k-mooney: IIRC modern kvm reboot is within the qemu process, not respawned by libvirt, so I would expect that works | |
| 20:21:04 | efried | Makes it harder for not-the-user to do things to the instance, but at least makes it harder consistently. | |
| 20:21:34 | dansmith | efried: well, I'd see the main gain as avoiding a case where we fail during cleanup and don't get to the secret-forgetting part | |
| 20:21:35 | dansmith | but yeah | |
| 20:21:47 | sean-k-mooney | dansmith: ya i thinkit sends an ahci shutdown and does nto exit the qemu process too so i think it would be ok i was just not sure | |
| 20:22:07 | dansmith | sean-k-mooney: acpi you mean? | |
| 20:22:14 | sean-k-mooney | sorry yes | |
| 20:22:21 | sean-k-mooney | not storage :) | |
| 20:22:28 | dansmith | otherwise you're talking about usb or firewire | |
| 20:22:35 | dansmith | usb I think | |
| 20:22:43 | sean-k-mooney | ahci is used for sata too | |
| 20:22:48 | efried | ...yeah, confirmed deleting the secret after the guest starts doesn't affect ability of guest to get at the vtpm. | |
| 20:22:53 | dansmith | oh sata right right | |
| 20:23:01 | dansmith | mixed up my *hci acronyms | |
| 20:23:24 | dansmith | efried: almost impossible for that to happen, since qemu and libvirt are fairly separate at runtime | |
| 20:23:33 | dansmith | efried: try a soft reboot from inside the guest tho | |
| 20:23:50 | efried | meaning, from within the guest, just 'reboot'? | |
| 20:23:55 | sean-k-mooney | yes | |
| 20:24:37 | sean-k-mooney | a soft reboot from the api would almost be the same excpet that will escalte to a hard reboot after a time out | |
| 20:25:35 | efried | that seems to have worked fine. Guest came back up, no errors in n-cpu, the tpm is still there and still works. | |
| 20:25:40 | dansmith | figured | |
| 20:25:50 | dansmith | so delete after start is probably the way to go | |
| 20:25:50 | efried | that's what we want though, right? | |
| 20:25:54 | sean-k-mooney | yes | |
| 20:25:57 | dansmith | makes it very clear how long we have the key | |
| 20:26:06 | dansmith | yep | |
| 20:26:08 | efried | like. | |
| 20:27:01 | efried | so assume live migration would carry the VM state, including its decrypted vtpm, and now the state on the dest is consistent: the key is not loaded on either side. | |
| 20:27:15 | dansmith | exactly | |
| 20:27:48 | efried | um, meaning this is an example of an admin operation resulting in a functional vtpm on an instance that didn't "exist" before (on the dest host) where we didn't need the user creds to retrieve the key. | |
| 20:27:56 | efried | I... guess that's still fine. | |
| 20:28:05 | mriedem | dansmith: replied on the cross-cell change https://review.opendev.org/#/c/635684/53/nova/compute/api.py@3852 | |
| 20:28:41 | dansmith | efried: yes, but they still can't cold migrate or start the guest after a host reboot, but that's expected and "desired" | |
| 20:28:45 | dansmith | from the security perspective | |
| 20:29:04 | efried | I guess it's a slightly different attack surface: if I have admin, and I have compromised root on one host in your cloud, I can get your secret by live-migrating your instance to that host and reading it out of memory. | |
| 20:29:23 | efried | um | |
| 20:29:36 | efried | I can read the *tpm* out of memory, not the secret. | |
| 20:29:41 | sean-k-mooney | it depnds on how qemu store it | |
| 20:29:45 | efried | which is just as bad. | |
| 20:29:54 | dansmith | efried: that's more work than you need to do to get the secret really | |