| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-11-11 | |||
| 15:00:30 | jroll | sigh. | |
| 15:00:34 | dansmith | efried: yeah and my point is virsh and nova look the same to libvirtd, which is the thing that manipulates the guests, xmls, etc | |
| 15:00:39 | jroll | the secret itself is in a file? | |
| 15:01:37 | sean-k-mooney | efried: ideally you would want to store the secret in the hosts tpm rather then on disk | |
| 15:01:51 | efried | dansmith: Okay, I think I understand what you're saying. There's just one libvirtd "server side" and it hears things the same way whether through API or virsh. | |
| 15:02:12 | dansmith | efried: virsh is just an api client like nova, but yes that's what I'm saying | |
| 15:02:13 | efried | jroll: Yes, the secret is in a file. Presumably the fact that said file is encrypted is the thing that makes this more "secure". | |
| 15:02:16 | sean-k-mooney | efried: ya kind of virsh is just like the nova client | |
| 15:02:33 | efried | sean-k-mooney: we're 100% virtual here, there is no TPM on the host. | |
| 15:02:33 | sean-k-mooney | all of the logic is implemetne in the libvirt deamon | |
| 15:02:46 | sean-k-mooney | efried: well there could be | |
| 15:02:54 | efried | for coding purposes, there isn't. | |
| 15:03:07 | efried | jroll: but the fact that it's just sitting around there on the disk means that your disk thief could just... boot up the VM? | |
| 15:03:22 | efried | and would then have to have root access to the VM I guess | |
| 15:03:55 | jroll | efried: the TPM state is in an encrypted file. there is a passphrase to decrypt the file, which we give to libvirt via `virsh secret-set-value` or the API. I'm wondering where libvirt stores that secret, is that also in a file? how does libvirt encrypt that file? (these are rhetorical questions, fwiw, digging now) | |
| 15:04:15 | sean-k-mooney | efried: shoudl we maybe look at barbican to store the secreat rather then libvirt | |
| 15:04:33 | sean-k-mooney | i havent really looked at how this worksin in libvirt yet | |
| 15:04:49 | sean-k-mooney | but ideally i think we dont want it sotred on the compute host right | |
| 15:05:00 | efried | jroll: I'm still not 100% on any of this, but my understanding is that libvirt uses whatever's in the "secret" to encrypt/decrypt the contents of the vTPM. | |
| 15:05:25 | efried | but once you've done the secret-set-value... that's it. | |
| 15:05:47 | jroll | efried: right, trying to figure out where "whatever's in the secret" is stored | |
| 15:05:51 | jroll | where/how | |
| 15:06:01 | sean-k-mooney | it might just be in memory | |
| 15:06:16 | sean-k-mooney | not everythin you do to a libvirt domain has to be stored to disk | |
| 15:06:30 | efried | -rw------- 1 root root 211 Nov 9 12:36 d3e91e0e-20a5-4f60-ae85-9c8bfed6939f.xml | |
| 15:06:30 | efried | -rw------- 1 root root 4 Nov 9 12:36 d3e91e0e-20a5-4f60-ae85-9c8bfed6939f.base64 | |
| 15:06:30 | efried | total 8 | |
| 15:06:30 | efried | stack@breakfast:~/nova$ sudo ls -l /etc/libvirt/secrets/ | |
| 15:06:30 | efried | jroll: | |
| 15:06:45 | jroll | oh goody | |
| 15:06:47 | jroll | thanks. | |
| 15:07:00 | sean-k-mooney | efried: is that encrypted or plain text? | |
| 15:07:10 | jroll | I'm gonna go find a table or three. definitely not to flip. | |
| 15:07:27 | efried | The .base64 is just the b64-encoded value of the passphrase I set | |
| 15:07:43 | efried | the XML is the XML chunk I started with. | |
| 15:07:46 | sean-k-mooney | so plain texted effectivly | |
| 15:07:56 | efried | yup | |
| 15:08:19 | efried | So I've been thinking of a model where we secret-set-value just long enough to boot the VM, then blow away the secret. | |
| 15:08:24 | sean-k-mooney | so you might as well not encrypt the tpm at that point | |
| 15:08:39 | lyarwood | that isn't passed as plain text to QEMU btw | |
| 15:08:42 | efried | ...assuming the tpm stays "unencrypted" while the domain is running. | |
| 15:08:54 | lyarwood | it's the same as with LUKS, it's encypted using the master key for the domain | |
| 15:08:56 | dansmith | efried: ahh, so... fake security then? :D | |
| 15:09:05 | efried | but .... what dansmith said. | |
| 15:09:09 | dansmith | heh | |
| 15:09:16 | efried | that doesn't seem like real security. | |
| 15:09:24 | Erdosip | I think it's under /etc/libvirt/security | |
| 15:09:27 | efried | I mean, I guess your disk thief would have to steal the disk while the VM is booting. | |
| 15:10:29 | sean-k-mooney | im not sure the vm woudl have to be running | |
| 15:10:44 | sean-k-mooney | there are cases were we leave the domain intact | |
| 15:10:52 | sean-k-mooney | such as a showdown form inside the vm | |
| 15:11:01 | sean-k-mooney | or suspend/pause | |
| 15:11:09 | sean-k-mooney | maybe even stop | |
| 15:11:31 | efried | sean-k-mooney: the secret in the domain xml is just the uuid of the secret. So presumably (hopefully) the decryption happens in-memory when the xml is loaded up to actually start the guest. | |
| 15:11:36 | sean-k-mooney | start calls power on which calls reboot which recreates the domain | |
| 15:12:20 | sean-k-mooney | sure but the issue is the secret is not encrypted on disk | |
| 15:12:29 | efried | so regardless of whether the domain xml is there or needs to be recreated, it's when the guest starts up that we would need the secret to be present. | |
| 15:12:53 | lyarwood | again, the secret isn't provided to QEMU as a plaintext string | |
| 15:13:01 | sean-k-mooney | efried: im wondering is there a way to pass it at that point and never save it to disk | |
| 15:13:27 | efried | lyarwood: but it's on disk in cleartext. (base-64 encoded, but effectively plaintext) | |
| 15:13:34 | sean-k-mooney | lyarwood: sure but efried just said it in /etc/libvirt/secrets/ and is just base64 encoded | |
| 15:13:46 | lyarwood | right and that's pretty locked down given the listing above | |
| 15:13:47 | efried | lyarwood: I suspect I'm missing what you're getting at. | |
| 15:14:00 | efried | lyarwood: you mean 600 owned by root locked down? | |
| 15:14:03 | lyarwood | if someone had access to root it's already game over | |
| 15:14:10 | sean-k-mooney | lyarwood: that not good enough in my view | |
| 15:14:22 | efried | pretty sure in jroll's view too. | |
| 15:14:23 | sean-k-mooney | the operator shoudl not be able to see the secreate | |
| 15:14:28 | lyarwood | they could already get in and decrypt various things from the running QEMU process | |
| 15:14:46 | canori01 | Hello, would I evacuate an instance that's running into 'AllocationUpdateFailed: Failed to update allocations for consumer' running on Stein | |
| 15:15:00 | lyarwood | well that just isn't within libvirt's security model AFAIK | |
| 15:15:02 | canori01 | I mean, how would I recover from that | |
| 15:15:31 | efried | lyarwood: Actually I think we're drawing the line at "compromised root is acceptable risk; stolen disk must be secure". | |
| 15:16:07 | jroll | more like we realize compromised root on a running hypervisor is game over anyway, but yeah | |
| 15:16:27 | lyarwood | efried: kk, danpb had a blog about secret handling a while ago that might cover a way around that ./me checks | |
| 15:16:27 | efried | IOW I think it's the nova admin creds that are the root of trust. | |
| 15:16:32 | efried | thanks lyarwood | |
| 15:16:43 | sean-k-mooney | efried: https://libvirt.org/formatsecret.html#SecretAttributes | |
| 15:16:53 | jroll | lyarwood: thanks, I was just going to ask that | |
| 15:16:55 | sean-k-mooney | so it look like you can mare a secret as ephemeral | |
| 15:16:56 | efried | having displayed any understanding whatsoever, lyarwood, you're now permanently on call for support. | |
| 15:17:06 | sean-k-mooney | so its only kept in memory and never on disk | |
| 15:17:17 | sean-k-mooney | so if we store it in barbican and then do that that would be better | |
| 15:17:34 | efried | oh, cool, I saw that but hadn't played with it yet, will have to try that out and see if it does the trick. | |
| 15:17:42 | sean-k-mooney | well ephemeral=yes private=yes | |
| 15:17:51 | efried | The secret I created while playing was private but not ephemeral. | |
| 15:18:14 | jroll | can confirm ephemeral=yes leaves it off of disk | |
| 15:18:35 | sean-k-mooney | so form a nova point of view can we use barbican to store it | |
| 15:18:41 | efried | cool, so the nova user is the root of trust because has access to the barbican entry. | |
| 15:18:49 | efried | yeah, what sean-k-mooney said. | |
| 15:18:56 | jroll | and restarting libvirt nukes an ephemeral secret | |
| 15:18:59 | jroll | \o/ | |
| 15:19:22 | sean-k-mooney | jroll: that is proably ok/good | |
| 15:19:25 | efried | now we just gotta hope you don't have to restart libvirt to make it pick up the secret :P | |
| 15:19:29 | lyarwood | thanks sean-k-mooney | |
| 15:19:46 | jroll | efried: fwiw, I didn't need to restart, but I'm on older libvirt playing with tls secrets, not tpm | |
| 15:20:10 | sean-k-mooney | jroll: i dont know if libvirt makes a difference | |
| 15:20:21 | efried | jroll: didn't need to restart to do what? | |
| 15:20:56 | efried | in my case everything appeared to be moving along fine until I tried booting the guest, then the nova log got an exception complaining it couldn't find my secret. | |
| 15:21:27 | efried | anyway, ephemeral would be entirely useless in that case, so it MUST work (famous last words) | |