| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-11-11 | |||
| 20:22:21 | sean-k-mooney | not storage :) | |
| 20:22:28 | dansmith | otherwise you're talking about usb or firewire | |
| 20:22:35 | dansmith | usb I think | |
| 20:22:43 | sean-k-mooney | ahci is used for sata too | |
| 20:22:48 | efried | ...yeah, confirmed deleting the secret after the guest starts doesn't affect ability of guest to get at the vtpm. | |
| 20:22:53 | dansmith | oh sata right right | |
| 20:23:01 | dansmith | mixed up my *hci acronyms | |
| 20:23:24 | dansmith | efried: almost impossible for that to happen, since qemu and libvirt are fairly separate at runtime | |
| 20:23:33 | dansmith | efried: try a soft reboot from inside the guest tho | |
| 20:23:50 | efried | meaning, from within the guest, just 'reboot'? | |
| 20:23:55 | sean-k-mooney | yes | |
| 20:24:37 | sean-k-mooney | a soft reboot from the api would almost be the same excpet that will escalte to a hard reboot after a time out | |
| 20:25:35 | efried | that seems to have worked fine. Guest came back up, no errors in n-cpu, the tpm is still there and still works. | |
| 20:25:40 | dansmith | figured | |
| 20:25:50 | efried | that's what we want though, right? | |
| 20:25:50 | dansmith | so delete after start is probably the way to go | |
| 20:25:54 | sean-k-mooney | yes | |
| 20:25:57 | dansmith | makes it very clear how long we have the key | |
| 20:26:06 | dansmith | yep | |
| 20:26:08 | efried | like. | |
| 20:27:01 | efried | so assume live migration would carry the VM state, including its decrypted vtpm, and now the state on the dest is consistent: the key is not loaded on either side. | |
| 20:27:15 | dansmith | exactly | |
| 20:27:48 | efried | um, meaning this is an example of an admin operation resulting in a functional vtpm on an instance that didn't "exist" before (on the dest host) where we didn't need the user creds to retrieve the key. | |
| 20:27:56 | efried | I... guess that's still fine. | |
| 20:28:05 | mriedem | dansmith: replied on the cross-cell change https://review.opendev.org/#/c/635684/53/nova/compute/api.py@3852 | |
| 20:28:41 | dansmith | efried: yes, but they still can't cold migrate or start the guest after a host reboot, but that's expected and "desired" | |
| 20:28:45 | dansmith | from the security perspective | |
| 20:29:04 | efried | I guess it's a slightly different attack surface: if I have admin, and I have compromised root on one host in your cloud, I can get your secret by live-migrating your instance to that host and reading it out of memory. | |
| 20:29:23 | efried | um | |
| 20:29:36 | efried | I can read the *tpm* out of memory, not the secret. | |
| 20:29:41 | sean-k-mooney | it depnds on how qemu store it | |
| 20:29:45 | efried | which is just as bad. | |
| 20:29:54 | dansmith | efried: that's more work than you need to do to get the secret really | |
| 20:30:02 | sean-k-mooney | well the tpm data shoudl stilly be encyrpted | |
| 20:30:04 | dansmith | efried: if you're root, you can read all of host memory, you can trigger a crash dump on the qemu, or ptrace it, etc | |
| 20:30:20 | sean-k-mooney | but read and writes to it shoudl be decypted by qemu | |
| 20:30:39 | efried | yeah, point is if you don't have root wherever the instance is currently running, but you do have root somewhere else in the cloud. | |
| 20:30:56 | dansmith | efried: ah yes, sorry.. | |
| 20:31:04 | efried | Is there already a way to mark an instance not migratable? | |
| 20:31:27 | dansmith | you don't even need root anywhere in the cluster to do that attack, fwiw | |
| 20:31:34 | dansmith | all you need is the ability to connect to rabbit and it's over | |
| 20:32:01 | dansmith | given that you can convince a compute you don't even have shell access to to send you the guest's memory contents to a non-secure port | |
| 20:33:23 | sean-k-mooney | so without going on too much of a tangent if you use SGX or other sercure memory enclave technologies you can protect form that | |
| 20:33:36 | sean-k-mooney | but i think that is out of scope fo vTPM | |
| 20:33:50 | sean-k-mooney | with SGX root cant read all memeory | |
| 20:34:06 | sean-k-mooney | and the kernel lockdown feature in 5.4 will also prevent that in some cases | |
| 20:35:27 | dansmith | sean-k-mooney: my attack doesn't require root to be able to read the memory | |
| 20:35:34 | dansmith | I said you don't even need shell on the box with the instance | |
| 20:35:54 | dansmith | my attack might not work with encrypted memory for other reasons, but not because of root | |
| 20:36:17 | sean-k-mooney | sure that was not the point i was makeing | |
| 20:36:19 | sean-k-mooney | https://thenewstack.io/linux-kernel-finally-gets-its-lockdown/ | |
| 20:36:33 | sean-k-mooney | ^ that is a cool feature in 5.4 that shoudl help | |
| 20:36:59 | sean-k-mooney | but anyway i like the idea of removing the key once we boot the instnace | |
| 20:37:16 | efried | So is there a way to indicate I don't want my instance live-migratable? | |
| 20:37:42 | sean-k-mooney | i used to say use sriov/pinning or hugepage but we fixed that :) | |
| 20:37:50 | dansmith | efried: meaning as a user asking the operator to not handle your memory contents over a migration? | |
| 20:37:52 | sean-k-mooney | ill check the libvirt xml | |
| 20:37:54 | sean-k-mooney | maybe | |
| 20:38:18 | dansmith | efried: the user isn't supposed to know that a live migration can or has happened really anyway, so what you're asking for is like an SLA which would happen outside of nova I think | |
| 20:38:30 | efried | dansmith: I guess I meant more like something on the instance so that n-api would simply refuse to migrate | |
| 20:38:50 | dansmith | efried: I know, and I'm saying that's not an appropriate thing for nova to be doing IMHO | |
| 20:38:51 | efried | other than incidentals, like PCI devices (right?), I mean something specific. | |
| 20:39:01 | sean-k-mooney | efried: i mean we coudl just check for vtpm and reject | |
| 20:39:06 | dansmith | no, | |
| 20:39:17 | dansmith | sean-k-mooney: he's saying some user wanting to not be live migrated for risk, | |
| 20:39:25 | dansmith | not nova refusing to migrate vtpm-having instances | |
| 20:39:30 | sean-k-mooney | ah | |
| 20:39:34 | efried | either user (so in the flavor) or admin (conf opt) | |
| 20:39:39 | dansmith | if we can migrate them (which we can I think) then we should | |
| 20:39:46 | dansmith | efried: user doesn't control the flavor | |
| 20:39:48 | sean-k-mooney | thats a similar discustion to the spec fo exposing post copy | |
| 20:39:51 | dansmith | the *details* of the flavor | |
| 20:40:22 | dansmith | but yeah, I'd say the cloud should say "Flavors that end in -secu will never be live migrated for security reasons" and then just handle that policy themselves | |
| 20:40:30 | sean-k-mooney | in that spec they wanted a way to mark a vm as not using post copy to avoid the risk of remote execution over a netwrok link | |
| 20:42:38 | umbSublime | That's something I was asked today. Is there a way to make a vm/flavor _NOT_ live-migratable. Some of our users would for some reason want to opt-out of being live-migratable and understand what that means if we need to do maintenance on the host | |
| 20:45:13 | sean-k-mooney | fyi just looking at the libvirt level i dont see anything in https://libvirt.org/formatdomain.html that would alow us to mark a vm as not live migratable but we could do int at the nova level | |
| 20:45:39 | efried | hm, there's a side effect of the hack that's necessary to transfer the vtpm data meaning that you could disable migration by simply not setting those conf opts. | |
| 20:45:51 | dansmith | sean-k-mooney: I assumed he was looking for a nova tunable not libvirt | |
| 20:46:11 | sean-k-mooney | ya i just said i would look at both levels | |
| 20:46:21 | dansmith | ack | |
| 20:47:16 | sean-k-mooney | efried: which conf options? | |
| 20:47:37 | efried | https://review.opendev.org/#/c/639934/14/nova/conf/libvirt.py | |
| 20:47:42 | sean-k-mooney | the nova migration uri? | |
| 20:48:24 | sean-k-mooney | that would block you form using swtpm too however right | |
| 20:49:00 | sean-k-mooney | why is swtpm_group a choices field | |
| 20:49:14 | sean-k-mooney | that kindof ties distos hands | |
| 20:49:22 | sean-k-mooney | same for user i guess | |
| 20:50:51 | sean-k-mooney | efried: with may kolla-ansible hat on it also forces me to run the swtpm as a specific user or group which is not ideal | |
| 20:52:31 | efried | sean-k-mooney: IIUC the libvirt process takes care of starting the swtpm. | |
| 20:52:47 | efried | I mean, I didn't have to do anything special to start it. | |
| 20:53:15 | sean-k-mooney | ok but im not sure we should be resticting the options in nova | |
| 20:53:33 | sean-k-mooney | e.g. i think that should just be a string field | |
| 20:54:12 | sean-k-mooney | if you do "ps aux | grep swtpm" what user/group is it running as | |
| 20:55:46 | efried | some processes have a `tss` user, some have `root`. | |
| 20:56:28 | efried | sorry, the root ones are just qemu. They have swtpm CLI opts in 'em. | |
| 22:21:40 | efried | Not sure why I shouldn't determine the uid/gid based on the swtpm process running on the dest host. | |
| 22:22:35 | efried | I guess discovering which process that is could be a bit tricky. | |
| 22:26:13 | efried | oh, ahem, the swtpm processes correspond to instances; they don't exist until the tpm does, chicken/egg. | |
| 22:27:15 | sean-k-mooney | efried: is this in relattion to the config options or something else? | |
| 22:27:22 | sean-k-mooney | *relation | |