| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-07-26 | |||
| 23:55:52 | dansmith | well, obviously I'm being hyperbolic here, but 16k for an identifier or something that lets you fetch larger things from something intended to serve such things seems pretty reasonable to me | |
| 23:55:57 | rm_work | it just stores the data directly in the DB unencrypted | |
| 23:56:28 | rm_work | which means it's just not viable from the get-go | |
| 23:56:33 | dansmith | I think you're missing the point of how insecure config drive is, but also, | |
| 23:56:43 | rm_work | please inform me! | |
| 23:56:50 | dansmith | we have a real problem with several of our major lifecycle operations not being possible if we let you pass us stuff that we can't reproduce later | |
| 23:56:55 | rm_work | I want to know what the security considerations you keep referring to are | |
| 23:57:52 | dansmith | we stored config drive unencrypted on the disk, we built it from writing the tree out somewhere in /tmp first, we sent all your data over the unencrytped message bus to anyone claiming with no auth to be the compute node in question | |
| 23:58:29 | rm_work | 1) unencrypted on the HV's disk? | |
| 23:58:44 | dansmith | yup, or on nfs if you're using that, so that's even better | |
| 23:58:47 | rm_work | 2) unencrypted message bus? you mean RMQ? we use TLS for that with client cert auth -- don't others? | |
| 23:59:28 | dansmith | rm_work: sure, some do, but there's no auth at the application layer.. if I can connect to RMQ, I can impersonate any compute node on the system until they realize and cut me off | |
| 23:59:30 | rm_work | for 1, if we lose the HV's integrity, it barely matters what else is secure | |
| 23:59:56 | rm_work | a HV-level breakout is basically game-over | |
| #openstack-nova - 2018-07-27 | |||
| 00:00:24 | rm_work | so I'm still not hearing anything that is a problem for my deployment | |
| 00:00:33 | dansmith | heh, okay | |
| 00:00:43 | rm_work | and if I can do it, others can do it | |
| 00:00:49 | dansmith | here's one problem: file injection has been deprecated :P | |
| 00:00:53 | rm_work | yes, thanks | |
| 00:01:05 | rm_work | I happened to notice that recently ;P | |
| 00:02:07 | penick | So the root problem is you need to put arbitrary files in an instance, or you need instances to have x509 certs chains? | |
| 00:02:16 | rm_work | i don't think we're going to get anywhere today on this, maybe we pick up at the PTG | |
| 00:02:34 | rm_work | right now it's a cert-chain, PK, and agent config | |
| 00:02:40 | rm_work | so I guess "arbitrary files" | |
| 00:03:36 | rm_work | and they all contain data that we consider "sensitive" | |
| 00:03:37 | rm_work | (obviously, in the case of the PK) | |
| 00:03:41 | penick | Are they "shared" secrets, like the keypair for a public website? Barbican might be the right place for those. | |
| 00:03:44 | rm_work | well, specifically the PK | |
| 00:03:56 | rm_work | we do use Barbican, but our instances have no way to auth against it | |
| 00:04:02 | rm_work | one is shared | |
| 00:04:10 | rm_work | the other is generated specifically for the VM in question | |
| 00:04:15 | rm_work | (the PK) | |
| 00:05:19 | penick | We generate secrets on our instances, then have another system the instances call to have their csr signed, it asserts their identity before it's signed by our root of trust | |
| 00:05:31 | rm_work | ok so that there is the important bit | |
| 00:05:41 | johnsom | Speaking of FF - had to go take care of that. Yeah I think at least of floppy disk's worth of storage is reasonable. lol Like the PXE boot image size. | |
| 00:05:43 | rm_work | we USE those certs/PK to assert identity | |
| 00:05:57 | rm_work | how do you assert VM's identity without that? | |
| 00:06:09 | rm_work | i mean, that is exactly our workflow | |
| 00:06:22 | rm_work | well ... ALMOST our workflow | |
| 00:06:30 | rm_work | we reach out to the VM, not the other way around | |
| 00:06:32 | johnsom | Yeah, this was the whole discussion that led use to what was implemented years ago. | |
| 00:06:41 | dansmith | penick is headed down the right path here, which is not to pass everything to nova and expect it to keep it (most people) or disavow it (some people), and only give nova enough information to let you interact with some service that can do what you want | |
| 00:06:58 | dansmith | information that is not sensitive forever | |
| 00:07:27 | johnsom | I'm just concerned that if we don't trust how we store and handle images we are in trouble before we even get to config data and establishing secure channels. | |
| 00:08:04 | penick | We create a signed bearer document that's time limited and place it in the instance, on boot the instance creates a PK and CSR, then sends those along with the attestation document (created as part of vendor data) to the token server, which verifies the signature in the attestation document (and then invalidates the document) then calls openstack to verify the details in the CSR | |
| 00:08:24 | rm_work | which "we" is that? | |
| 00:08:25 | penick | eg, ensure the IP, UUID, etc in the CSR match the instance | |
| 00:08:26 | rm_work | which service | |
| 00:08:40 | rm_work | because that does sound like the workflow we're aiming for | |
| 00:08:56 | penick | the service is called Athenz, and the system we've built to integrate it into OpenStack is called copper argos | |
| 00:09:22 | penick | I have a talk on it, one sec.. | |
| 00:09:31 | rm_work | i was hoping to just glance at the repo | |
| 00:09:58 | rm_work | https://github.com/yahoo/athenz ? | |
| 00:10:05 | penick | https://www.openstack.org/videos/vancouver-2018/attestable-service-identity-with-copper-argos | |
| 00:10:13 | penick | yup | |
| 00:10:16 | rm_work | https://github.com/yahoo/athenz/blob/master/docs/copper_argos_dev.md | |
| 00:10:37 | penick | Ayup, that's it | |
| 00:11:10 | rm_work | so basically, we're screwed once Stein hits, and we have to get something like this working before then? :P | |
| 00:11:27 | rm_work | sounds like another day at the office, lol | |
| 00:11:29 | penick | I feel like it benefits me to say Yes :) | |
| 00:11:40 | rm_work | we'll investigate | |
| 00:11:50 | dansmith | rm_work: you should really read the spec you're freaking out about | |
| 00:11:54 | rm_work | I did | |
| 00:11:55 | dansmith | "Since personality file injection will still be supported with older microversions, there will be nothing removed from the backend compute code related to file injection" | |
| 00:12:08 | penick | We're eager to have other people use this, so lmk if y'all (who..are..you?) are interested in using Athenz. It'd be good to get other organizations using/contributing to Athenz | |
| 00:12:14 | rm_work | yeah, but in Octavia we don't necessarily control the nova deployments | |
| 00:12:21 | rm_work | so we can't guarantee they have the thing enabled | |
| 00:12:25 | rm_work | but we still need our stuff to work | |
| 00:12:38 | dansmith | rm_work: oooh, I have good news for you | |
| 00:12:42 | rm_work | penick: we'd be writing something like that into Octavia | |
| 00:12:59 | dansmith | rm_work: user_data will always work? see how nice it is to have features that don't come and go with the deployment choices? :) | |
| 00:13:03 | rm_work | lol | |
| 00:13:15 | rm_work | except user-data already doesn't work :P | |
| 00:13:26 | johnsom | Well, nova is a stable api, so it shouldn't be going away any time soon or they are dropping their stable assertion.... | |
| 00:13:28 | penick | We'll be using octavia with this in the near future. It's one of the things we have to suss out this qtr | |
| 00:13:34 | dansmith | you mean jamming a bus into your wallet won't work | |
| 00:13:35 | penick | But, we already have Athenz in place | |
| 00:13:51 | penick | dansmith: Well not with that attitude | |
| 00:13:57 | dansmith | johnsom: that's what I'm trying to point out | |
| 00:14:08 | rm_work | but you're saying it's already disabled in most nova deploys? | |
| 00:14:25 | dansmith | johnsom: which is what you get if you read a paragraph down below "and now lose your mind" | |
| 00:14:42 | dansmith | rm_work: no, we're saying that file injection is disabled, but as you pointed out we're putting those personality files into the config drive the first time we make it | |
| 00:15:13 | rm_work | [16:38:53] <dansmith>so this has been disabled by default for libvirt for a long time, | |
| 00:15:16 | rm_work | ^^ so what did that mean? | |
| 00:15:24 | dansmith | rm_work: file. injection. | |
| 00:15:41 | rm_work | yes, which has always worked via personality files? | |
| 00:15:48 | dansmith | rm_work: you saw the part where I said "I'm not sure how this is going into config drive" and then ... found and quoted the code right? | |
| 00:15:54 | rm_work | maybe? | |
| 00:16:00 | johnsom | dansmith I was shocked because we hadn't heard of this and it was the *way* to do this securely and reliably and user-data was .... less than ideal | |
| 00:16:35 | rm_work | https://github.com/openstack/nova/blob/master/nova/api/metadata/base.py#L191-L194 this link? | |
| 00:16:43 | rm_work | I thought that was via libvirt using the thing you said was disabled | |
| 00:16:48 | dansmith | johnsom: you know that config drive is disable-able and depending on it is also not reliable yeah? | |
| 00:16:58 | dansmith | rm_work: no | |
| 00:17:08 | dansmith | rm_work: I get that it says libvirt there, but... | |
| 00:17:35 | johnsom | dansmith We force require it as the metadata service was swiss cheese and blew up if you booted more than a few instances at a time | |
| 00:17:35 | rm_work | if that's not "file injection" then I don't know | |
| 00:17:39 | dansmith | rm_work: the rest of the spec is talking about file injection specifically, which has nothing to do with config drive and is all about violating the very sanctity of the image by forcing large things into small holes | |
| 00:18:10 | rm_work | err | |
| 00:18:16 | penick | rm_work what's generating the secrets that you're putting into the instance? (amphora vms?) | |
| 00:18:16 | rm_work | so *are we using file injection or not*? | |