| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-07-26 | |||
| 23:40:43 | johnsom | Yeah, this is what we were told to use to drop cert chains in via config drive years ago. The issue was user-data was not secure as it is stored in the DB and it had a really small max content size | |
| 23:40:43 | rm_work | which config file | |
| 23:40:48 | dansmith | because we're totally willy-nilly with that | |
| 23:41:09 | dansmith | johnsom: who told you that? | |
| 23:41:16 | rm_work | like, this? http://logs.openstack.org/49/552549/13/check/octavia-v2-dsvm-scenario/f7f027d/controller/logs/etc/nova/nova_conf.txt.gz | |
| 23:41:20 | dansmith | johnsom: this has been on its death bed for, gosh, five years at least | |
| 23:41:42 | dansmith | rm_work: that's on a controller | |
| 23:42:01 | rm_work | don't know where else to look? | |
| 23:42:03 | johnsom | The nova PTL at the time we were spinning this project up. I know I tried to use user-data last year and it was still pretty small. I couldn't load the test app I was hoping to drop into cirros | |
| 23:42:05 | rm_work | this is a single-node test | |
| 23:42:41 | dansmith | rm_work: okay, then probably the same, so yeah it's disabled there for actual file injection | |
| 23:42:46 | johnsom | We don't do anything special to enable that on devstack in the gates | |
| 23:42:48 | rm_work | then ... how is this working | |
| 23:42:51 | dansmith | I'll have to look and see if and how it ends up in the config drive | |
| 23:43:20 | dansmith | but, your assertion about the "it isn't stored is good for security" is a huge problem, because we can't migrate or rebuild or anything the instance | |
| 23:43:27 | dansmith | and don't have a way to provide that stuff again you know | |
| 23:43:28 | dansmith | and | |
| 23:43:43 | dansmith | storing anything sensitive in configdrive is kinda crazy :) | |
| 23:43:59 | rm_work | we don't WANT to migrate or restore the instances | |
| 23:44:02 | rm_work | that would explicitly be bad | |
| 23:44:09 | rm_work | if an instance dies, it's dead and we trash it | |
| 23:44:13 | rm_work | and we will handle making a replacement | |
| 23:44:25 | johnsom | Well some have use cases for migrate, but the content should migrate with the config drive | |
| 23:45:05 | dansmith | rm_work: okay but you understand that nova is a thing that lots of people use, and most want to be able to migrate, and if we break that model for users that don't know anything then, that's like bad and stuff yeah? :) | |
| 23:45:08 | dansmith | I mean, | |
| 23:45:15 | dansmith | nova has a wider audience :) | |
| 23:45:32 | dansmith | if you want to enforce different lifecycle semantics on the thing you use nova for, then that's cool, but.. | |
| 23:45:36 | johnsom | Yeah, we have use cases for migrating. I think that is one operator talking | |
| 23:45:43 | rm_work | yes, but BECAUSE of that philosophy, because something doesn't work for a couple of use cases, you shouldn't remove it when it still works for others? | |
| 23:45:55 | rm_work | seems like it should work both ways :P | |
| 23:46:15 | johnsom | Is the supported data size for "user-data" documented? I can't find it | |
| 23:46:29 | rm_work | isn't saying "we should kill your use-case, because nova has to be generic for lots of use-cases" seems kinda backward | |
| 23:46:32 | johnsom | beyond "a small opaque blob of data" | |
| 23:46:51 | dansmith | here's where it shows up in the config drive: https://github.com/openstack/nova/blob/master/nova/api/metadata/base.py#L191-L194 | |
| 23:48:16 | dansmith | rm_work: naw dude, but you know, we try to make features that are rather predictable and understandable from the outside, especially by the users that don't have a view into the config of the deployment | |
| 23:48:26 | dansmith | rm_work: it's kindof what we do here. we're trying to be an abstraction | |
| 23:48:49 | dansmith | rm_work: and, as we said, we have a method that *does* work for all cases, except it has a size limitation for you, which I get | |
| 23:49:10 | dansmith | and the other thing had some really terrible security semantics that I guess you didn't know about :) | |
| 23:49:15 | melwitt | johnsom: looks like this is the size limit 16384 bytes https://help.ubuntu.com/community/CloudInit#fnref-6a17aff5cf9eab5a99b5ab3dbd49854b0e9681c1 | |
| 23:49:43 | dansmith | johnsom: it's a MediumText in the db | |
| 23:49:48 | dansmith | so that's likely the real limit | |
| 23:49:56 | melwitt | oh, okay | |
| 23:50:03 | rm_work | what are the terrible security semantics for the other thing? it mentions "evil images could do whatever with the data", but if we only boot with a known good image... | |
| 23:50:08 | johnsom | lol, yeah, 16k seems about what I ran into last year. | |
| 23:50:09 | openstackgerrit | Merged openstack/nova master: Stop using HostAPI.service_delete https://review.openstack.org/582680 | |
| 23:50:27 | dansmith | api limits it at 64k, FWIW | |
| 23:50:38 | johnsom | Yeah, I mean if you don't trust the images... the whole guest OS is at risk | |
| 23:50:39 | dansmith | https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/schemas/servers.py#L212-L216 | |
| 23:51:53 | rm_work | so what is the new alternative? we have to create a cinder volume for every VM we boot, and pass that to the server create? | |
| 23:52:31 | rm_work | since user_data is obviously unacceptable for multiple reasons | |
| 23:52:44 | rm_work | and I'm not seeing how that helps at all with the "evil image" case | |
| 23:52:50 | dansmith | the new alternative is user_data, and providing this feedback not on FF day at 5pm when we can actually do something for you :) | |
| 23:53:01 | rm_work | user_data is explicitly insecure | |
| 23:53:18 | rm_work | and size-limited to a point where it is not even possible to use for a very common use-case (passing in a cert chain) | |
| 23:53:43 | rm_work | we'll just have to revisit this later, it just happened that we discovered this *just now* | |
| 23:53:51 | dansmith | how much data do you expect nova to keep track of for you through our control API? | |
| 23:54:06 | rm_work | i don't want nova to keep track of ANY data | |
| 23:54:12 | rm_work | i just want the data passed once to the booting VM | |
| 23:54:15 | rm_work | and not tracked | |
| 23:54:26 | dansmith | how much data do you want to be able to pass to nova once? | |
| 23:54:35 | dansmith | like, should we let you include a 50G custom image instead of a ref? | |
| 23:54:48 | Sundar | efried: please ping me when you have the time. | |
| 23:55:08 | rm_work | i mean, i understand the sarcasm (?), but it does seem seriously small | |
| 23:55:49 | rm_work | but even if it wasn't too small, user_data is even more insecure | |
| 23:55:52 | dansmith | well, obviously I'm being hyperbolic here, but 16k for an identifier or something that lets you fetch larger things from something intended to serve such things seems pretty reasonable to me | |
| 23:55:57 | rm_work | it just stores the data directly in the DB unencrypted | |
| 23:56:28 | rm_work | which means it's just not viable from the get-go | |
| 23:56:33 | dansmith | I think you're missing the point of how insecure config drive is, but also, | |
| 23:56:43 | rm_work | please inform me! | |
| 23:56:50 | dansmith | we have a real problem with several of our major lifecycle operations not being possible if we let you pass us stuff that we can't reproduce later | |
| 23:56:55 | rm_work | I want to know what the security considerations you keep referring to are | |
| 23:57:52 | dansmith | we stored config drive unencrypted on the disk, we built it from writing the tree out somewhere in /tmp first, we sent all your data over the unencrytped message bus to anyone claiming with no auth to be the compute node in question | |
| 23:58:29 | rm_work | 1) unencrypted on the HV's disk? | |
| 23:58:44 | dansmith | yup, or on nfs if you're using that, so that's even better | |
| 23:58:47 | rm_work | 2) unencrypted message bus? you mean RMQ? we use TLS for that with client cert auth -- don't others? | |
| 23:59:28 | dansmith | rm_work: sure, some do, but there's no auth at the application layer.. if I can connect to RMQ, I can impersonate any compute node on the system until they realize and cut me off | |
| 23:59:30 | rm_work | for 1, if we lose the HV's integrity, it barely matters what else is secure | |
| 23:59:56 | rm_work | a HV-level breakout is basically game-over | |
| #openstack-nova - 2018-07-27 | |||
| 00:00:24 | rm_work | so I'm still not hearing anything that is a problem for my deployment | |
| 00:00:33 | dansmith | heh, okay | |
| 00:00:43 | rm_work | and if I can do it, others can do it | |
| 00:00:49 | dansmith | here's one problem: file injection has been deprecated :P | |
| 00:00:53 | rm_work | yes, thanks | |
| 00:01:05 | rm_work | I happened to notice that recently ;P | |
| 00:02:07 | penick | So the root problem is you need to put arbitrary files in an instance, or you need instances to have x509 certs chains? | |
| 00:02:16 | rm_work | i don't think we're going to get anywhere today on this, maybe we pick up at the PTG | |
| 00:02:34 | rm_work | right now it's a cert-chain, PK, and agent config | |
| 00:02:40 | rm_work | so I guess "arbitrary files" | |
| 00:03:36 | rm_work | and they all contain data that we consider "sensitive" | |
| 00:03:37 | rm_work | (obviously, in the case of the PK) | |
| 00:03:41 | penick | Are they "shared" secrets, like the keypair for a public website? Barbican might be the right place for those. | |
| 00:03:44 | rm_work | well, specifically the PK | |
| 00:03:56 | rm_work | we do use Barbican, but our instances have no way to auth against it | |
| 00:04:02 | rm_work | one is shared | |
| 00:04:10 | rm_work | the other is generated specifically for the VM in question | |
| 00:04:15 | rm_work | (the PK) | |
| 00:05:19 | penick | We generate secrets on our instances, then have another system the instances call to have their csr signed, it asserts their identity before it's signed by our root of trust | |
| 00:05:31 | rm_work | ok so that there is the important bit | |
| 00:05:41 | johnsom | Speaking of FF - had to go take care of that. Yeah I think at least of floppy disk's worth of storage is reasonable. lol Like the PXE boot image size. | |
| 00:05:43 | rm_work | we USE those certs/PK to assert identity | |