Posted	Nick	Remark
#openstack-nova - 2018-07-26
23:37:27	johnsom	no
23:37:30	rm_work	https://github.com/openstack/python-novaclient/blob/master/novaclient/v2/servers.py#L1298-L1300
23:37:40	rm_work	https://github.com/openstack/octavia/blob/master/octavia/compute/drivers/nova_driver.py#L150
23:37:46	dansmith	johnsom: cool, then this isn't what you're looking for
23:37:51	rm_work	^^ dansmith see that
23:37:55	rm_work	I think the answer is yes?
23:38:07	rm_work	we pass "files" to the server create
23:38:35	dansmith	yeah, I think files in the client goes to personality
23:38:39	rm_work	https://github.com/openstack/python-novaclient/blob/master/novaclient/v2/servers.py#L688-L692
23:38:40	johnsom	Does "files" in the client get converted to personalities in the nova API?
23:38:42	rm_work	yes
23:38:53	dansmith	so this has been disabled by default for libvirt for a long time,
23:39:03	dansmith	and strongly encouraged not to be used for longer
23:39:14	dansmith	I guess you have been turning it on in the compute node?
23:39:17	rm_work	we even say as much here: https://github.com/openstack/octavia/blob/920a15c620eada38f40d827c89fb9881ffd29fdb/octavia/common/config.py#L368-L370
23:39:32	rm_work	dansmith: i don't think we have done anything to force it to be enabled in our gates...
23:39:42	dansmith	rm_work: it's off by default
23:39:49	dansmith	in the libvirt driver
23:39:53	rm_work	could we tell from nova logs from a gate run?
23:39:56	dansmith	not sure if the other drivers even support it
23:40:12	dansmith	rm_work: just look at the config files from the gate run, like inject_file_partition or whatever
23:40:41	dansmith	I didn't think that goes into the config drive, fwiw.. and that would be bad for security if it did and you use it for sensitive things,
23:40:43	rm_work	which config file
23:40:43	johnsom	Yeah, this is what we were told to use to drop cert chains in via config drive years ago. The issue was user-data was not secure as it is stored in the DB and it had a really small max content size
23:40:48	dansmith	because we're totally willy-nilly with that
23:41:09	dansmith	johnsom: who told you that?
23:41:16	rm_work	like, this? http://logs.openstack.org/49/552549/13/check/octavia-v2-dsvm-scenario/f7f027d/controller/logs/etc/nova/nova_conf.txt.gz
23:41:20	dansmith	johnsom: this has been on its death bed for, gosh, five years at least
23:41:42	dansmith	rm_work: that's on a controller
23:42:01	rm_work	don't know where else to look?
23:42:03	johnsom	The nova PTL at the time we were spinning this project up. I know I tried to use user-data last year and it was still pretty small. I couldn't load the test app I was hoping to drop into cirros
23:42:05	rm_work	this is a single-node test
23:42:41	dansmith	rm_work: okay, then probably the same, so yeah it's disabled there for actual file injection
23:42:46	johnsom	We don't do anything special to enable that on devstack in the gates
23:42:48	rm_work	then ... how is this working
23:42:51	dansmith	I'll have to look and see if and how it ends up in the config drive
23:43:20	dansmith	but, your assertion about the "it isn't stored is good for security" is a huge problem, because we can't migrate or rebuild or anything the instance
23:43:27	dansmith	and don't have a way to provide that stuff again you know
23:43:28	dansmith	and
23:43:43	dansmith	storing anything sensitive in configdrive is kinda crazy :)
23:43:59	rm_work	we don't WANT to migrate or restore the instances
23:44:02	rm_work	that would explicitly be bad
23:44:09	rm_work	if an instance dies, it's dead and we trash it
23:44:13	rm_work	and we will handle making a replacement
23:44:25	johnsom	Well some have use cases for migrate, but the content should migrate with the config drive
23:45:05	dansmith	rm_work: okay but you understand that nova is a thing that lots of people use, and most want to be able to migrate, and if we break that model for users that don't know anything then, that's like bad and stuff yeah? :)
23:45:08	dansmith	I mean,
23:45:15	dansmith	nova has a wider audience :)
23:45:32	dansmith	if you want to enforce different lifecycle semantics on the thing you use nova for, then that's cool, but..
23:45:36	johnsom	Yeah, we have use cases for migrating. I think that is one operator talking
23:45:43	rm_work	yes, but BECAUSE of that philosophy, because something doesn't work for a couple of use cases, you shouldn't remove it when it still works for others?
23:45:55	rm_work	seems like it should work both ways :P
23:46:15	johnsom	Is the supported data size for "user-data" documented? I can't find it
23:46:29	rm_work	isn't saying "we should kill your use-case, because nova has to be generic for lots of use-cases" seems kinda backward
23:46:32	johnsom	beyond "a small opaque blob of data"
23:46:51	dansmith	here's where it shows up in the config drive: https://github.com/openstack/nova/blob/master/nova/api/metadata/base.py#L191-L194
23:48:16	dansmith	rm_work: naw dude, but you know, we try to make features that are rather predictable and understandable from the outside, especially by the users that don't have a view into the config of the deployment
23:48:26	dansmith	rm_work: it's kindof what we do here. we're trying to be an abstraction
23:48:49	dansmith	rm_work: and, as we said, we have a method that *does* work for all cases, except it has a size limitation for you, which I get
23:49:10	dansmith	and the other thing had some really terrible security semantics that I guess you didn't know about :)
23:49:15	melwitt	johnsom: looks like this is the size limit 16384 bytes https://help.ubuntu.com/community/CloudInit#fnref-6a17aff5cf9eab5a99b5ab3dbd49854b0e9681c1
23:49:43	dansmith	johnsom: it's a MediumText in the db
23:49:48	dansmith	so that's likely the real limit
23:49:56	melwitt	oh, okay
23:50:03	rm_work	what are the terrible security semantics for the other thing? it mentions "evil images could do whatever with the data", but if we only boot with a known good image...
23:50:08	johnsom	lol, yeah, 16k seems about what I ran into last year.
23:50:09	openstackgerrit	Merged openstack/nova master: Stop using HostAPI.service_delete https://review.openstack.org/582680
23:50:27	dansmith	api limits it at 64k, FWIW
23:50:38	johnsom	Yeah, I mean if you don't trust the images... the whole guest OS is at risk
23:50:39	dansmith	https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/schemas/servers.py#L212-L216
23:51:53	rm_work	so what is the new alternative? we have to create a cinder volume for every VM we boot, and pass that to the server create?
23:52:31	rm_work	since user_data is obviously unacceptable for multiple reasons
23:52:44	rm_work	and I'm not seeing how that helps at all with the "evil image" case
23:52:50	dansmith	the new alternative is user_data, and providing this feedback not on FF day at 5pm when we can actually do something for you :)
23:53:01	rm_work	user_data is explicitly insecure
23:53:18	rm_work	and size-limited to a point where it is not even possible to use for a very common use-case (passing in a cert chain)
23:53:43	rm_work	we'll just have to revisit this later, it just happened that we discovered this *just now*
23:53:51	dansmith	how much data do you expect nova to keep track of for you through our control API?
23:54:06	rm_work	i don't want nova to keep track of ANY data
23:54:12	rm_work	i just want the data passed once to the booting VM
23:54:15	rm_work	and not tracked
23:54:26	dansmith	how much data do you want to be able to pass to nova once?
23:54:35	dansmith	like, should we let you include a 50G custom image instead of a ref?
23:54:48	Sundar	efried: please ping me when you have the time.
23:55:08	rm_work	i mean, i understand the sarcasm (?), but it does seem seriously small
23:55:49	rm_work	but even if it wasn't too small, user_data is even more insecure
23:55:52	dansmith	well, obviously I'm being hyperbolic here, but 16k for an identifier or something that lets you fetch larger things from something intended to serve such things seems pretty reasonable to me
23:55:57	rm_work	it just stores the data directly in the DB unencrypted
23:56:28	rm_work	which means it's just not viable from the get-go
23:56:33	dansmith	I think you're missing the point of how insecure config drive is, but also,
23:56:43	rm_work	please inform me!
23:56:50	dansmith	we have a real problem with several of our major lifecycle operations not being possible if we let you pass us stuff that we can't reproduce later
23:56:55	rm_work	I want to know what the security considerations you keep referring to are
23:57:52	dansmith	we stored config drive unencrypted on the disk, we built it from writing the tree out somewhere in /tmp first, we sent all your data over the unencrytped message bus to anyone claiming with no auth to be the compute node in question
23:58:29	rm_work	1) unencrypted on the HV's disk?
23:58:44	dansmith	yup, or on nfs if you're using that, so that's even better
23:58:47	rm_work	2) unencrypted message bus? you mean RMQ? we use TLS for that with client cert auth -- don't others?
23:59:28	dansmith	rm_work: sure, some do, but there's no auth at the application layer.. if I can connect to RMQ, I can impersonate any compute node on the system until they realize and cut me off
23:59:30	rm_work	for 1, if we lose the HV's integrity, it barely matters what else is secure
23:59:56	rm_work	a HV-level breakout is basically game-over

Earlier   Later