Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-07-26
22:48:12 dansmith yeah, revert is too old I'm sure
22:48:20 dansmith just have to mangle it manually
22:50:19 melwitt yeah, okay. I think we should because from what dansmith has explained, using this in its current state could badly corrupt a deployment and then we'd have to come up with some code to migrate people out of it if it happens to them
22:50:40 dansmith it'd be heal_allocations I guess
22:50:43 dansmith but still
22:54:49 dansmith might be able to functional test our way
22:55:30 melwitt I'll put this on the rc1 todo list https://etherpad.openstack.org/p/nova-rocky-release-candidate-todo
22:55:41 melwitt and get efried's thoughts tomorrow morning
23:22:34 rm_work hey nova folks! the deprecation discussed here: https://review.openstack.org/#/c/522027/ / https://specs.openstack.org/openstack/nova-specs/specs/queens/implemented/deprecate-file-injection.html -- is that what we're using here? https://github.com/openstack/octavia/blob/master/octavia/compute/drivers/nova_driver.py#L150
23:26:04 melwitt rm_work: yes, that is file injection, deprecated with microversion 2.57 https://github.com/openstack/python-novaclient/blob/master/novaclient/v2/servers.py#L1210-L1215
23:26:29 rm_work johnsom: ^^
23:27:08 johnsom melwitt What? You removed the ability to load data via config drive? That will break a ton of services.
23:27:41 melwitt johnsom: no, the ability to specify files to inject into the server. the supported way to do that is user data
23:28:33 johnsom melwitt but user_data was supposed to go away given it's non-standard and significantly limited in size.
23:29:21 melwitt I think you might be reversing the two? user_data isn't going away and is supported by cloud-init
23:29:24 johnsom Wow, this went under the radar. I know it impacts Octavia and Trove, probably manila and a few others
23:30:11 johnsom The file method we were using is supported by cloud-init via config drive.
23:31:31 johnsom What we are talking about is in the "content" folder in config drive
23:31:38 rm_work it's interesting because in https://specs.openstack.org/openstack/nova-specs/specs/queens/implemented/deprecate-file-injection.html#problem-description
23:32:11 rm_work #1 is not a problem for services that use fixed images, #2 is a GOOD thing (for security), and #3 is simply false, because the alternatives are worse (bad security)
23:33:22 johnsom Yeah user-data has a ridiculously small size limitation too. Like you can't even get a full certificate chain in it.
23:35:13 melwitt okay, I don't know that much detail about the deprecation. so it'd be best to visit the channel tomorrow when people who know more about it are around
23:35:32 melwitt as for the use case, the only other way I know of to handle something like that would be vendor data
23:37:14 dansmith rm_work: johnsom just to be clear, you're talking about putting stuff in the "personality" part of the server post right?
23:37:27 johnsom no
23:37:30 rm_work https://github.com/openstack/python-novaclient/blob/master/novaclient/v2/servers.py#L1298-L1300
23:37:40 rm_work https://github.com/openstack/octavia/blob/master/octavia/compute/drivers/nova_driver.py#L150
23:37:46 dansmith johnsom: cool, then this isn't what you're looking for
23:37:51 rm_work ^^ dansmith see that
23:37:55 rm_work I think the answer is yes?
23:38:07 rm_work we pass "files" to the server create
23:38:35 dansmith yeah, I think files in the client goes to personality
23:38:39 rm_work https://github.com/openstack/python-novaclient/blob/master/novaclient/v2/servers.py#L688-L692
23:38:40 johnsom Does "files" in the client get converted to personalities in the nova API?
23:38:42 rm_work yes
23:38:53 dansmith so this has been disabled by default for libvirt for a long time,
23:39:03 dansmith and strongly encouraged not to be used for longer
23:39:14 dansmith I guess you have been turning it on in the compute node?
23:39:17 rm_work we even say as much here: https://github.com/openstack/octavia/blob/920a15c620eada38f40d827c89fb9881ffd29fdb/octavia/common/config.py#L368-L370
23:39:32 rm_work dansmith: i don't think we have done anything to force it to be enabled in our gates...
23:39:42 dansmith rm_work: it's off by default
23:39:49 dansmith in the libvirt driver
23:39:53 rm_work could we tell from nova logs from a gate run?
23:39:56 dansmith not sure if the other drivers even support it
23:40:12 dansmith rm_work: just look at the config files from the gate run, like inject_file_partition or whatever
23:40:41 dansmith I didn't think that goes into the config drive, fwiw.. and that would be bad for security if it did and you use it for sensitive things,
23:40:43 johnsom Yeah, this is what we were told to use to drop cert chains in via config drive years ago. The issue was user-data was not secure as it is stored in the DB and it had a really small max content size
23:40:43 rm_work which config file
23:40:48 dansmith because we're totally willy-nilly with that
23:41:09 dansmith johnsom: who told you that?
23:41:16 rm_work like, this? http://logs.openstack.org/49/552549/13/check/octavia-v2-dsvm-scenario/f7f027d/controller/logs/etc/nova/nova_conf.txt.gz
23:41:20 dansmith johnsom: this has been on its death bed for, gosh, five years at least
23:41:42 dansmith rm_work: that's on a controller
23:42:01 rm_work don't know where else to look?
23:42:03 johnsom The nova PTL at the time we were spinning this project up. I know I tried to use user-data last year and it was still pretty small. I couldn't load the test app I was hoping to drop into cirros
23:42:05 rm_work this is a single-node test
23:42:41 dansmith rm_work: okay, then probably the same, so yeah it's disabled there for actual file injection
23:42:46 johnsom We don't do anything special to enable that on devstack in the gates
23:42:48 rm_work then ... how is this working
23:42:51 dansmith I'll have to look and see if and how it ends up in the config drive
23:43:20 dansmith but, your assertion about the "it isn't stored is good for security" is a huge problem, because we can't migrate or rebuild or anything the instance
23:43:27 dansmith and don't have a way to provide that stuff again you know
23:43:28 dansmith and
23:43:43 dansmith storing anything sensitive in configdrive is kinda crazy :)
23:43:59 rm_work we don't WANT to migrate or restore the instances
23:44:02 rm_work that would explicitly be bad
23:44:09 rm_work if an instance dies, it's dead and we trash it
23:44:13 rm_work and we will handle making a replacement
23:44:25 johnsom Well some have use cases for migrate, but the content should migrate with the config drive
23:45:05 dansmith rm_work: okay but you understand that nova is a thing that lots of people use, and most want to be able to migrate, and if we break that model for users that don't know anything then, that's like bad and stuff yeah? :)
23:45:08 dansmith I mean,
23:45:15 dansmith nova has a wider audience :)
23:45:32 dansmith if you want to enforce different lifecycle semantics on the thing you use nova for, then that's cool, but..
23:45:36 johnsom Yeah, we have use cases for migrating. I think that is one operator talking
23:45:43 rm_work yes, but BECAUSE of that philosophy, because something doesn't work for a couple of use cases, you shouldn't remove it when it still works for others?
23:45:55 rm_work seems like it should work both ways :P
23:46:15 johnsom Is the supported data size for "user-data" documented? I can't find it
23:46:29 rm_work isn't saying "we should kill your use-case, because nova has to be generic for lots of use-cases" seems kinda backward
23:46:32 johnsom beyond "a small opaque blob of data"
23:46:51 dansmith here's where it shows up in the config drive: https://github.com/openstack/nova/blob/master/nova/api/metadata/base.py#L191-L194
23:48:16 dansmith rm_work: naw dude, but you know, we try to make features that are rather predictable and understandable from the outside, especially by the users that don't have a view into the config of the deployment
23:48:26 dansmith rm_work: it's kindof what we do here. we're trying to be an abstraction
23:48:49 dansmith rm_work: and, as we said, we have a method that *does* work for all cases, except it has a size limitation for you, which I get
23:49:10 dansmith and the other thing had some really terrible security semantics that I guess you didn't know about :)
23:49:15 melwitt johnsom: looks like this is the size limit 16384 bytes https://help.ubuntu.com/community/CloudInit#fnref-6a17aff5cf9eab5a99b5ab3dbd49854b0e9681c1
23:49:43 dansmith johnsom: it's a MediumText in the db
23:49:48 dansmith so that's likely the real limit
23:49:56 melwitt oh, okay
23:50:03 rm_work what are the terrible security semantics for the other thing? it mentions "evil images could do whatever with the data", but if we only boot with a known good image...
23:50:08 johnsom lol, yeah, 16k seems about what I ran into last year.
23:50:09 openstackgerrit Merged openstack/nova master: Stop using HostAPI.service_delete https://review.openstack.org/582680
23:50:27 dansmith api limits it at 64k, FWIW
23:50:38 johnsom Yeah, I mean if you don't trust the images... the whole guest OS is at risk
23:50:39 dansmith https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/schemas/servers.py#L212-L216
23:51:53 rm_work so what is the new alternative? we have to create a cinder volume for every VM we boot, and pass that to the server create?
23:52:31 rm_work since user_data is obviously unacceptable for multiple reasons
23:52:44 rm_work and I'm not seeing how that helps at all with the "evil image" case
23:52:50 dansmith the new alternative is user_data, and providing this feedback not on FF day at 5pm when we can actually do something for you :)
23:53:01 rm_work user_data is explicitly insecure
23:53:18 rm_work and size-limited to a point where it is not even possible to use for a very common use-case (passing in a cert chain)

Earlier   Later