Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-07-26
22:25:43 dansmith and before this, setting it would do nothing
22:25:50 dansmith people do read and try stuff that are in specs
22:26:05 dansmith so I think we should just patch it out of the driver if we're not going to mention it personally (assuming it's broken like we expect)
22:26:07 dansmith just MHO
22:26:37 dansmith if we start claiming in stein that it works, people will try it in older releases too
22:30:45 melwitt mriedem_afk: fyi nova r-3 release patch https://review.openstack.org/586368
22:31:38 dansmith first hit: https://docs.openstack.org/nova/latest/reference/update-provider-tree.html
22:33:39 melwitt ok ... thinking. the change was done as a bug fix, https://bugs.launchpad.net/nova/+bug/1771728 which was found via someone setting MISC_SHARES_VIA_AGGREGATE
22:33:39 openstack Launchpad bug 1771728 in OpenStack Compute (nova) "libvirt: Shared Resource Provider (RP) DISK_GB is NOT taken into account if it's configured with Compute Node RPs" [Undecided,Fix released] - Assigned to Bhagyashri Shewale (bhagyashri-shewale)
22:35:02 melwitt if we patch the bug fix out, then it goes back to setting it does nothing. but that still seems weird because people think they can use it (because of this doc, possibly)
22:36:53 dansmith yeah, so someone thought they could use it and reported it did nothing,
22:36:59 melwitt I dunno, maybe just a doc update to this page saying MISC_SHARES_VIA_AGGREGATE isn't tested in tempest?
22:37:09 melwitt like a warning?
22:37:17 dansmith but if that's the case and it actually breaks resource reporting and probably migrations...
22:37:48 dansmith if it totally doesn't do anything other than break things we should probably not do it until the rest of the feature works
22:37:55 dansmith but at least a known issue reno, IMHO
22:38:18 dansmith not tested in tempest is different than "totally broken and we know it"
22:38:20 melwitt what I mean is, was the previous state of it doing nothing actually doing nothing? or could it have been breaking things underneath?
22:38:49 dansmith no, it was just being ignored when people expected it to magically make the compute node do the shared storage accounting right
22:39:12 dansmith now, setting it will cause the compute node to not expose disk inventory,
22:39:31 dansmith and likely in a migration situation we will clobber the allocation against the shared pool,
22:39:37 dansmith which will result in instances with no disk allocation,
22:40:02 melwitt okay ... so maybe, revert the bug fix, add a warning to the doc to mention MISC_SHARES_VIA_AGGREGATE isn't usable in nova yet, and then don't allow the bug fix back in until we have tempest setup with ceph etc?
22:40:05 dansmith which we'll have to fix via migration somehow, and potentially result in oversubscribing the pool since anything that has migrated once will have no disk allocation
22:40:28 dansmith that's certainly one approach yeah
22:40:44 dansmith the other being just a reno that warns that it's half implemented and known broken
22:41:20 melwitt I'm worried people will miss that reno, I guess
22:41:35 dansmith yup, and they'll corrupt their data if they do and try it
22:41:39 melwitt and the price for missing it is high
22:42:16 melwitt efried: are you around? what do you think about ^
22:48:01 mriedem rather than revert, you could just comment out the one line in the libvirt driver
22:48:12 dansmith yeah, revert is too old I'm sure
22:48:20 dansmith just have to mangle it manually
22:50:19 melwitt yeah, okay. I think we should because from what dansmith has explained, using this in its current state could badly corrupt a deployment and then we'd have to come up with some code to migrate people out of it if it happens to them
22:50:40 dansmith it'd be heal_allocations I guess
22:50:43 dansmith but still
22:54:49 dansmith might be able to functional test our way
22:55:30 melwitt I'll put this on the rc1 todo list https://etherpad.openstack.org/p/nova-rocky-release-candidate-todo
22:55:41 melwitt and get efried's thoughts tomorrow morning
23:22:34 rm_work hey nova folks! the deprecation discussed here: https://review.openstack.org/#/c/522027/ / https://specs.openstack.org/openstack/nova-specs/specs/queens/implemented/deprecate-file-injection.html -- is that what we're using here? https://github.com/openstack/octavia/blob/master/octavia/compute/drivers/nova_driver.py#L150
23:26:04 melwitt rm_work: yes, that is file injection, deprecated with microversion 2.57 https://github.com/openstack/python-novaclient/blob/master/novaclient/v2/servers.py#L1210-L1215
23:26:29 rm_work johnsom: ^^
23:27:08 johnsom melwitt What? You removed the ability to load data via config drive? That will break a ton of services.
23:27:41 melwitt johnsom: no, the ability to specify files to inject into the server. the supported way to do that is user data
23:28:33 johnsom melwitt but user_data was supposed to go away given it's non-standard and significantly limited in size.
23:29:21 melwitt I think you might be reversing the two? user_data isn't going away and is supported by cloud-init
23:29:24 johnsom Wow, this went under the radar. I know it impacts Octavia and Trove, probably manila and a few others
23:30:11 johnsom The file method we were using is supported by cloud-init via config drive.
23:31:31 johnsom What we are talking about is in the "content" folder in config drive
23:31:38 rm_work it's interesting because in https://specs.openstack.org/openstack/nova-specs/specs/queens/implemented/deprecate-file-injection.html#problem-description
23:32:11 rm_work #1 is not a problem for services that use fixed images, #2 is a GOOD thing (for security), and #3 is simply false, because the alternatives are worse (bad security)
23:33:22 johnsom Yeah user-data has a ridiculously small size limitation too. Like you can't even get a full certificate chain in it.
23:35:13 melwitt okay, I don't know that much detail about the deprecation. so it'd be best to visit the channel tomorrow when people who know more about it are around
23:35:32 melwitt as for the use case, the only other way I know of to handle something like that would be vendor data
23:37:14 dansmith rm_work: johnsom just to be clear, you're talking about putting stuff in the "personality" part of the server post right?
23:37:27 johnsom no
23:37:30 rm_work https://github.com/openstack/python-novaclient/blob/master/novaclient/v2/servers.py#L1298-L1300
23:37:40 rm_work https://github.com/openstack/octavia/blob/master/octavia/compute/drivers/nova_driver.py#L150
23:37:46 dansmith johnsom: cool, then this isn't what you're looking for
23:37:51 rm_work ^^ dansmith see that
23:37:55 rm_work I think the answer is yes?
23:38:07 rm_work we pass "files" to the server create
23:38:35 dansmith yeah, I think files in the client goes to personality
23:38:39 rm_work https://github.com/openstack/python-novaclient/blob/master/novaclient/v2/servers.py#L688-L692
23:38:40 johnsom Does "files" in the client get converted to personalities in the nova API?
23:38:42 rm_work yes
23:38:53 dansmith so this has been disabled by default for libvirt for a long time,
23:39:03 dansmith and strongly encouraged not to be used for longer
23:39:14 dansmith I guess you have been turning it on in the compute node?
23:39:17 rm_work we even say as much here: https://github.com/openstack/octavia/blob/920a15c620eada38f40d827c89fb9881ffd29fdb/octavia/common/config.py#L368-L370
23:39:32 rm_work dansmith: i don't think we have done anything to force it to be enabled in our gates...
23:39:42 dansmith rm_work: it's off by default
23:39:49 dansmith in the libvirt driver
23:39:53 rm_work could we tell from nova logs from a gate run?
23:39:56 dansmith not sure if the other drivers even support it
23:40:12 dansmith rm_work: just look at the config files from the gate run, like inject_file_partition or whatever
23:40:41 dansmith I didn't think that goes into the config drive, fwiw.. and that would be bad for security if it did and you use it for sensitive things,
23:40:43 johnsom Yeah, this is what we were told to use to drop cert chains in via config drive years ago. The issue was user-data was not secure as it is stored in the DB and it had a really small max content size
23:40:43 rm_work which config file
23:40:48 dansmith because we're totally willy-nilly with that
23:41:09 dansmith johnsom: who told you that?
23:41:16 rm_work like, this? http://logs.openstack.org/49/552549/13/check/octavia-v2-dsvm-scenario/f7f027d/controller/logs/etc/nova/nova_conf.txt.gz
23:41:20 dansmith johnsom: this has been on its death bed for, gosh, five years at least
23:41:42 dansmith rm_work: that's on a controller
23:42:01 rm_work don't know where else to look?
23:42:03 johnsom The nova PTL at the time we were spinning this project up. I know I tried to use user-data last year and it was still pretty small. I couldn't load the test app I was hoping to drop into cirros
23:42:05 rm_work this is a single-node test
23:42:41 dansmith rm_work: okay, then probably the same, so yeah it's disabled there for actual file injection
23:42:46 johnsom We don't do anything special to enable that on devstack in the gates
23:42:48 rm_work then ... how is this working
23:42:51 dansmith I'll have to look and see if and how it ends up in the config drive
23:43:20 dansmith but, your assertion about the "it isn't stored is good for security" is a huge problem, because we can't migrate or rebuild or anything the instance
23:43:27 dansmith and don't have a way to provide that stuff again you know
23:43:28 dansmith and
23:43:43 dansmith storing anything sensitive in configdrive is kinda crazy :)
23:43:59 rm_work we don't WANT to migrate or restore the instances
23:44:02 rm_work that would explicitly be bad
23:44:09 rm_work if an instance dies, it's dead and we trash it
23:44:13 rm_work and we will handle making a replacement
23:44:25 johnsom Well some have use cases for migrate, but the content should migrate with the config drive

Earlier   Later