| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-07-26 | |||
| 22:13:01 | openstackgerrit | OpenStack Release Bot proposed openstack/python-novaclient stable/rocky: Update .gitreview for stable/rocky https://review.openstack.org/586364 | |
| 22:13:03 | openstackgerrit | OpenStack Release Bot proposed openstack/python-novaclient stable/rocky: Update UPPER_CONSTRAINTS_FILE for stable/rocky https://review.openstack.org/586365 | |
| 22:13:05 | openstackgerrit | OpenStack Release Bot proposed openstack/python-novaclient master: Update reno for stable/rocky https://review.openstack.org/586366 | |
| 22:13:39 | melwitt | mriedem: oh, coincidence, ok. but you reminded me, we need your test to determine whether or not to reno a warning not to use the shared aggregate thing. does that have to be done before my EOD or just before RC1? | |
| 22:14:16 | mriedem | we don't have to warn about that in a reno | |
| 22:14:20 | mriedem | it's totally undocumented | |
| 22:14:25 | mriedem | we don't need to say "yup this still doesn't work" | |
| 22:14:51 | melwitt | ok, thanks | |
| 22:25:37 | dansmith | we have it in specs, | |
| 22:25:43 | dansmith | and before this, setting it would do nothing | |
| 22:25:50 | dansmith | people do read and try stuff that are in specs | |
| 22:26:05 | dansmith | so I think we should just patch it out of the driver if we're not going to mention it personally (assuming it's broken like we expect) | |
| 22:26:07 | dansmith | just MHO | |
| 22:26:37 | dansmith | if we start claiming in stein that it works, people will try it in older releases too | |
| 22:30:45 | melwitt | mriedem_afk: fyi nova r-3 release patch https://review.openstack.org/586368 | |
| 22:31:38 | dansmith | first hit: https://docs.openstack.org/nova/latest/reference/update-provider-tree.html | |
| 22:33:39 | melwitt | ok ... thinking. the change was done as a bug fix, https://bugs.launchpad.net/nova/+bug/1771728 which was found via someone setting MISC_SHARES_VIA_AGGREGATE | |
| 22:33:39 | openstack | Launchpad bug 1771728 in OpenStack Compute (nova) "libvirt: Shared Resource Provider (RP) DISK_GB is NOT taken into account if it's configured with Compute Node RPs" [Undecided,Fix released] - Assigned to Bhagyashri Shewale (bhagyashri-shewale) | |
| 22:35:02 | melwitt | if we patch the bug fix out, then it goes back to setting it does nothing. but that still seems weird because people think they can use it (because of this doc, possibly) | |
| 22:36:53 | dansmith | yeah, so someone thought they could use it and reported it did nothing, | |
| 22:36:59 | melwitt | I dunno, maybe just a doc update to this page saying MISC_SHARES_VIA_AGGREGATE isn't tested in tempest? | |
| 22:37:09 | melwitt | like a warning? | |
| 22:37:17 | dansmith | but if that's the case and it actually breaks resource reporting and probably migrations... | |
| 22:37:48 | dansmith | if it totally doesn't do anything other than break things we should probably not do it until the rest of the feature works | |
| 22:37:55 | dansmith | but at least a known issue reno, IMHO | |
| 22:38:18 | dansmith | not tested in tempest is different than "totally broken and we know it" | |
| 22:38:20 | melwitt | what I mean is, was the previous state of it doing nothing actually doing nothing? or could it have been breaking things underneath? | |
| 22:38:49 | dansmith | no, it was just being ignored when people expected it to magically make the compute node do the shared storage accounting right | |
| 22:39:12 | dansmith | now, setting it will cause the compute node to not expose disk inventory, | |
| 22:39:31 | dansmith | and likely in a migration situation we will clobber the allocation against the shared pool, | |
| 22:39:37 | dansmith | which will result in instances with no disk allocation, | |
| 22:40:02 | melwitt | okay ... so maybe, revert the bug fix, add a warning to the doc to mention MISC_SHARES_VIA_AGGREGATE isn't usable in nova yet, and then don't allow the bug fix back in until we have tempest setup with ceph etc? | |
| 22:40:05 | dansmith | which we'll have to fix via migration somehow, and potentially result in oversubscribing the pool since anything that has migrated once will have no disk allocation | |
| 22:40:28 | dansmith | that's certainly one approach yeah | |
| 22:40:44 | dansmith | the other being just a reno that warns that it's half implemented and known broken | |
| 22:41:20 | melwitt | I'm worried people will miss that reno, I guess | |
| 22:41:35 | dansmith | yup, and they'll corrupt their data if they do and try it | |
| 22:41:39 | melwitt | and the price for missing it is high | |
| 22:42:16 | melwitt | efried: are you around? what do you think about ^ | |
| 22:48:01 | mriedem | rather than revert, you could just comment out the one line in the libvirt driver | |
| 22:48:12 | dansmith | yeah, revert is too old I'm sure | |
| 22:48:20 | dansmith | just have to mangle it manually | |
| 22:50:19 | melwitt | yeah, okay. I think we should because from what dansmith has explained, using this in its current state could badly corrupt a deployment and then we'd have to come up with some code to migrate people out of it if it happens to them | |
| 22:50:40 | dansmith | it'd be heal_allocations I guess | |
| 22:50:43 | dansmith | but still | |
| 22:54:49 | dansmith | might be able to functional test our way | |
| 22:55:30 | melwitt | I'll put this on the rc1 todo list https://etherpad.openstack.org/p/nova-rocky-release-candidate-todo | |
| 22:55:41 | melwitt | and get efried's thoughts tomorrow morning | |
| 23:22:34 | rm_work | hey nova folks! the deprecation discussed here: https://review.openstack.org/#/c/522027/ / https://specs.openstack.org/openstack/nova-specs/specs/queens/implemented/deprecate-file-injection.html -- is that what we're using here? https://github.com/openstack/octavia/blob/master/octavia/compute/drivers/nova_driver.py#L150 | |
| 23:26:04 | melwitt | rm_work: yes, that is file injection, deprecated with microversion 2.57 https://github.com/openstack/python-novaclient/blob/master/novaclient/v2/servers.py#L1210-L1215 | |
| 23:26:29 | rm_work | johnsom: ^^ | |
| 23:27:08 | johnsom | melwitt What? You removed the ability to load data via config drive? That will break a ton of services. | |
| 23:27:41 | melwitt | johnsom: no, the ability to specify files to inject into the server. the supported way to do that is user data | |
| 23:28:33 | johnsom | melwitt but user_data was supposed to go away given it's non-standard and significantly limited in size. | |
| 23:29:21 | melwitt | I think you might be reversing the two? user_data isn't going away and is supported by cloud-init | |
| 23:29:24 | johnsom | Wow, this went under the radar. I know it impacts Octavia and Trove, probably manila and a few others | |
| 23:30:11 | johnsom | The file method we were using is supported by cloud-init via config drive. | |
| 23:31:31 | johnsom | What we are talking about is in the "content" folder in config drive | |
| 23:31:38 | rm_work | it's interesting because in https://specs.openstack.org/openstack/nova-specs/specs/queens/implemented/deprecate-file-injection.html#problem-description | |
| 23:32:11 | rm_work | #1 is not a problem for services that use fixed images, #2 is a GOOD thing (for security), and #3 is simply false, because the alternatives are worse (bad security) | |
| 23:33:22 | johnsom | Yeah user-data has a ridiculously small size limitation too. Like you can't even get a full certificate chain in it. | |
| 23:35:13 | melwitt | okay, I don't know that much detail about the deprecation. so it'd be best to visit the channel tomorrow when people who know more about it are around | |
| 23:35:32 | melwitt | as for the use case, the only other way I know of to handle something like that would be vendor data | |
| 23:37:14 | dansmith | rm_work: johnsom just to be clear, you're talking about putting stuff in the "personality" part of the server post right? | |
| 23:37:27 | johnsom | no | |
| 23:37:30 | rm_work | https://github.com/openstack/python-novaclient/blob/master/novaclient/v2/servers.py#L1298-L1300 | |
| 23:37:40 | rm_work | https://github.com/openstack/octavia/blob/master/octavia/compute/drivers/nova_driver.py#L150 | |
| 23:37:46 | dansmith | johnsom: cool, then this isn't what you're looking for | |
| 23:37:51 | rm_work | ^^ dansmith see that | |
| 23:37:55 | rm_work | I think the answer is yes? | |
| 23:38:07 | rm_work | we pass "files" to the server create | |
| 23:38:35 | dansmith | yeah, I think files in the client goes to personality | |
| 23:38:39 | rm_work | https://github.com/openstack/python-novaclient/blob/master/novaclient/v2/servers.py#L688-L692 | |
| 23:38:40 | johnsom | Does "files" in the client get converted to personalities in the nova API? | |
| 23:38:42 | rm_work | yes | |
| 23:38:53 | dansmith | so this has been disabled by default for libvirt for a long time, | |
| 23:39:03 | dansmith | and strongly encouraged not to be used for longer | |
| 23:39:14 | dansmith | I guess you have been turning it on in the compute node? | |
| 23:39:17 | rm_work | we even say as much here: https://github.com/openstack/octavia/blob/920a15c620eada38f40d827c89fb9881ffd29fdb/octavia/common/config.py#L368-L370 | |
| 23:39:32 | rm_work | dansmith: i don't think we have done anything to force it to be enabled in our gates... | |
| 23:39:42 | dansmith | rm_work: it's off by default | |
| 23:39:49 | dansmith | in the libvirt driver | |
| 23:39:53 | rm_work | could we tell from nova logs from a gate run? | |
| 23:39:56 | dansmith | not sure if the other drivers even support it | |
| 23:40:12 | dansmith | rm_work: just look at the config files from the gate run, like inject_file_partition or whatever | |
| 23:40:41 | dansmith | I didn't think that goes into the config drive, fwiw.. and that would be bad for security if it did and you use it for sensitive things, | |
| 23:40:43 | johnsom | Yeah, this is what we were told to use to drop cert chains in via config drive years ago. The issue was user-data was not secure as it is stored in the DB and it had a really small max content size | |
| 23:40:43 | rm_work | which config file | |
| 23:40:48 | dansmith | because we're totally willy-nilly with that | |
| 23:41:09 | dansmith | johnsom: who told you that? | |
| 23:41:16 | rm_work | like, this? http://logs.openstack.org/49/552549/13/check/octavia-v2-dsvm-scenario/f7f027d/controller/logs/etc/nova/nova_conf.txt.gz | |
| 23:41:20 | dansmith | johnsom: this has been on its death bed for, gosh, five years at least | |
| 23:41:42 | dansmith | rm_work: that's on a controller | |
| 23:42:01 | rm_work | don't know where else to look? | |
| 23:42:03 | johnsom | The nova PTL at the time we were spinning this project up. I know I tried to use user-data last year and it was still pretty small. I couldn't load the test app I was hoping to drop into cirros | |
| 23:42:05 | rm_work | this is a single-node test | |
| 23:42:41 | dansmith | rm_work: okay, then probably the same, so yeah it's disabled there for actual file injection | |
| 23:42:46 | johnsom | We don't do anything special to enable that on devstack in the gates | |
| 23:42:48 | rm_work | then ... how is this working | |
| 23:42:51 | dansmith | I'll have to look and see if and how it ends up in the config drive | |