| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-06-01 | |||
| 17:25:13 | openstackgerrit | Merged openstack/nova master: libvirt: place emulator threads on CONF.compute.cpu_shared_set https://review.openstack.org/510897 | |
| 17:33:44 | cfriesen | mnaser: for what it's worth I've played with kingbird a bit and it's somewhat useful but lacking robustness and functionality. (It only supports specific quotas for example.) | |
| 17:43:53 | fried_rice | superdan, mnaser, SpamapS, jgwentworth: I think I'm providing a way to do it live without the operator/admin having to do anything or even know it's happening. | |
| 17:44:43 | superdan | fried_rice: yeah I think I acknowledged that you may be in my reply, but -EFRIDAY on processing it | |
| 17:45:17 | fried_rice | superdan: ack | |
| 17:45:53 | jgwentworth | kewl | |
| 17:49:54 | SpamapS | was hoping to discuss https://review.openstack.org/#/c/568953/ and https://bugs.launchpad.net/nova/+bug/1742102 today | |
| 17:49:55 | openstack | Launchpad bug 1742102 in OpenStack Compute (nova) "Simple user can disable compute" [High,In progress] - Assigned to Matt Riedemann (mriedem) | |
| 17:50:11 | SpamapS | but I don't see a mriedem so maybe will have to wait until next week | |
| 17:50:24 | SpamapS | I'm a little concerned that it may be very easy to DoS clouds that are set up to "pack" instead of "spread". | |
| 17:50:53 | superdan | SpamapS: so turn it off | |
| 17:51:29 | superdan | SpamapS: mriedem is hans_lunch today, btw | |
| 17:52:16 | SpamapS | superdan: yes I"m suggesting that telling people to turn it off should be a CVE | |
| 17:52:22 | jgwentworth | default is "pack", so this is a problem out-of-the-box | |
| 17:52:41 | SpamapS | since the default is to have it turned on | |
| 17:52:55 | superdan | calling it a CVE is way overblown, IMHO | |
| 17:53:11 | SpamapS | and one basically just has to get nova to try and send a bunch of broken image+flavor reqs to a single compute node to disable it... and then keep doing that until they're all disabled. | |
| 17:53:38 | SpamapS | If a regular user can disable your compute nodes, that rises to CVE IMO. | |
| 17:53:55 | jgwentworth | mnaser: I think you hit this too, right? ^ | |
| 17:54:05 | SpamapS | There are a lot of examples of advisory CVE's where certain configurations are vulnerable, and no code fix is available because it requires heavy refactoring. | |
| 17:54:20 | SpamapS | I don't really want to write an exploit for this | |
| 17:54:40 | SpamapS | but if you guys want to suggest it's not feasible.. we can go down that road, and maybe disprove it is feasible and forget about a CVE. | |
| 17:55:13 | SpamapS | But we already had our stage cloud get all of its compute nodes disabled because of a bad image. | |
| 17:55:26 | mnaser | yes i ran into this too | |
| 17:55:29 | jgwentworth | fwiw, I think it's feasible and I think it's a serious problem. I'm just not so experienced with CVEs | |
| 17:55:36 | mnaser | um one second | |
| 17:56:01 | superdan | SpamapS: that feature was specifically requested by a bunch of ops in Boston, you know | |
| 17:56:13 | SpamapS | Feature is great! Implementation, not so much. | |
| 17:56:29 | superdan | um | |
| 17:56:31 | mnaser | sometimes it would be like | |
| 17:56:47 | mnaser | nova-compute tries to create volume, user hit their quota, volume create fails, that gets labeled as a failed deploy | |
| 17:57:01 | mnaser | do enough of that and you'll start disabling everything, for us, we kinda just disabled that for now | |
| 17:57:03 | SpamapS | Though IMO it should have included a back-off re-enabler too since, presumably, a disabled compute node may recover on its own and be able to serve traffic again. It's worth it to retry nodes that were in bad shape before. | |
| 17:57:46 | superdan | mnaser: right, but that's just because volume create shouldn't be included in the list of disable-able things | |
| 17:57:56 | superdan | it was intended to only be things that were obviously fatally broken | |
| 17:58:05 | SpamapS | mnaser: yeah we're disabling the feature, and likely won't re-enable it until it also re-enables compute nodes automatically. But I figure there are likely Nova users out there that have it enabled, and are vulnerable to a malicious or even just poorly-configured user disabling all their compute nodes. | |
| 17:58:08 | superdan | SpamapS: well, we could do that, but when I brought it up in the room, nobody wanted it to re-enable | |
| 17:58:17 | mnaser | superdan: agreed, it includes things like ports, i think i worked partially on this but i forgot what progress i had :( | |
| 17:59:27 | superdan | for what it's worth, | |
| 17:59:36 | jgwentworth | mnaser: yeah. I think we saw that it would quickly become whack-a-mole, so we didn't have a straightforward way to solve it. trying to whitelist a bunch of things is a mess | |
| 17:59:48 | superdan | anything that accidentally falls into the disable bucket are also things that generate retries, | |
| 18:00:02 | superdan | so a user that can abuse that can also generate a ton of extra churn in the system, being DoSish on its own | |
| 18:00:11 | superdan | should we CVE for having max_attempts>1? | |
| 18:00:21 | SpamapS | Anyway, there are two things I'd like to see happen and I'm happy to drive either or both. (1) Fix it so that it only increments on *specific* faults that are permanent failures on the compute node, instead of just a whitelist for exceptions to ignore. And (2) inform the user community of the danger they may be in. | |
| 18:00:43 | superdan | SpamapS: it already only increments for specific things | |
| 18:00:44 | superdan | SpamapS: it's just that set needs tweaking | |
| 18:00:57 | SpamapS | did you see the list harlowja made? | |
| 18:01:01 | mnaser | superdan:, SpamapS: http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2018-04-12.log.html | |
| 18:01:03 | mnaser | brief discussion there | |
| 18:01:07 | SpamapS | Those don't seem well thought out. | |
| 18:01:12 | SpamapS | And aren't explicitly listed. | |
| 18:01:23 | SpamapS | mnaser: indeed, I found that yesterday. | |
| 18:01:39 | superdan | well, part of the problem is that we convert stupid exceptions to stupid build results | |
| 18:01:48 | superdan | and it operates on the latter | |
| 18:02:01 | SpamapS | https://bugs.launchpad.net/nova/+bug/1774527 | |
| 18:02:02 | openstack | Launchpad bug 1742102 in OpenStack Compute (nova) "duplicate for #1774527 Simple user can disable compute" [High,In progress] - Assigned to Matt Riedemann (mriedem) | |
| 18:02:03 | SpamapS | has Josh's list | |
| 18:02:15 | mnaser | and then previous discussion here too http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2018-01-23.log.html#t2018-01-23T19:36:41 | |
| 18:02:30 | mnaser | but i have to get back to doing some $work stuff but yeah, we disabled it on ourside | |
| 18:02:39 | SpamapS | superdan: perhaps we shouldn't disable on stupid things? | |
| 18:02:48 | superdan | SpamapS: I think I've said I agree with that statement :) | |
| 18:02:57 | SpamapS | indeed | |
| 18:03:01 | superdan | how come you guys think this is CVEish and just filed a non-security bug for it? | |
| 18:03:14 | superdan | that kinda eliminates the supposed desire to make a big deal over it :) | |
| 18:03:30 | jgwentworth | superdan: they filed it as security but it's already known | |
| 18:03:48 | jgwentworth | so I'm not sure how that works. | |
| 18:03:55 | superdan | it's public, so not a security bug | |
| 18:04:03 | SpamapS | and agreed that if you can cause a bunch of retries you are already causing the cloud some slowness. But the compute nodes disable somewhat silently, and permanently, creating a far worse situation than just "it's slow now" | |
| 18:04:20 | jgwentworth | well, IMHO it should have been a security bug from the start, but the original bug reporter reported it as public | |
| 18:04:28 | SpamapS | We didn't actually think it was a security bug, just reporting "hey this broke our cloud" | |
| 18:04:38 | SpamapS | and then we debugged more and were like "zomg security" | |
| 18:04:43 | SpamapS | and then we found out there was already public knowledge | |
| 18:05:03 | superdan | SpamapS: you're going to DoS the first three nodes in your pack scenario | |
| 18:05:12 | superdan | if you're just hitting those | |
| 18:05:21 | superdan | so, anyway, | |
| 18:05:21 | mnaser | except if you boot with --num-instances 100 | |
| 18:05:39 | superdan | mnaser: well, true, but you're going for DoS so you don't do that if you're an attacker, right? | |
| 18:05:58 | superdan | anyway, I'd really like to fix the thing and not rip it out, | |
| 18:06:08 | superdan | but if people want to disable it by default for now, that's cool | |
| 18:06:12 | superdan | although I think that means we'll never fix it | |
| 18:06:17 | mnaser | maybe i should hush but nova doesn't even let us control the # to provide in --num-instances (and i'm agreeing that we clean it up rather than remove it) | |
| 18:06:30 | mnaser | so if you have enough quota you can --num-instances one-zillion | |
| 18:06:34 | SpamapS | I think the right fix is to pick *specific* build results that have high or absolute confidence are the fault of the compute node. | |
| 18:06:41 | superdan | mnaser: OMG CVE! :) | |
| 18:06:46 | mnaser | shhh its friday | |
| 18:06:47 | mnaser | we don't want that | |
| 18:06:58 | superdan | SpamapS: we do pick specific build results, that's the thing | |
| 18:07:09 | superdan | SpamapS: we convert _exceptions_ to build results in not the right way | |
| 18:07:18 | mnaser | anything that results in a rescheduled build will trigger that counter to increase i think | |
| 18:07:26 | SpamapS | How does an image too big for flavor failure land in that bucket then? | |
| 18:07:45 | SpamapS | Or a "failure to download image" | |
| 18:07:45 | mnaser | i think that's a bug that nova considers that a reschedule-able failure | |
| 18:08:04 | SpamapS | yeah see re-schedulable doesn't mean it's the compute node's fault. | |
| 18:08:10 | jgwentworth | I dunno, I do think it's worth an advisory to call this out because if any ol user could disable a lot of compute nodes by just making normal requests, it's a major problem | |
| 18:08:13 | mnaser | failure to download image could be that specific compute node cannot talk to glance | |
| 18:08:25 | mnaser | but it's a tricky weird thing to balance | |
| 18:08:27 | SpamapS | could be | |
| 18:08:30 | SpamapS | but you don't *know* | |
| 18:08:34 | mnaser | but if your glance is down then | |
| 18:08:44 | mnaser | all your compute nodes are going disabled | |