| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-06-01 | |||
| 16:40:44 | jgwentworth | mxevgenis: what version of nova are you using? do you mean that you launch N instances at the same time and you expect some to go to the rest of the nodes once the first nodes are full? by default, nova will try to reschedule to another host when one is full https://docs.openstack.org/nova/latest/configuration/config.html#scheduler.max_attempts | |
| 16:43:03 | mxevgenis | i deployed the ocata version of openstack using the openstack-ansible project. I launched many instances not simultaneously until all of my resources (of the first two nodes) where reserved. The n+1 instance which should be launched on one of the rest nodes failed | |
| 16:43:13 | mxevgenis | with no valid host error | |
| 16:43:51 | mxevgenis | The wired thing is that in the compute nodes tab the hosts are displayed as active | |
| 16:45:02 | jgwentworth | and any subsequent separate attempt to boot an instance is resulting in NoValidHost even though the rest of the nods are available? in that case, like hans_lunch mentioned, something else is wrong and you need to take a look at your nova-scheduler and nova-compute logs to see why it's being rejected | |
| 16:45:11 | jgwentworth | *nodes | |
| 16:45:37 | mxevgenis | thanks a lot. i appreciate your help!!! | |
| 16:46:00 | jgwentworth | prior to pike, it was possible for parallel requests to race in a way that even with reschedules, some requests could fail with NoValidHost as nodes filled up. but it sounds like that's not what you're hitting | |
| 16:46:43 | mxevgenis | i should take a look on the logs and see what happens | |
| 16:47:35 | jgwentworth | yeah, agreed. good luck | |
| 16:47:45 | mxevgenis | the wired thing is that the nodes which are working fine are different in terms of hardware to the rest servers | |
| 16:48:44 | mxevgenis | thanks a lot for your help again | |
| 16:49:37 | jgwentworth | yeah, there might be some host aggregate metadata mismatch going on (which if so, you will see in the logs) if you have things set differently depending on the hardware config of the servers. so yeah, have to check the logs to find out what's happening | |
| 17:02:00 | openstackgerrit | Dan Smith proposed openstack/nova master: Use oslo.messaging per-call monitoring https://review.openstack.org/566696 | |
| 17:25:13 | openstackgerrit | Merged openstack/nova master: libvirt: place emulator threads on CONF.compute.cpu_shared_set https://review.openstack.org/510897 | |
| 17:33:44 | cfriesen | mnaser: for what it's worth I've played with kingbird a bit and it's somewhat useful but lacking robustness and functionality. (It only supports specific quotas for example.) | |
| 17:43:53 | fried_rice | superdan, mnaser, SpamapS, jgwentworth: I think I'm providing a way to do it live without the operator/admin having to do anything or even know it's happening. | |
| 17:44:43 | superdan | fried_rice: yeah I think I acknowledged that you may be in my reply, but -EFRIDAY on processing it | |
| 17:45:17 | fried_rice | superdan: ack | |
| 17:45:53 | jgwentworth | kewl | |
| 17:49:54 | SpamapS | was hoping to discuss https://review.openstack.org/#/c/568953/ and https://bugs.launchpad.net/nova/+bug/1742102 today | |
| 17:49:55 | openstack | Launchpad bug 1742102 in OpenStack Compute (nova) "Simple user can disable compute" [High,In progress] - Assigned to Matt Riedemann (mriedem) | |
| 17:50:11 | SpamapS | but I don't see a mriedem so maybe will have to wait until next week | |
| 17:50:24 | SpamapS | I'm a little concerned that it may be very easy to DoS clouds that are set up to "pack" instead of "spread". | |
| 17:50:53 | superdan | SpamapS: so turn it off | |
| 17:51:29 | superdan | SpamapS: mriedem is hans_lunch today, btw | |
| 17:52:16 | SpamapS | superdan: yes I"m suggesting that telling people to turn it off should be a CVE | |
| 17:52:22 | jgwentworth | default is "pack", so this is a problem out-of-the-box | |
| 17:52:41 | SpamapS | since the default is to have it turned on | |
| 17:52:55 | superdan | calling it a CVE is way overblown, IMHO | |
| 17:53:11 | SpamapS | and one basically just has to get nova to try and send a bunch of broken image+flavor reqs to a single compute node to disable it... and then keep doing that until they're all disabled. | |
| 17:53:38 | SpamapS | If a regular user can disable your compute nodes, that rises to CVE IMO. | |
| 17:53:55 | jgwentworth | mnaser: I think you hit this too, right? ^ | |
| 17:54:05 | SpamapS | There are a lot of examples of advisory CVE's where certain configurations are vulnerable, and no code fix is available because it requires heavy refactoring. | |
| 17:54:20 | SpamapS | I don't really want to write an exploit for this | |
| 17:54:40 | SpamapS | but if you guys want to suggest it's not feasible.. we can go down that road, and maybe disprove it is feasible and forget about a CVE. | |
| 17:55:13 | SpamapS | But we already had our stage cloud get all of its compute nodes disabled because of a bad image. | |
| 17:55:26 | mnaser | yes i ran into this too | |
| 17:55:29 | jgwentworth | fwiw, I think it's feasible and I think it's a serious problem. I'm just not so experienced with CVEs | |
| 17:55:36 | mnaser | um one second | |
| 17:56:01 | superdan | SpamapS: that feature was specifically requested by a bunch of ops in Boston, you know | |
| 17:56:13 | SpamapS | Feature is great! Implementation, not so much. | |
| 17:56:29 | superdan | um | |
| 17:56:31 | mnaser | sometimes it would be like | |
| 17:56:47 | mnaser | nova-compute tries to create volume, user hit their quota, volume create fails, that gets labeled as a failed deploy | |
| 17:57:01 | mnaser | do enough of that and you'll start disabling everything, for us, we kinda just disabled that for now | |
| 17:57:03 | SpamapS | Though IMO it should have included a back-off re-enabler too since, presumably, a disabled compute node may recover on its own and be able to serve traffic again. It's worth it to retry nodes that were in bad shape before. | |
| 17:57:46 | superdan | mnaser: right, but that's just because volume create shouldn't be included in the list of disable-able things | |
| 17:57:56 | superdan | it was intended to only be things that were obviously fatally broken | |
| 17:58:05 | SpamapS | mnaser: yeah we're disabling the feature, and likely won't re-enable it until it also re-enables compute nodes automatically. But I figure there are likely Nova users out there that have it enabled, and are vulnerable to a malicious or even just poorly-configured user disabling all their compute nodes. | |
| 17:58:08 | superdan | SpamapS: well, we could do that, but when I brought it up in the room, nobody wanted it to re-enable | |
| 17:58:17 | mnaser | superdan: agreed, it includes things like ports, i think i worked partially on this but i forgot what progress i had :( | |
| 17:59:27 | superdan | for what it's worth, | |
| 17:59:36 | jgwentworth | mnaser: yeah. I think we saw that it would quickly become whack-a-mole, so we didn't have a straightforward way to solve it. trying to whitelist a bunch of things is a mess | |
| 17:59:48 | superdan | anything that accidentally falls into the disable bucket are also things that generate retries, | |
| 18:00:02 | superdan | so a user that can abuse that can also generate a ton of extra churn in the system, being DoSish on its own | |
| 18:00:11 | superdan | should we CVE for having max_attempts>1? | |
| 18:00:21 | SpamapS | Anyway, there are two things I'd like to see happen and I'm happy to drive either or both. (1) Fix it so that it only increments on *specific* faults that are permanent failures on the compute node, instead of just a whitelist for exceptions to ignore. And (2) inform the user community of the danger they may be in. | |
| 18:00:43 | superdan | SpamapS: it already only increments for specific things | |
| 18:00:44 | superdan | SpamapS: it's just that set needs tweaking | |
| 18:00:57 | SpamapS | did you see the list harlowja made? | |
| 18:01:01 | mnaser | superdan:, SpamapS: http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2018-04-12.log.html | |
| 18:01:03 | mnaser | brief discussion there | |
| 18:01:07 | SpamapS | Those don't seem well thought out. | |
| 18:01:12 | SpamapS | And aren't explicitly listed. | |
| 18:01:23 | SpamapS | mnaser: indeed, I found that yesterday. | |
| 18:01:39 | superdan | well, part of the problem is that we convert stupid exceptions to stupid build results | |
| 18:01:48 | superdan | and it operates on the latter | |
| 18:02:01 | SpamapS | https://bugs.launchpad.net/nova/+bug/1774527 | |
| 18:02:02 | openstack | Launchpad bug 1742102 in OpenStack Compute (nova) "duplicate for #1774527 Simple user can disable compute" [High,In progress] - Assigned to Matt Riedemann (mriedem) | |
| 18:02:03 | SpamapS | has Josh's list | |
| 18:02:15 | mnaser | and then previous discussion here too http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2018-01-23.log.html#t2018-01-23T19:36:41 | |
| 18:02:30 | mnaser | but i have to get back to doing some $work stuff but yeah, we disabled it on ourside | |
| 18:02:39 | SpamapS | superdan: perhaps we shouldn't disable on stupid things? | |
| 18:02:48 | superdan | SpamapS: I think I've said I agree with that statement :) | |
| 18:02:57 | SpamapS | indeed | |
| 18:03:01 | superdan | how come you guys think this is CVEish and just filed a non-security bug for it? | |
| 18:03:14 | superdan | that kinda eliminates the supposed desire to make a big deal over it :) | |
| 18:03:30 | jgwentworth | superdan: they filed it as security but it's already known | |
| 18:03:48 | jgwentworth | so I'm not sure how that works. | |
| 18:03:55 | superdan | it's public, so not a security bug | |
| 18:04:03 | SpamapS | and agreed that if you can cause a bunch of retries you are already causing the cloud some slowness. But the compute nodes disable somewhat silently, and permanently, creating a far worse situation than just "it's slow now" | |
| 18:04:20 | jgwentworth | well, IMHO it should have been a security bug from the start, but the original bug reporter reported it as public | |
| 18:04:28 | SpamapS | We didn't actually think it was a security bug, just reporting "hey this broke our cloud" | |
| 18:04:38 | SpamapS | and then we debugged more and were like "zomg security" | |
| 18:04:43 | SpamapS | and then we found out there was already public knowledge | |
| 18:05:03 | superdan | SpamapS: you're going to DoS the first three nodes in your pack scenario | |
| 18:05:12 | superdan | if you're just hitting those | |
| 18:05:21 | mnaser | except if you boot with --num-instances 100 | |
| 18:05:21 | superdan | so, anyway, | |
| 18:05:39 | superdan | mnaser: well, true, but you're going for DoS so you don't do that if you're an attacker, right? | |
| 18:05:58 | superdan | anyway, I'd really like to fix the thing and not rip it out, | |
| 18:06:08 | superdan | but if people want to disable it by default for now, that's cool | |
| 18:06:12 | superdan | although I think that means we'll never fix it | |
| 18:06:17 | mnaser | maybe i should hush but nova doesn't even let us control the # to provide in --num-instances (and i'm agreeing that we clean it up rather than remove it) | |
| 18:06:30 | mnaser | so if you have enough quota you can --num-instances one-zillion | |
| 18:06:34 | SpamapS | I think the right fix is to pick *specific* build results that have high or absolute confidence are the fault of the compute node. | |
| 18:06:41 | superdan | mnaser: OMG CVE! :) | |
| 18:06:46 | mnaser | shhh its friday | |
| 18:06:47 | mnaser | we don't want that | |