| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-06-01 | |||
| 18:06:46 | mnaser | shhh its friday | |
| 18:06:47 | mnaser | we don't want that | |
| 18:06:58 | superdan | SpamapS: we do pick specific build results, that's the thing | |
| 18:07:09 | superdan | SpamapS: we convert _exceptions_ to build results in not the right way | |
| 18:07:18 | mnaser | anything that results in a rescheduled build will trigger that counter to increase i think | |
| 18:07:26 | SpamapS | How does an image too big for flavor failure land in that bucket then? | |
| 18:07:45 | mnaser | i think that's a bug that nova considers that a reschedule-able failure | |
| 18:07:45 | SpamapS | Or a "failure to download image" | |
| 18:08:04 | SpamapS | yeah see re-schedulable doesn't mean it's the compute node's fault. | |
| 18:08:10 | jgwentworth | I dunno, I do think it's worth an advisory to call this out because if any ol user could disable a lot of compute nodes by just making normal requests, it's a major problem | |
| 18:08:13 | mnaser | failure to download image could be that specific compute node cannot talk to glance | |
| 18:08:25 | mnaser | but it's a tricky weird thing to balance | |
| 18:08:27 | SpamapS | could be | |
| 18:08:30 | SpamapS | but you don't *know* | |
| 18:08:34 | mnaser | but if your glance is down then | |
| 18:08:44 | mnaser | all your compute nodes are going disabled | |
| 18:08:52 | jrollinhatin | SpamapS: looks like two members from the VMT are already aware of this bug and triaging how we notify about it; if you're looking to have it tagged with a CVE perhaps engage them instead (since they're the people that do that thing)? | |
| 18:09:07 | SpamapS | yeah if glance is down or just partitioned from 1/3 of your cloud for right now, you don't want that to take that entire part of the cloud down without also re-enabling it when it comes back or unpartitions. | |
| 18:09:08 | superdan | SpamapS: failure to download an image is pretty legit to disable, no? not flavor too small though, obviously | |
| 18:09:34 | superdan | yeah, if glance is down that's fair I guess | |
| 18:09:37 | SpamapS | jrollinhatin: ah good point. K, I've not actually driven a CVE before. | |
| 18:09:49 | SpamapS | superdan: failure to download an image is pretty ambiguous. | |
| 18:10:08 | SpamapS | And very likely to be temporary. | |
| 18:10:13 | superdan | SpamapS: it's easy for some computes to be unable to download images, partition like you said | |
| 18:10:39 | SpamapS | Like, to be able to really know where the fault is, you need something like Vitrage. | |
| 18:10:40 | superdan | SpamapS: so this came up as one compute node being unable to do basic stuff like that can _also_ take down your whole cloud because it attracts all new builds and fails them | |
| 18:10:56 | superdan | scientific cloud with no reschedules, pack mode, one compute node takes the whole thing down with it | |
| 18:11:15 | SpamapS | Right, so maybe the right thing is to split into two categories. | |
| 18:11:27 | SpamapS | Things to disable permanently for, and things to drop from retry list. | |
| 18:11:28 | superdan | maybe what we should have done is just make a weigher or filter based on the fail score | |
| 18:11:36 | SpamapS | or.. not retry | |
| 18:11:39 | mnaser | i like that idea | |
| 18:11:40 | SpamapS | but things to stop scheduling for | |
| 18:11:56 | superdan | mnaser: that's a fairly easy iteration from where we are | |
| 18:12:21 | SpamapS | This is where k8s's scheduler does something smarter. If they get a failure on a node, they retry for a while on that node, and then they start backing off from it.. scheduling less and less frequently to it. | |
| 18:12:27 | SpamapS | It's not disabled.. they keep trying it.. but less and less often. | |
| 18:13:05 | SpamapS | A weighter would be perfect. | |
| 18:13:17 | SpamapS | Just expose the consecutive failures counter and have a coeficient based on that. | |
| 18:13:35 | SpamapS | So as it gets higher, the node gets less and less attempts. | |
| 18:14:09 | jgwentworth | makes sense | |
| 18:14:11 | SpamapS | Arguably it should not reset to 0 on success, but just decrement. | |
| 18:14:22 | superdan | SpamapS: why? | |
| 18:14:29 | SpamapS | Flapping. | |
| 18:14:42 | superdan | SpamapS: you end up with a pretty hard to explain situation for why some compute nodes rarely get used, | |
| 18:14:54 | SpamapS | If you reset it to 0, you start dumping lots on it again, which may make it start failing again. | |
| 18:14:58 | superdan | which boils down to "it was partitioned for a long time a month ago and hasn't recovered" | |
| 18:15:15 | SpamapS | Yeah maybe there needs to be a time based decrementer too. | |
| 18:15:16 | superdan | so decrement by ten on each success | |
| 18:15:30 | SpamapS | Like, decrement by 1 every x seconds, and 1 every success. | |
| 18:15:33 | superdan | counter = min(0, counter - 10) | |
| 18:15:43 | superdan | SpamapS: you're building a complicated thing that generates DB traffic | |
| 18:15:49 | SpamapS | Yeah or you could weight heavily toward successes. | |
| 18:16:29 | SpamapS | am I? I was thinking this number lives in the compute node and just goes along with ram/cpu/etc. stats? | |
| 18:16:54 | superdan | SpamapS: we're trying to get rid of those stats that get reported all the time for no reason | |
| 18:17:00 | SpamapS | and thus gets included in the current weighters? | |
| 18:17:29 | SpamapS | Oh, how are we going to schedule without them? (Sorry for the basic questions, I'm not up to speed on current refactors) | |
| 18:18:14 | superdan | we report resources in different ways now, and we don't constantly report "yep, the compute node still has a total of 192G of ram, same as last minute" | |
| 18:18:35 | SpamapS | Yeah we just report on changes or something, yes? | |
| 18:18:47 | SpamapS | so could be the same for this score, no? | |
| 18:18:52 | superdan | well, we do via the old mechanism, but that's what we want to remove, and eventually hopefully the need to even run periodically | |
| 18:19:21 | SpamapS | can you point me at a description of those different ways? I want to understand. :) | |
| 18:19:24 | superdan | the thing that is responsible for checking resources has nothing to do with this either | |
| 18:19:48 | superdan | everything we've done with placement lately? | |
| 18:19:52 | superdan | I can't point you at one thing | |
| 18:20:05 | SpamapS | I don't want you to have to type it all into IRC.. if there's just a description, I can think more clearly about how to make a weighter based on the scheduling health, which I think might be a nice way to evolve this feature. | |
| 18:20:34 | superdan | it's not the weigher that's a problem, of course, | |
| 18:21:20 | superdan | it's just more complicated if you have a periodic, which has a decay interval (config) and runs either independently (we have so many) or glommed onto something else like resource audit, which it has nothing to do with | |
| 18:21:52 | superdan | and a successful boot either zeroing or aggressively decrementing the value is semantically closer to what we implemented initially, which people liked | |
| 18:22:12 | SpamapS | I think it makes sense to decay it based on time, but maybe there are better ways. I like the idea of just having a coeficient to pull the fail counter down faster than it rises. | |
| 18:23:32 | superdan | the less we change the behavior, the more likely we are to be able to maybe backport something too | |
| 18:23:59 | jgwentworth | SpamapS: coincidentally I happened upon this earlier today when trying to answer a different question, might be a good starting point for learning more if you're interested https://docs.openstack.org/nova/latest/reference/scheduling.html | |
| 18:24:10 | SpamapS | Yeah, I think we'd just have to tell people to turn it off if they're in a situation where they might get DoS'd. | |
| 18:24:30 | SpamapS | jgwentworth: thanks, was just reading that! :) | |
| 18:24:36 | jgwentworth | oh, heh | |
| 18:25:15 | SpamapS | like, 30s before you sent, so, we're on the same page. Literally. | |
| 18:25:24 | jgwentworth | haha | |
| 18:26:00 | SpamapS | The thing is, it would have to go down over time or nodes that got way off the rails might never see activity again. | |
| 18:26:21 | superdan | if you're packing, you're opting into empty nodes right? | |
| 18:26:57 | SpamapS | like if some aggregate got really full and a poorly weighted HV that is unhealthy gets scheduled and fails a lot for a while.. its score gets really high, then you add capacity somehow.. that node may never get any attempts unless you decrement or reset the counter. | |
| 18:27:03 | superdan | I'm just trying to think about how we can do this initially with minimal change (config, code) and minimal semantic difference | |
| 18:27:12 | superdan | for the purposes of applying to to existing stuff | |
| 18:27:31 | superdan | if we're not interested in backporting it (backporting a weigher would be a first I bet) then maybe it doesn't matter | |
| 18:27:56 | SpamapS | Yeah I'm not sure. Still thinking through what might be a more self-managing place for the feature is all. | |
| 18:29:16 | SpamapS | Good chat. I'll give it some thought, and see if we can also help the VMT determine if we should notify users about the potential for DoS. | |
| 18:35:00 | superdan | I just looked over all our periodics and I don't think it fits with any of the existing ones | |
| 18:35:26 | superdan | there are a couple that would be closeish, but would still look really random to just do this other thing in the middle, | |
| 18:35:58 | harlowja | people can if they want try http://paste.openstack.org/show/722481/ on some public openstack cloud, though i'd recommend u communicate with their operators before doing it at any scale | |
| 18:36:05 | superdan | plus you probably want to be able to control the interval of this, which means it needs its own knob | |
| 18:36:17 | harlowja | we ran that on one of our idle clouds and it was able to knock off 2 compute nodes in about 15 minutes | |
| 18:36:34 | harlowja | afaik because of https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1804-L1810 | |
| 18:36:48 | harlowja | line 81 -> 85 are the 'triggers' | |
| 18:37:01 | harlowja | boot faster than nova can build | |
| 18:37:06 | jrollinhatin | probably best not to post a working exploit for a security bug in an irc channel this large :| | |
| 18:37:07 | harlowja | *boot and delete | |
| 18:37:15 | superdan | jrollinhatin: yeah nfs | |
| 18:39:00 | penick | harlowja dude | |
| 18:41:13 | harlowja | i'd have kept it on https://bugs.launchpad.net/nova/+bug/1774527 but that is already public as well | |
| 18:41:14 | openstack | Launchpad bug 1742102 in OpenStack Compute (nova) "duplicate for #1774527 Simple user can disable compute" [High,In progress] - Assigned to Matt Riedemann (mriedem) | |
| 18:41:53 | SpamapS | Cat's been out of the bag since January guys. That's not exactly rocket science. | |
| 18:43:24 | superdan | we've discussed it in public many times before then, | |