| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-02-23 | |||
| 16:09:15 | mriedem | see the bp | |
| 16:09:22 | mriedem | https://blueprints.launchpad.net/nova/+spec/complex-soft-anti-affinity-policies | |
| 16:09:32 | leakypipes | oh... 113355 | |
| 16:10:00 | leakypipes | ok, not 113355 | |
| 16:10:16 | mriedem | https://review.openstack.org/#/c/546925/ | |
| 16:10:38 | mriedem | https://review.openstack.org/#/c/224325/ is the old backlog spec | |
| 16:10:44 | leakypipes | mriedem: yeah, was confused since the spec is called something different than the blueprint. | |
| 16:13:56 | leakypipes | mriedem: ftr, k8s does this very well. | |
| 16:14:07 | leakypipes | mriedem: because, you know, it's an *orchestrator*. | |
| 16:14:31 | leakypipes | mriedem: would be much better, IMHO, to recommend to your group to install k8s on the nova compute hosts in the NFVi and just have at it. | |
| 16:14:36 | dansmith | leakypipes: do we provide heat enough control to do this externally? | |
| 16:15:47 | jroll | k8s does this well because the orchestrator and the scheduler can work together, imo | |
| 16:15:52 | leakypipes | dansmith: initially, yes. but k8s will do it on an ongoing basis -- i.e. if the state of the pods on a node changes, k8s will re-evaluate the tolerances and scheduling conditions and place pods on different nodes accordingly. | |
| 16:16:09 | leakypipes | jroll: actually, they don't. | |
| 16:16:11 | dansmith | right, sure | |
| 16:16:19 | dansmith | I guess I'm not sure how heat would do this with our api | |
| 16:19:00 | leakypipes | jroll: what actually happens in k8s is the descheduler (an orchestrator piece, not part of the scheduler) will evict pods from nodes that no longer meet scheduling constraints and the scheduler will try to re-place those evicted pods somewhere else (or the same node if conditions happen to be right at that specific moment) | |
| 16:19:58 | leakypipes | jroll: the descheduler was originally called the rescheduler, but there's actually a rescheduler thing in k8s that does a different type of eviction, and that rescheduler piece is being deprecated. | |
| 16:20:03 | jroll | leakypipes: sure, but the scheduler is aware of the orchestrator's requirements, right? or does it randomly choose and the descheduler continues to evict until things are right? | |
| 16:20:23 | leakypipes | jroll: the reason the k8s works well is specifically because there is no coupling between the orchestrator and the resource tracker/scheduler. | |
| 16:21:15 | leakypipes | jroll: scheduler is not aware of descheduler, no. | |
| 16:21:52 | jroll | leakypipes: I probably need to do some reading to fully understand | |
| 16:22:30 | leakypipes | jroll: descheduler is basically just a daemon that runs a periodic loop that checks to see if the constraints that were met on initial placement are still met. and if not, simply evicts the pod. that puts the pod into a "ready for scheduling" state and the scheduler, on its next loop, will scheduler that pod again. | |
| 16:23:16 | openstackgerrit | Claudiu Belu proposed openstack/nova master: hyper-v: autospec classes before they are instantiated https://review.openstack.org/342211 | |
| 16:23:22 | jroll | curious how heat would accomplish this affinity model, though. does heat know about compute nodes, which instances are on them, the affinity characteristics of said compute nodes (to know if two instances are actually in a separate power domain, for example?) | |
| 16:23:24 | leakypipes | incidentally, this is precisely how I proposed that spot instances be handled in Nova instead of a highly-coupled-to-the-scheduler architecture that the guys at Inria proposed. | |
| 16:23:59 | jroll | leakypipes: is the scheduler aware of the constraints, like affinity, that need to be met? (and therefore that the descheduler is trying to meet?) | |
| 16:25:00 | leakypipes | jroll: sure. the constraints are simply part of the pod scheduling policy that is associated with the pod when it's originally created. | |
| 16:25:19 | bauwser | okay, so let's pretend we have "affinity knobs" :p | |
| 16:26:07 | bauwser | leakypipes: I guess you know that some other openstack projects are trying to achieve the descheduler thing in OpenStack right? | |
| 16:26:37 | bauwser | the reconciliation of resource usage IIUC what the k8s descheduler is trying to achieve | |
| 16:26:43 | jroll | leakypipes: cool, so does nova expose sufficient data (instances, instance groups, affinity policies, compute node information, etc) for an external thing like heat to accomplish things like this spec proposes? | |
| 16:26:55 | leakypipes | jroll: yes. | |
| 16:27:07 | leakypipes | jroll: for the most part :) | |
| 16:27:14 | jroll | heh | |
| 16:27:22 | leakypipes | jroll: but yes. | |
| 16:27:33 | leakypipes | jroll: heat is admin user, no? | |
| 16:27:44 | bauwser | leakypipes: heat can have trusted users | |
| 16:27:48 | bauwser | I mean trusts | |
| 16:27:51 | jroll | leakypipes: I've no clue, honestly | |
| 16:28:01 | bauwser | it's user delegation | |
| 16:28:10 | leakypipes | bauwser: and trusts don't work with federated keystone, so... | |
| 16:28:10 | bauwser | but anyway | |
| 16:28:19 | bauwser | it's 5pm here and I need to pack my stuff :) | |
| 16:28:22 | leakypipes | :) | |
| 16:28:30 | dansmith | leakypipes: so how would heat go about doing this? | |
| 16:28:34 | bauwser | so, see you folks in a pub or somewhere else | |
| 16:28:44 | bauwser | like in a stadium | |
| 16:28:52 | dansmith | leakypipes: it could boot the first three instances in an anti-affinity group and then after that, the fourth and later go anywhere? | |
| 16:28:53 | jroll | leakypipes: but, this means that an openstack user needs to rely on their cloud provider to provide heat to be able to do this, rather than being able to write a little program to diy it | |
| 16:29:00 | jroll | which might be fine but kinda sucks | |
| 16:29:14 | leakypipes | dansmith: are you asking about heat or k8s? | |
| 16:29:30 | dansmith | jroll: does it? I kinda expected people would be able to use heat themselves with just their creds | |
| 16:29:33 | dansmith | leakypipes: heat and nova | |
| 16:29:38 | leakypipes | jroll: "a little program"? :) | |
| 16:29:43 | jroll | :P | |
| 16:29:55 | jroll | dansmith: not if you need admin rights to nova to get compute node information | |
| 16:30:10 | dansmith | jroll: you don't need that, and even if you had it I don't think it would help | |
| 16:30:17 | leakypipes | dansmith: no.. if heat specifies the same instance group for all 5 instances, Nova guarantees they will go on different compute hosts... | |
| 16:30:20 | dansmith | jroll: users can tell compute nodes apart, just not the actual hostnames | |
| 16:30:29 | leakypipes | dansmith: but you already knew that, so I suspect I am missing yoru point. | |
| 16:31:05 | dansmith | leakypipes: right, but how would heat implement this "no more than three instances per host" requirement with nova? | |
| 16:31:07 | jroll | dansmith: fair point, I guess I'm thinking ahead to affinity around power/network domains, sorry | |
| 16:31:32 | dansmith | jroll: yeah, but unless you let heat create aggregates, I still don't think being admin is helpful | |
| 16:31:47 | dansmith | jroll: and each tenant creating aggregates via heat would be kinda crazy I think | |
| 16:32:01 | dansmith | maybe just using them with custom flavors? I dunno | |
| 16:32:20 | jroll | dansmith: do regular users have access to aggregate info? | |
| 16:32:21 | dansmith | I'm legit asking because I don't know much about heat | |
| 16:32:23 | leakypipes | dansmith: how would it implement it in a transactional/atomic way? it couldn't. but it could certainly call a compute API to get the compute node -> instance association for all instances in a group and do the calculation itself, no?' | |
| 16:32:27 | dansmith | if this is doable outside of nova with heat, then I'm super -2 on the spec, I just don't know that it's possible | |
| 16:32:37 | jroll | I don't either, so I hope you aren't asking me :) | |
| 16:32:47 | dansmith | jroll: no, but that's what I'm saying, I don't know that you'd give your heat admin user ability to do much with them either | |
| 16:33:17 | dansmith | leakypipes: but it can't control placement of those things | |
| 16:34:03 | leakypipes | dansmith: hmm, true nuf | |
| 16:34:33 | leakypipes | dansmith: yeah, you're right. it would need to be a part of the request spec sent to placement/scheduler :( | |
| 16:34:36 | jroll | dansmith: I would think you'd need enough info to be able to map instances -> aggregates, decide if the spread is sufficient (>3 power domains or whatever), and I was going to say whatever jay is thinking from there :) | |
| 16:34:54 | dansmith | that's why I'm saying I expect this is a primitive we need to provide so heat can provide the automatic kill-one-spawn-one rebalancing sort of behavior | |
| 16:35:15 | dansmith | jroll: well, you still can't control where a new instance lands enough to effect this policy I think | |
| 16:35:18 | jroll | just boot and delete instances until it looks right :P | |
| 16:35:29 | dansmith | jroll: monte carlo scheduling? I'm in | |
| 16:35:30 | jroll | dansmith: right, I was hoping jay was going to solve that problem as I typed | |
| 16:35:46 | jroll | :P | |
| 16:36:36 | leakypipes | dansmith: there will be a point (already reached?) where this stuff is just too expensive to try and calculate for each scheduling request and makes the interface between scheduler and placement overly cumbersome. | |
| 16:37:04 | dansmith | leakypipes: I don't think this has anything to do with what scheduler asks of placement | |
| 16:37:17 | dansmith | leakypipes: what is described in the spec is a simple change to the current hard affinity filter | |
| 16:37:31 | dansmith | and what I was proposing would be similar but with some different math | |
| 16:37:56 | dansmith | the biggest change is "oh now we're going to have extra specs on instance groups" | |
| 16:38:08 | dansmith | which I think is a very big step we should not take lightly | |
| 16:38:11 | dansmith | (or maybe at all) | |
| 16:38:50 | dansmith | especially my point about changing that value on an existing group, because people will expect that if they change that from 3 to 2, existing instances get moved around | |
| 16:38:54 | dansmith | and that ain't hap'nan | |
| 16:48:02 | leakypipes | dansmith, jroll: if you want some mind-bending reading from k8s on this subject: https://github.com/kubernetes/kubernetes/pull/18265/files | |
| 16:48:17 | leakypipes | dansmith, jroll: note that k8s affinity stuff is still in beta... | |
| 16:48:50 | jroll | oh my | |
| 16:49:29 | mriedem | dansmith: i think the 'extra spec' term in his spec is a mistake, or just a WIP thing, | |
| 16:49:39 | mriedem | i don't think we should add a random bag of extra specs to instance groups either | |
| 16:50:04 | mriedem | dansmith: i also said updating the max-per-host value on an existing group isn't going to happen | |
| 16:50:15 | mriedem | for the same reasons we don't allow changing policy on an existing group, or changing membership | |
| 16:51:00 | leakypipes | what about an awkward bag of flaccid specs? | |
| 16:51:27 | mriedem | if it's as simple as putting an attribute on a group in a hard affinity policy that says, instead of allowing no more than 1 vm from this group per host, you can allow up to 3 (or whatever), then that seems to fit the bill | |