| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-02-23 | |||
| 16:00:49 | jroll | leakypipes: there's vomit everywhere in here from that | |
| 16:00:54 | leakypipes | :) | |
| 16:01:18 | dansmith | bauwser: how many groups? you lost me there :) | |
| 16:01:31 | mriedem | i think he means group of VMs per host in the group? | |
| 16:01:32 | bauwser | man | |
| 16:01:40 | bauwser | mriedem: bonus point | |
| 16:01:44 | dansmith | oh min-redundancy you mean? | |
| 16:01:54 | jroll | dansmith: I read through it, I agree with what you're suggesting there, thanks | |
| 16:01:59 | dansmith | I think that's how most people will want to define a thing like this | |
| 16:02:08 | dansmith | and jroll does, so.. end of story | |
| 16:02:11 | bauwser | call it failure zone | |
| 16:02:34 | bauwser | but if we call it failure zone, then leakypipes would kill me | |
| 16:02:40 | bauwser | and I'm scared | |
| 16:02:46 | leakypipes | bauwser: indeed I would. | |
| 16:03:15 | bauwser | why people are doing instance groups besides the idea that they expect groups of failure domains | |
| 16:03:16 | bauwser | ? | |
| 16:03:49 | bauwser | leakypipes: you don't imagine my mood now... restating https://www.thelocal.fr/20180223/french-cheese-wars-are-the-days-of-the-real-normandy-camembert-numbered | |
| 16:03:52 | leakypipes | bauwser: because... N. F. V. | |
| 16:04:06 | leakypipes | hahaha | |
| 16:04:20 | leakypipes | that's awesome. | |
| 16:04:41 | leakypipes | only in La France | |
| 16:08:51 | leakypipes | mriedem: did you say that https://review.openstack.org/#/c/546925/2/specs/rocky/approved/allow-specifying-limit-for-affrinity-group.rst was somehow related to something sgordon put together? | |
| 16:09:15 | mriedem | see the bp | |
| 16:09:22 | mriedem | https://blueprints.launchpad.net/nova/+spec/complex-soft-anti-affinity-policies | |
| 16:09:32 | leakypipes | oh... 113355 | |
| 16:10:00 | leakypipes | ok, not 113355 | |
| 16:10:16 | mriedem | https://review.openstack.org/#/c/546925/ | |
| 16:10:38 | mriedem | https://review.openstack.org/#/c/224325/ is the old backlog spec | |
| 16:10:44 | leakypipes | mriedem: yeah, was confused since the spec is called something different than the blueprint. | |
| 16:13:56 | leakypipes | mriedem: ftr, k8s does this very well. | |
| 16:14:07 | leakypipes | mriedem: because, you know, it's an *orchestrator*. | |
| 16:14:31 | leakypipes | mriedem: would be much better, IMHO, to recommend to your group to install k8s on the nova compute hosts in the NFVi and just have at it. | |
| 16:14:36 | dansmith | leakypipes: do we provide heat enough control to do this externally? | |
| 16:15:47 | jroll | k8s does this well because the orchestrator and the scheduler can work together, imo | |
| 16:15:52 | leakypipes | dansmith: initially, yes. but k8s will do it on an ongoing basis -- i.e. if the state of the pods on a node changes, k8s will re-evaluate the tolerances and scheduling conditions and place pods on different nodes accordingly. | |
| 16:16:09 | leakypipes | jroll: actually, they don't. | |
| 16:16:11 | dansmith | right, sure | |
| 16:16:19 | dansmith | I guess I'm not sure how heat would do this with our api | |
| 16:19:00 | leakypipes | jroll: what actually happens in k8s is the descheduler (an orchestrator piece, not part of the scheduler) will evict pods from nodes that no longer meet scheduling constraints and the scheduler will try to re-place those evicted pods somewhere else (or the same node if conditions happen to be right at that specific moment) | |
| 16:19:58 | leakypipes | jroll: the descheduler was originally called the rescheduler, but there's actually a rescheduler thing in k8s that does a different type of eviction, and that rescheduler piece is being deprecated. | |
| 16:20:03 | jroll | leakypipes: sure, but the scheduler is aware of the orchestrator's requirements, right? or does it randomly choose and the descheduler continues to evict until things are right? | |
| 16:20:23 | leakypipes | jroll: the reason the k8s works well is specifically because there is no coupling between the orchestrator and the resource tracker/scheduler. | |
| 16:21:15 | leakypipes | jroll: scheduler is not aware of descheduler, no. | |
| 16:21:52 | jroll | leakypipes: I probably need to do some reading to fully understand | |
| 16:22:30 | leakypipes | jroll: descheduler is basically just a daemon that runs a periodic loop that checks to see if the constraints that were met on initial placement are still met. and if not, simply evicts the pod. that puts the pod into a "ready for scheduling" state and the scheduler, on its next loop, will scheduler that pod again. | |
| 16:23:16 | openstackgerrit | Claudiu Belu proposed openstack/nova master: hyper-v: autospec classes before they are instantiated https://review.openstack.org/342211 | |
| 16:23:22 | jroll | curious how heat would accomplish this affinity model, though. does heat know about compute nodes, which instances are on them, the affinity characteristics of said compute nodes (to know if two instances are actually in a separate power domain, for example?) | |
| 16:23:24 | leakypipes | incidentally, this is precisely how I proposed that spot instances be handled in Nova instead of a highly-coupled-to-the-scheduler architecture that the guys at Inria proposed. | |
| 16:23:59 | jroll | leakypipes: is the scheduler aware of the constraints, like affinity, that need to be met? (and therefore that the descheduler is trying to meet?) | |
| 16:25:00 | leakypipes | jroll: sure. the constraints are simply part of the pod scheduling policy that is associated with the pod when it's originally created. | |
| 16:25:19 | bauwser | okay, so let's pretend we have "affinity knobs" :p | |
| 16:26:07 | bauwser | leakypipes: I guess you know that some other openstack projects are trying to achieve the descheduler thing in OpenStack right? | |
| 16:26:37 | bauwser | the reconciliation of resource usage IIUC what the k8s descheduler is trying to achieve | |
| 16:26:43 | jroll | leakypipes: cool, so does nova expose sufficient data (instances, instance groups, affinity policies, compute node information, etc) for an external thing like heat to accomplish things like this spec proposes? | |
| 16:26:55 | leakypipes | jroll: yes. | |
| 16:27:07 | leakypipes | jroll: for the most part :) | |
| 16:27:14 | jroll | heh | |
| 16:27:22 | leakypipes | jroll: but yes. | |
| 16:27:33 | leakypipes | jroll: heat is admin user, no? | |
| 16:27:44 | bauwser | leakypipes: heat can have trusted users | |
| 16:27:48 | bauwser | I mean trusts | |
| 16:27:51 | jroll | leakypipes: I've no clue, honestly | |
| 16:28:01 | bauwser | it's user delegation | |
| 16:28:10 | bauwser | but anyway | |
| 16:28:10 | leakypipes | bauwser: and trusts don't work with federated keystone, so... | |
| 16:28:19 | bauwser | it's 5pm here and I need to pack my stuff :) | |
| 16:28:22 | leakypipes | :) | |
| 16:28:30 | dansmith | leakypipes: so how would heat go about doing this? | |
| 16:28:34 | bauwser | so, see you folks in a pub or somewhere else | |
| 16:28:44 | bauwser | like in a stadium | |
| 16:28:52 | dansmith | leakypipes: it could boot the first three instances in an anti-affinity group and then after that, the fourth and later go anywhere? | |
| 16:28:53 | jroll | leakypipes: but, this means that an openstack user needs to rely on their cloud provider to provide heat to be able to do this, rather than being able to write a little program to diy it | |
| 16:29:00 | jroll | which might be fine but kinda sucks | |
| 16:29:14 | leakypipes | dansmith: are you asking about heat or k8s? | |
| 16:29:30 | dansmith | jroll: does it? I kinda expected people would be able to use heat themselves with just their creds | |
| 16:29:33 | dansmith | leakypipes: heat and nova | |
| 16:29:38 | leakypipes | jroll: "a little program"? :) | |
| 16:29:43 | jroll | :P | |
| 16:29:55 | jroll | dansmith: not if you need admin rights to nova to get compute node information | |
| 16:30:10 | dansmith | jroll: you don't need that, and even if you had it I don't think it would help | |
| 16:30:17 | leakypipes | dansmith: no.. if heat specifies the same instance group for all 5 instances, Nova guarantees they will go on different compute hosts... | |
| 16:30:20 | dansmith | jroll: users can tell compute nodes apart, just not the actual hostnames | |
| 16:30:29 | leakypipes | dansmith: but you already knew that, so I suspect I am missing yoru point. | |
| 16:31:05 | dansmith | leakypipes: right, but how would heat implement this "no more than three instances per host" requirement with nova? | |
| 16:31:07 | jroll | dansmith: fair point, I guess I'm thinking ahead to affinity around power/network domains, sorry | |
| 16:31:32 | dansmith | jroll: yeah, but unless you let heat create aggregates, I still don't think being admin is helpful | |
| 16:31:47 | dansmith | jroll: and each tenant creating aggregates via heat would be kinda crazy I think | |
| 16:32:01 | dansmith | maybe just using them with custom flavors? I dunno | |
| 16:32:20 | jroll | dansmith: do regular users have access to aggregate info? | |
| 16:32:21 | dansmith | I'm legit asking because I don't know much about heat | |
| 16:32:23 | leakypipes | dansmith: how would it implement it in a transactional/atomic way? it couldn't. but it could certainly call a compute API to get the compute node -> instance association for all instances in a group and do the calculation itself, no?' | |
| 16:32:27 | dansmith | if this is doable outside of nova with heat, then I'm super -2 on the spec, I just don't know that it's possible | |
| 16:32:37 | jroll | I don't either, so I hope you aren't asking me :) | |
| 16:32:47 | dansmith | jroll: no, but that's what I'm saying, I don't know that you'd give your heat admin user ability to do much with them either | |
| 16:33:17 | dansmith | leakypipes: but it can't control placement of those things | |
| 16:34:03 | leakypipes | dansmith: hmm, true nuf | |
| 16:34:33 | leakypipes | dansmith: yeah, you're right. it would need to be a part of the request spec sent to placement/scheduler :( | |
| 16:34:36 | jroll | dansmith: I would think you'd need enough info to be able to map instances -> aggregates, decide if the spread is sufficient (>3 power domains or whatever), and I was going to say whatever jay is thinking from there :) | |
| 16:34:54 | dansmith | that's why I'm saying I expect this is a primitive we need to provide so heat can provide the automatic kill-one-spawn-one rebalancing sort of behavior | |
| 16:35:15 | dansmith | jroll: well, you still can't control where a new instance lands enough to effect this policy I think | |