| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-02-23 | |||
| 15:56:19 | jroll | (disclaimer, haven't read the spec) so, use case: I want to be able to say "I want 20 instances spread across at least 5 hosts, but preferably across more" - I think that's what is proposed here (based on this conversation), is that correct? | |
| 15:56:19 | dansmith | they just don't know it | |
| 15:56:34 | dansmith | jroll: no, that's what I was saying though | |
| 15:56:39 | dansmith | that's what I would prefer I mean | |
| 15:56:55 | jroll | dansmith: and this is what the spec thinks it is proposing, but it's really not? :) | |
| 15:56:57 | bauwser | jroll: that's exactly what weighers aim to achieve | |
| 15:57:11 | dansmith | jroll: no, that's not what the spec thinks its proposing | |
| 15:57:11 | bauwser | for that usecase | |
| 15:57:22 | mriedem | jroll: dansmith: well one thing is when you create the group, you don't say how many instances are going to be in it | |
| 15:57:26 | bauwser | ie. ponder your list of hosts based on some criterias that are given by weighers | |
| 15:57:35 | dansmith | jroll: it's proposing that you know how big the group will ever be, and set the allowed overlap per host to some number so you get the desired amount of minimum redundancy | |
| 15:57:37 | mriedem | so "I want 20 instances spread across at least 5 hosts" is a bit chicken and egg to me | |
| 15:57:41 | dansmith | which is much harder to reason about | |
| 15:57:54 | jroll | dansmith: ah, ok, will read again | |
| 15:57:55 | mriedem | but i guess if you already know when you create the group how many VMs are going to be in it, then that works | |
| 15:58:12 | dansmith | mriedem: that's what this spec requires, is you to know how many instances will be in there | |
| 15:58:46 | dansmith | mriedem: but, if you say "I want minimum redundancy of three" then you don't get any overlap until you boot the fourth and now you can start to double up because you've achieved that minimum level | |
| 16:00:23 | leakypipes | dansmith: I think you saw the use case section begin with "As a NFV user, ..." and immediately threw up in your mouth ;) | |
| 16:00:25 | bauwser | dansmith: it's very different | |
| 16:00:47 | bauwser | dansmith: but I agree, describing for anti-affinity how many groups you want is somehow reasonable | |
| 16:00:49 | jroll | leakypipes: there's vomit everywhere in here from that | |
| 16:00:54 | leakypipes | :) | |
| 16:01:18 | dansmith | bauwser: how many groups? you lost me there :) | |
| 16:01:31 | mriedem | i think he means group of VMs per host in the group? | |
| 16:01:32 | bauwser | man | |
| 16:01:40 | bauwser | mriedem: bonus point | |
| 16:01:44 | dansmith | oh min-redundancy you mean? | |
| 16:01:54 | jroll | dansmith: I read through it, I agree with what you're suggesting there, thanks | |
| 16:01:59 | dansmith | I think that's how most people will want to define a thing like this | |
| 16:02:08 | dansmith | and jroll does, so.. end of story | |
| 16:02:11 | bauwser | call it failure zone | |
| 16:02:34 | bauwser | but if we call it failure zone, then leakypipes would kill me | |
| 16:02:40 | bauwser | and I'm scared | |
| 16:02:46 | leakypipes | bauwser: indeed I would. | |
| 16:03:15 | bauwser | why people are doing instance groups besides the idea that they expect groups of failure domains | |
| 16:03:16 | bauwser | ? | |
| 16:03:49 | bauwser | leakypipes: you don't imagine my mood now... restating https://www.thelocal.fr/20180223/french-cheese-wars-are-the-days-of-the-real-normandy-camembert-numbered | |
| 16:03:52 | leakypipes | bauwser: because... N. F. V. | |
| 16:04:06 | leakypipes | hahaha | |
| 16:04:20 | leakypipes | that's awesome. | |
| 16:04:41 | leakypipes | only in La France | |
| 16:08:51 | leakypipes | mriedem: did you say that https://review.openstack.org/#/c/546925/2/specs/rocky/approved/allow-specifying-limit-for-affrinity-group.rst was somehow related to something sgordon put together? | |
| 16:09:15 | mriedem | see the bp | |
| 16:09:22 | mriedem | https://blueprints.launchpad.net/nova/+spec/complex-soft-anti-affinity-policies | |
| 16:09:32 | leakypipes | oh... 113355 | |
| 16:10:00 | leakypipes | ok, not 113355 | |
| 16:10:16 | mriedem | https://review.openstack.org/#/c/546925/ | |
| 16:10:38 | mriedem | https://review.openstack.org/#/c/224325/ is the old backlog spec | |
| 16:10:44 | leakypipes | mriedem: yeah, was confused since the spec is called something different than the blueprint. | |
| 16:13:56 | leakypipes | mriedem: ftr, k8s does this very well. | |
| 16:14:07 | leakypipes | mriedem: because, you know, it's an *orchestrator*. | |
| 16:14:31 | leakypipes | mriedem: would be much better, IMHO, to recommend to your group to install k8s on the nova compute hosts in the NFVi and just have at it. | |
| 16:14:36 | dansmith | leakypipes: do we provide heat enough control to do this externally? | |
| 16:15:47 | jroll | k8s does this well because the orchestrator and the scheduler can work together, imo | |
| 16:15:52 | leakypipes | dansmith: initially, yes. but k8s will do it on an ongoing basis -- i.e. if the state of the pods on a node changes, k8s will re-evaluate the tolerances and scheduling conditions and place pods on different nodes accordingly. | |
| 16:16:09 | leakypipes | jroll: actually, they don't. | |
| 16:16:11 | dansmith | right, sure | |
| 16:16:19 | dansmith | I guess I'm not sure how heat would do this with our api | |
| 16:19:00 | leakypipes | jroll: what actually happens in k8s is the descheduler (an orchestrator piece, not part of the scheduler) will evict pods from nodes that no longer meet scheduling constraints and the scheduler will try to re-place those evicted pods somewhere else (or the same node if conditions happen to be right at that specific moment) | |
| 16:19:58 | leakypipes | jroll: the descheduler was originally called the rescheduler, but there's actually a rescheduler thing in k8s that does a different type of eviction, and that rescheduler piece is being deprecated. | |
| 16:20:03 | jroll | leakypipes: sure, but the scheduler is aware of the orchestrator's requirements, right? or does it randomly choose and the descheduler continues to evict until things are right? | |
| 16:20:23 | leakypipes | jroll: the reason the k8s works well is specifically because there is no coupling between the orchestrator and the resource tracker/scheduler. | |
| 16:21:15 | leakypipes | jroll: scheduler is not aware of descheduler, no. | |
| 16:21:52 | jroll | leakypipes: I probably need to do some reading to fully understand | |
| 16:22:30 | leakypipes | jroll: descheduler is basically just a daemon that runs a periodic loop that checks to see if the constraints that were met on initial placement are still met. and if not, simply evicts the pod. that puts the pod into a "ready for scheduling" state and the scheduler, on its next loop, will scheduler that pod again. | |
| 16:23:16 | openstackgerrit | Claudiu Belu proposed openstack/nova master: hyper-v: autospec classes before they are instantiated https://review.openstack.org/342211 | |
| 16:23:22 | jroll | curious how heat would accomplish this affinity model, though. does heat know about compute nodes, which instances are on them, the affinity characteristics of said compute nodes (to know if two instances are actually in a separate power domain, for example?) | |
| 16:23:24 | leakypipes | incidentally, this is precisely how I proposed that spot instances be handled in Nova instead of a highly-coupled-to-the-scheduler architecture that the guys at Inria proposed. | |
| 16:23:59 | jroll | leakypipes: is the scheduler aware of the constraints, like affinity, that need to be met? (and therefore that the descheduler is trying to meet?) | |
| 16:25:00 | leakypipes | jroll: sure. the constraints are simply part of the pod scheduling policy that is associated with the pod when it's originally created. | |
| 16:25:19 | bauwser | okay, so let's pretend we have "affinity knobs" :p | |
| 16:26:07 | bauwser | leakypipes: I guess you know that some other openstack projects are trying to achieve the descheduler thing in OpenStack right? | |
| 16:26:37 | bauwser | the reconciliation of resource usage IIUC what the k8s descheduler is trying to achieve | |
| 16:26:43 | jroll | leakypipes: cool, so does nova expose sufficient data (instances, instance groups, affinity policies, compute node information, etc) for an external thing like heat to accomplish things like this spec proposes? | |
| 16:26:55 | leakypipes | jroll: yes. | |
| 16:27:07 | leakypipes | jroll: for the most part :) | |
| 16:27:14 | jroll | heh | |
| 16:27:22 | leakypipes | jroll: but yes. | |
| 16:27:33 | leakypipes | jroll: heat is admin user, no? | |
| 16:27:44 | bauwser | leakypipes: heat can have trusted users | |
| 16:27:48 | bauwser | I mean trusts | |
| 16:27:51 | jroll | leakypipes: I've no clue, honestly | |
| 16:28:01 | bauwser | it's user delegation | |
| 16:28:10 | bauwser | but anyway | |
| 16:28:10 | leakypipes | bauwser: and trusts don't work with federated keystone, so... | |
| 16:28:19 | bauwser | it's 5pm here and I need to pack my stuff :) | |
| 16:28:22 | leakypipes | :) | |
| 16:28:30 | dansmith | leakypipes: so how would heat go about doing this? | |
| 16:28:34 | bauwser | so, see you folks in a pub or somewhere else | |
| 16:28:44 | bauwser | like in a stadium | |
| 16:28:52 | dansmith | leakypipes: it could boot the first three instances in an anti-affinity group and then after that, the fourth and later go anywhere? | |
| 16:28:53 | jroll | leakypipes: but, this means that an openstack user needs to rely on their cloud provider to provide heat to be able to do this, rather than being able to write a little program to diy it | |
| 16:29:00 | jroll | which might be fine but kinda sucks | |
| 16:29:14 | leakypipes | dansmith: are you asking about heat or k8s? | |
| 16:29:30 | dansmith | jroll: does it? I kinda expected people would be able to use heat themselves with just their creds | |
| 16:29:33 | dansmith | leakypipes: heat and nova | |
| 16:29:38 | leakypipes | jroll: "a little program"? :) | |
| 16:29:43 | jroll | :P | |
| 16:29:55 | jroll | dansmith: not if you need admin rights to nova to get compute node information | |
| 16:30:10 | dansmith | jroll: you don't need that, and even if you had it I don't think it would help | |