| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-02-11 | |||
| 17:03:32 | sean-k-mooney | am maybe but the performace of that might not be great | |
| 17:03:39 | sean-k-mooney | or acccptable. | |
| 17:03:44 | artom | dansmith, that would assign a NUMA topology even if the user didn't request it (explicitly or implicitly), no? | |
| 17:04:00 | sean-k-mooney | artom: ya but that would actully be fine | |
| 17:04:10 | artom | NUMA topologies have a whole bunch of limitations :) | |
| 17:04:10 | sean-k-mooney | they expressed no prefernce | |
| 17:04:15 | sean-k-mooney | so we can decided what that is | |
| 17:04:32 | sean-k-mooney | artom: you removed the main one e.g. live migration | |
| 17:04:40 | artom | True... | |
| 17:04:49 | openstackgerrit | Douglas Mendizábal proposed openstack/nova master: Allow TLS ciphers/protocols to be configurable for console proxies https://review.opendev.org/679502 | |
| 17:04:49 | openstackgerrit | Douglas Mendizábal proposed openstack/nova master: Allow TLS ciphers/protocols to be configurable for console proxies https://review.opendev.org/679502 | |
| 17:05:04 | dansmith | artom: yes, it would, but I definitely think that we should be getting to the place where we don't just pretend numa doesn't exist | |
| 17:05:08 | artom | I'd still feel uncomfortable springing that on people | |
| 17:05:23 | dansmith | artom: so we want people to not have to worry about it in detail, but not just let them totally ignore it, IMHO | |
| 17:06:09 | sean-k-mooney | i think that is something we shoudl consider in V when we consider changing the default | |
| 17:06:26 | dansmith | yup | |
| 17:06:35 | sean-k-mooney | if there is a sensable way to express it to placmenet or a sensible way to make it work form our side we should explore it | |
| 17:06:43 | dansmith | what I don't think we should do, is continue to have two distinct ways of running instances forever | |
| 17:07:10 | sean-k-mooney | i agree with that | |
| 17:07:31 | artom | Same here | |
| 17:07:49 | artom | Though I'm not entirely convinced giving everyone a NUMA topology is the way to do it | |
| 17:07:57 | dansmith | similar to cellsv1, that didn't work out well, and we never closed the feature and bug gap for people using it until cellsv2 where we just make everyone use it.. it's a little more overhead, but the gap is way smaller | |
| 17:08:17 | dansmith | artom: but everyone _has_ a numa topology | |
| 17:08:34 | artom | dansmith, you know what I mean ;) | |
| 17:08:43 | dansmith | if we need to retool the numa stuff in nova (and probably placement) then that's what we need to do | |
| 17:08:58 | artom | I think I'd lean towards the can_split stuff in placement | |
| 17:09:20 | artom | Though I admittedly have no idea how complicated that would be | |
| 17:09:24 | dansmith | yep, it seems like the major barrier here is lack of expressivity with what we ask of placement when we don't care as much | |
| 17:09:42 | artom | Like, if the host is NUMA, fine, expose it | |
| 17:10:06 | artom | But as you said, being able to say "I don't care about NUMA" would be the best way to solve this, I think | |
| 17:10:15 | sean-k-mooney | artom: can split is very ineefincet to implement in sql | |
| 17:10:36 | artom | So, I have beef with "placement has to be SQL", but that's just me ;) | |
| 17:10:55 | sean-k-mooney | artom: if "i dont care about numa" means we are free to invent numa toploigies then sure | |
| 17:11:13 | artom | sean-k-mooney, weeelll... wouldn't that lead to a bunch of packing problems? | |
| 17:11:17 | bauzas | folks, I have to disappear | |
| 17:11:28 | bauzas | leave comments, disagreements, concers on the spec | |
| 17:11:29 | sean-k-mooney | artom: oh it doesnt have to be sql. but it would be very hard to support can split with the current impeneation | |
| 17:11:41 | sean-k-mooney | artom: no | |
| 17:11:46 | artom | I guess not, if you retry with enough combinations of guest NUMA nodes and CPUs per node | |
| 17:12:48 | artom | So if you some how end up with NUMA0 with 1 CPU free, and NUMA1 with 3 CPUs free, and boot an instance with 4 CPUs, you would need to retry until you get to the numa0_cpus:1,numa2_cpus=3 combo | |
| 17:12:56 | sean-k-mooney | honestly doing 4 placment queries with 1-4 numa nodes is still proably faster then the numa toplogy filter today | |
| 17:13:16 | sean-k-mooney | but i dont think this is productive to continue now | |
| 17:13:20 | artom | True | |
| 17:13:26 | artom | (On the second point) | |
| 17:13:35 | dansmith | if, like I said, we had a range of nodes, I would think placement could just loop internally much faster than even our retry process | |
| 17:13:40 | dansmith | and just dump us more options in the first go | |
| 17:14:15 | sean-k-mooney | dansmith: ya its really a proablem of being able to express our actul constratin to placmenet so they can do an efficent thing | |
| 17:14:22 | dansmith | right | |
| 17:14:42 | sean-k-mooney | at the moment we are over spcifying and underspecifying at the same time | |
| 17:14:49 | artom | dansmith, I think placement would want to now know about NUMA | |
| 17:14:57 | artom | And only think of things in terms of generic RPs | |
| 17:14:58 | sean-k-mooney | since the way we express the query does not fully match what we need | |
| 17:15:06 | artom | *to not know | |
| 17:15:19 | dansmith | artom: that's the proposal, AFAIK | |
| 17:15:32 | dansmith | artom: to model this in placement as RPs with the relevant hierarchy | |
| 17:15:47 | artom | dansmith, how does this fit with your "internal placement retry loop" idea though? | |
| 17:15:48 | sean-k-mooney | artom: right but we added some emantic meaning that is not helpful to come concpets in placment. | |
| 17:15:53 | dansmith | artom: ? | |
| 17:16:08 | artom | dansmith, well, wouldn't placement need to understand what a NUMA node is to do that? | |
| 17:16:14 | sean-k-mooney | for example resouces:vcpu=2 must allcaote 2 cpus form the same RP | |
| 17:16:40 | sean-k-mooney | artom: no it just need to know that some resouce need to be in the same sub tree | |
| 17:17:01 | artom | sean-k-mooney, I guess I can see that | |
| 17:17:01 | sean-k-mooney | with a parent contiing a opacue trating "HW_NUMA_ROOT" | |
| 17:17:08 | dansmith | artom: no, I don't think so, we just need to provide some way to communicate to placement that it can satisfy the required resources by allowing them to exist in multiple parts in the tree, with some minimum amount per provider, with maybe matching ratios of cpus and memory | |
| 17:17:56 | dansmith | artom: I don't mean an "&numa_nodes=1-4" level of explicitness (even though I've said that just as an example), but it really just needs to be the generic form of that | |
| 17:18:12 | artom | dansmith, I wonder if we could tell placement something like "aggregate_at_root=true", and then it could sum the child RPs resources temporarily when handling that query | |
| 17:18:26 | dansmith | artom: we need more than that, I think | |
| 17:18:35 | sean-k-mooney | no that is not enouch | |
| 17:18:48 | dansmith | artom: we need to be able to say "don't give me 7 cpus on one node with 1MB of ram, and 1 CPU on the other with 8G" | |
| 17:18:54 | sean-k-mooney | the thing that breaks with simple models is alwasy disk space | |
| 17:19:00 | sean-k-mooney | if i asked for 100 | |
| 17:19:01 | dansmith | artom: something like "cpus and mem must match 60/40 across the split" or something | |
| 17:19:15 | sean-k-mooney | 100G you cant give me 2x50G | |
| 17:19:24 | dansmith | yep | |
| 17:20:01 | sean-k-mooney | so it has to be per resouce class and we need to express grouping constratins and potaentail sizing info like the 64/40 split | |
| 17:20:48 | sean-k-mooney | artom: right now to do ^ we have to specify the toplogy exactly rather then saying these are the limits give me anyting that matches | |
| 17:20:52 | artom | But... the disk would never be on a NUMA child RP... | |
| 17:21:07 | dansmith | artom: he's giving an example of a resource splitting constraint | |
| 17:22:05 | artom | TBH I still don't see why we need to express splitting constraints, but I'm probably just being thick | |
| 17:22:25 | artom | And as sean-k-mooney said, it's an academic discussion not relevant to the current spec | |
| 17:22:27 | sean-k-mooney | same thing applies to hugpegaes. if i ask for 512mb of hugepages and the host only has 1G hugepages allocated you cant split it | |
| 17:22:36 | sean-k-mooney | disk is just more approchable | |
| 17:22:44 | dansmith | because if you ask for 16 CPUs and 32G of RAM, we need to be able to say "we're willing to take that split across two nodes as long as the ratio of the split for those resources is at most 60/40" | |
| 17:22:52 | artom | sean-k-mooney, right, but hugepages we model as their own RP | |
| 17:23:37 | artom | So you'd never get 1GB hugepages anyways, you'd get, for example, 400MB from 1 RP, and 112 from another (unrealistic numbers, I know) | |
| 17:24:30 | artom | dansmith, so that's my thing - why do we want to say that? What's wrong with CPUs split 1/15 and memory 30GB/2GB | |
| 17:24:39 | sean-k-mooney | it would get rejected by the step size yes | |
| 17:24:51 | dansmith | artom: because that would be pretty unhelpful? | |
| 17:25:06 | artom | dansmith, hey, they user they don't care about NUMA topologies ^_^ | |
| 17:25:09 | sean-k-mooney | but point is there are limits on how thing can be split that depned on the resouce class and how it willl be used | |
| 17:25:17 | artom | But seriously, unhelpful from a performance POV? | |
| 17:25:35 | dansmith | artom: right, the user isn't opinionated, but they still want a sane instance | |
| 17:25:55 | dansmith | artom: not caring about numa doesn't mean we should give them something completely pathologically stupid | |
| 17:26:17 | artom | lulz - we need a hw:sanity extra specs | |
| 17:26:27 | artom | And if they set it to ludicrous we do ^^ | |
| 17:26:44 | artom | dansmith, but yeah, I get your point | |
| 17:26:46 | dansmith | if we did this, we'd want to be able to specify what those splits are, and if the op really doesn't care, then they can set the split policy to something very fine | |
| 17:27:03 | dansmith | but it wouldn't make much sense for them to do that | |
| 17:27:18 | artom | I was mostly joking about that extra spec | |
| 17:27:27 | sean-k-mooney | yes we know | |