| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-02-11 | |||
| 17:15:19 | dansmith | artom: that's the proposal, AFAIK | |
| 17:15:32 | dansmith | artom: to model this in placement as RPs with the relevant hierarchy | |
| 17:15:47 | artom | dansmith, how does this fit with your "internal placement retry loop" idea though? | |
| 17:15:48 | sean-k-mooney | artom: right but we added some emantic meaning that is not helpful to come concpets in placment. | |
| 17:15:53 | dansmith | artom: ? | |
| 17:16:08 | artom | dansmith, well, wouldn't placement need to understand what a NUMA node is to do that? | |
| 17:16:14 | sean-k-mooney | for example resouces:vcpu=2 must allcaote 2 cpus form the same RP | |
| 17:16:40 | sean-k-mooney | artom: no it just need to know that some resouce need to be in the same sub tree | |
| 17:17:01 | artom | sean-k-mooney, I guess I can see that | |
| 17:17:01 | sean-k-mooney | with a parent contiing a opacue trating "HW_NUMA_ROOT" | |
| 17:17:08 | dansmith | artom: no, I don't think so, we just need to provide some way to communicate to placement that it can satisfy the required resources by allowing them to exist in multiple parts in the tree, with some minimum amount per provider, with maybe matching ratios of cpus and memory | |
| 17:17:56 | dansmith | artom: I don't mean an "&numa_nodes=1-4" level of explicitness (even though I've said that just as an example), but it really just needs to be the generic form of that | |
| 17:18:12 | artom | dansmith, I wonder if we could tell placement something like "aggregate_at_root=true", and then it could sum the child RPs resources temporarily when handling that query | |
| 17:18:26 | dansmith | artom: we need more than that, I think | |
| 17:18:35 | sean-k-mooney | no that is not enouch | |
| 17:18:48 | dansmith | artom: we need to be able to say "don't give me 7 cpus on one node with 1MB of ram, and 1 CPU on the other with 8G" | |
| 17:18:54 | sean-k-mooney | the thing that breaks with simple models is alwasy disk space | |
| 17:19:00 | sean-k-mooney | if i asked for 100 | |
| 17:19:01 | dansmith | artom: something like "cpus and mem must match 60/40 across the split" or something | |
| 17:19:15 | sean-k-mooney | 100G you cant give me 2x50G | |
| 17:19:24 | dansmith | yep | |
| 17:20:01 | sean-k-mooney | so it has to be per resouce class and we need to express grouping constratins and potaentail sizing info like the 64/40 split | |
| 17:20:48 | sean-k-mooney | artom: right now to do ^ we have to specify the toplogy exactly rather then saying these are the limits give me anyting that matches | |
| 17:20:52 | artom | But... the disk would never be on a NUMA child RP... | |
| 17:21:07 | dansmith | artom: he's giving an example of a resource splitting constraint | |
| 17:22:05 | artom | TBH I still don't see why we need to express splitting constraints, but I'm probably just being thick | |
| 17:22:25 | artom | And as sean-k-mooney said, it's an academic discussion not relevant to the current spec | |
| 17:22:27 | sean-k-mooney | same thing applies to hugpegaes. if i ask for 512mb of hugepages and the host only has 1G hugepages allocated you cant split it | |
| 17:22:36 | sean-k-mooney | disk is just more approchable | |
| 17:22:44 | dansmith | because if you ask for 16 CPUs and 32G of RAM, we need to be able to say "we're willing to take that split across two nodes as long as the ratio of the split for those resources is at most 60/40" | |
| 17:22:52 | artom | sean-k-mooney, right, but hugepages we model as their own RP | |
| 17:23:37 | artom | So you'd never get 1GB hugepages anyways, you'd get, for example, 400MB from 1 RP, and 112 from another (unrealistic numbers, I know) | |
| 17:24:30 | artom | dansmith, so that's my thing - why do we want to say that? What's wrong with CPUs split 1/15 and memory 30GB/2GB | |
| 17:24:39 | sean-k-mooney | it would get rejected by the step size yes | |
| 17:24:51 | dansmith | artom: because that would be pretty unhelpful? | |
| 17:25:06 | artom | dansmith, hey, they user they don't care about NUMA topologies ^_^ | |
| 17:25:09 | sean-k-mooney | but point is there are limits on how thing can be split that depned on the resouce class and how it willl be used | |
| 17:25:17 | artom | But seriously, unhelpful from a performance POV? | |
| 17:25:35 | dansmith | artom: right, the user isn't opinionated, but they still want a sane instance | |
| 17:25:55 | dansmith | artom: not caring about numa doesn't mean we should give them something completely pathologically stupid | |
| 17:26:17 | artom | lulz - we need a hw:sanity extra specs | |
| 17:26:27 | artom | And if they set it to ludicrous we do ^^ | |
| 17:26:44 | artom | dansmith, but yeah, I get your point | |
| 17:26:46 | dansmith | if we did this, we'd want to be able to specify what those splits are, and if the op really doesn't care, then they can set the split policy to something very fine | |
| 17:27:03 | dansmith | but it wouldn't make much sense for them to do that | |
| 17:27:18 | artom | I was mostly joking about that extra spec | |
| 17:27:27 | sean-k-mooney | yes we know | |
| 17:28:32 | sean-k-mooney | anyway form my view point if you dont set hw:numa_nodes at all it gives nova the freedome to do something sane | |
| 17:28:49 | sean-k-mooney | that can be create a numa toplogy if it chooese | |
| 17:29:24 | sean-k-mooney | for now we leave libvirt invent a single numa node with no affinity | |
| 17:30:36 | sean-k-mooney | haveing one or multiple numa nodes in the guest and mapping them to 1 or more numa node on the host are two different thing | |
| 17:31:00 | sean-k-mooney | so can if we want expose 1 numa node to the guest and on the host map it across them | |
| 17:31:51 | sean-k-mooney | i think we can do better then that howwever. anyway time to go review something else | |
| 17:51:21 | stephenfin | ah, crap. Today was supposed to be spec review day | |
| 17:51:30 | stephenfin | Guess tomorrow is spec review day for me now \o/ | |
| 17:53:37 | yoctozepto | stephenfin: the most busy today - tomorrow :-) | |
| 17:57:00 | stephenfin | sean-k-mooney: Tempest is failing on my extra spec validation patch because it's using generic e.g. 'key1=value1' extra specs. What's the most generic extra spec we've got? | |
| 17:57:14 | stephenfin | I've been using 'hw:numa_nodes' but that's libvirt/HyperV specific | |
| 17:57:50 | spatel | sean-k-mooney: sorry i was in meeting | |
| 17:59:58 | spatel | sean-k-mooney: as per your recommendation i have added cpu_threads=2 and cpu_sockets=2 in flavor and run test but result was OK (compare to 16 vCPU with single numa0) | |
| 18:00:55 | spatel | still i don't understand why, erlang correctly detected CPUTopology on VM but still result was poor with 28vCPU | |
| 18:11:29 | openstackgerrit | Lee Yarwood proposed openstack/nova master: images: Move qemu-img info calls into privsep https://review.opendev.org/706897 | |
| 18:11:30 | openstackgerrit | Lee Yarwood proposed openstack/nova master: virt: Pass request context to extend_volume https://review.opendev.org/706899 | |
| 18:11:30 | openstackgerrit | Lee Yarwood proposed openstack/nova master: images: Allow the output format of qemu-img info to be controlled https://review.opendev.org/706898 | |
| 18:11:31 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP libvirt: Fix attached encrypted LUKSv1 volume extension https://review.opendev.org/706900 | |
| 18:13:11 | openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: api: Add support for extra spec validation https://review.opendev.org/704643 | |
| 18:13:12 | openstackgerrit | Stephen Finucane proposed openstack/nova master: conf: Deprecate '[scheduler] driver' https://review.opendev.org/707225 | |
| 18:13:12 | openstackgerrit | Stephen Finucane proposed openstack/nova master: trivial: Remove FakeScheduler https://review.opendev.org/707224 | |
| 18:13:13 | openstackgerrit | Stephen Finucane proposed openstack/nova master: doc: Improve documentation on writing custom scheduler filters https://review.opendev.org/707226 | |
| 18:13:42 | stephenfin | sean-k-mooney: I went with 'hw:numa_nodes' and 'hw:cpu_policy' for want of something better | |
| 18:44:45 | efried | sean-k-mooney, bauzas, gibi: I think I would actually prefer the approach dansmith suggests, even if we only do it at the coarsest level, rather than effectively disable the topology modeling in ussuri. | |
| 18:49:49 | efried | - Flavors without hw:numa*-isms get looped for $n in range(0, $max_sane_number_of_numa_nodes_we_expect_a_host_to_ever_have) to behave as if they had specified hw:numa_nodes=$n, stopping as soon as we get a hit. | |
| 18:49:49 | efried | - Flavors with hw:numa*-isms get translated as specced. | |
| 18:49:49 | efried | - Model with NUMA topology by default. Provide the [workaround] option to *disable* (and un-reshape) for situations where the following is just sh*tting all over itself and nothing is landing. | |
| 18:49:49 | efried | in other words: | |
| 18:50:58 | efried | This is without changing anything in placement. | |
| 18:52:44 | efried | For uneven splits, we just get as close as we can, but make no attempt to be "fuzzy" (like "up to 60/40" or anything like that). So like, for $n=2 and VCPU=10, we try 10, then 5/5, then 3/3/4, then 2/3/2/3. | |
| 18:56:04 | dansmith | there has to be some sort of policy tunable for how wild we're able to get I think | |
| 18:56:29 | efried | max_numa_nodes_guessed ? | |
| 18:57:23 | efried | Are there really hosts out there with more than, say, 8 NUMA cells? | |
| 18:57:27 | dansmith | no, I mean how small of a footprint on a given node we're going to allow | |
| 18:58:34 | efried | To be clear, I'm talking about splitting as evenly as possible, always. | |
| 18:58:45 | efried | You're saying that sometimes that would result in unreasonably small footprint on one numa node anyway? | |
| 18:59:05 | efried | obv we stop trying to split if any of the cells are going to get 0 of anything. | |
| 18:59:21 | efried | so if you ask for 2 vcpu, we go to a max of hw:numa_nodes=2 | |
| 18:59:45 | dansmith | sure, but I think we need to not be willing to split memory at 1MB and 8191MB | |
| 18:59:55 | dansmith | or 31 cpus and 1 | |
| 19:00:05 | dansmith | and the ratio needs to be close/equal for cpus and memory | |
| 19:00:08 | efried | Right, I'm saying that happens implicitly | |
| 19:00:25 | dansmith | how? | |
| 19:00:50 | efried | each iteration of the loop simply behaves as if you said hw:numa_nodes=$n. | |
| 19:01:07 | efried | which simply splits your CPU and mem "evenly" across $n nodes. | |
| 19:01:15 | dansmith | that just seems to naive to me | |
| 19:01:19 | efried | yes | |
| 19:01:20 | efried | it is | |
| 19:01:32 | efried | but so is "I don't care about NUMA"? | |
| 19:01:41 | efried | And it ought to work 99% of the time | |
| 19:01:52 | efried | and if it doesn't, switch on your [workaround] for a couple of hosts. | |
| 19:02:07 | dansmith | I want the workaround to go away, remember | |
| 19:02:23 | efried | Yes, the workaround goes away completely once we beef up placement to understand can_split | |