| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-27 | |||
| 20:20:00 | efried | dansmith: Now we can spread 'em any which way they fit. | |
| 20:20:04 | dansmith | efried: you can't be serious | |
| 20:20:32 | dansmith | efried: and if I want 32G of memory allocated anywhere? 32*24 query params? :) | |
| 20:20:44 | jaypipes | sean-k-mooney[m]: then you would request VCPU=16 | |
| 20:20:44 | sean-k-mooney[m] | jaypipes: so non granular can be form multiple RPs and granular resuest each request is guarteeed to be form differnt RPs | |
| 20:20:57 | efried | dansmith: I imagine step sizes come into play there | |
| 20:21:07 | jaypipes | sean-k-mooney[m]: yes. | |
| 20:21:12 | dansmith | efried: okay but that's still a damn lot of params for silly reasons | |
| 20:21:28 | dansmith | tbh, I'm kinda frustrated by this conversation at this stage in the game and I'm late leaving for something else | |
| 20:21:30 | efried | dansmith: How else do you ensure generic spreadability? Is that a silly reason? | |
| 20:21:44 | efried | Okay. Y'all have made your position clear. | |
| 20:21:50 | dansmith | so I shall bow out, but jaypipes I agree about that third bullet change, or at least putting a stake in the ground about needing to revisit it | |
| 20:22:02 | dansmith | back later | |
| 20:22:16 | jaypipes | efried: what is generic spreadability? is that like I can't believe it's not butter? | |
| 20:22:40 | efried | Oh, you better believe it. | |
| 20:22:43 | jaypipes | heh | |
| 20:24:49 | efried | jaypipes: It means, "I know I want 4 VCPUs. I don't care about NUMA affinity. I just want my instance." Some hosts have no NUMA (all VCPU on the compute RP). Some have two NUMA nodes, some have four. It's a fullish cloud. I sure would like it if I could land my instance, whether there's a host with 4 VCPUs in one NUMA node, or one with 3,1, or one with 2,1,1,0, etc. | |
| 20:25:35 | sean-k-mooney[m] | jaypipes: so just to confim. non granuarl resouce resutes can from from multiple RP. each Granular resouce request of the same resouce class is guarenteed to come form different RPs and Granular Resouce request can or cannot spread across multiple RP im assuming Cannot? | |
| 20:25:37 | jaypipes | efried: resources:VCPU=4 | |
| 20:25:40 | jaypipes | simply as that. | |
| 20:25:50 | efried | jaypipes: Then we're treating VCPU special. | |
| 20:25:58 | efried | jaypipes: Cause you definitely can't treat DISK_GB=1024 the same way. | |
| 20:26:32 | cdent | efried: are you saying in the case where the vcpu inventory is registered on the numa nested provider, or something else? | |
| 20:26:40 | jaypipes | sean-k-mooney[m]: a single granular request group of an amount of a resource class cannot spread that bucket across multiple providers, no. | |
| 20:26:44 | efried | sean-k-mooney[m]: As currently designed [1], separate request groups with the same resource class *may* or *may not* come from the same RP. | |
| 20:26:44 | efried | [1] http://specs.openstack.org/openstack/nova-specs/specs/rocky/approved/granular-resource-requests.html#semantics | |
| 20:26:55 | sean-k-mooney[m] | efried: that would be covered by just Resource:VCPU=4 | |
| 20:26:59 | efried | sean-k-mooney[m]: Oh, also what jaypipes says. | |
| 20:27:05 | efried | sean-k-mooney[m]: No - see above about DISK_GB. | |
| 20:27:06 | jaypipes | efried: how so? all of those resource providers have 4 VCPU available. | |
| 20:27:27 | sean-k-mooney[m] | efried: why? | |
| 20:27:27 | efried | jaypipes: No, 3,1 means "3 in NUMA node 0, 1 in NUMA node 1" etc. | |
| 20:28:01 | efried | sean-k-mooney[m]: If VCPU=4 means I can spread VCPU across multiple numa node RPs, then DISK_GB=1024 means I can spread... individual gigabytes? across multiple storage RPs. | |
| 20:28:08 | jaypipes | oh, in that case then no, no providers would be returned (since none have 4 VCPU available) | |
| 20:28:15 | jaypipes | efried: ^ | |
| 20:28:22 | jaypipes | efried: and that is A-OK in my book. | |
| 20:28:30 | efried | jaypipes: Exactly. I get NoValidHosts. But there *were* hosts with available procs. | |
| 20:28:40 | jaypipes | no there were not. | |
| 20:28:50 | jaypipes | efried: there were no resource providers that had 4 VCPU. | |
| 20:29:03 | efried | But there were *hosts* that had 4 VCPu. | |
| 20:29:20 | jaypipes | efried: the host isn't the provider of the VCPU in your scenario, though. | |
| 20:29:25 | efried | why not just say you can't have the same RC in different RPs then? | |
| 20:30:04 | sean-k-mooney[m] | jaypipes: well in his case there were 3cpus on 1 numa node an 1 on the second | |
| 20:30:08 | efried | jaypipes: But I didn't ask for my instance to land on a RP. I asked for it to land on a host. Which has a tree of RPs. But I don't know (and shouldn't care in this case) where the VCPUs are in that tree. | |
| 20:30:26 | jaypipes | sean-k-mooney[m]: exactly. there were no providers that had 4 VCPU available. | |
| 20:30:26 | sean-k-mooney[m] | but there was not host with 1 RP with 4 VCPUS | |
| 20:31:14 | efried | tbc, I'm agreeing that VCPU=4 should give you no hits in this case. | |
| 20:31:34 | efried | I'm saying we need a way to express a request that *does* land this instance. | |
| 20:31:51 | jaypipes | efried: I pretty specifically remember you telling me that my proposed "sum the inventories for like resource classes within a provider tree" was absolutely the wrong way to handle nested providers. | |
| 20:31:56 | sean-k-mooney[m] | jaypipes: right so today. before placement i can have a host with 2 socket each with 10 cores and i can boot a vm with no numa topology with 16 cores | |
| 20:32:02 | jaypipes | efried: and I'm saying I don't care about that. | |
| 20:32:14 | efried | jaypipes: Yes, exactly, because the DISK_GB case breaks it unequivocally. | |
| 20:32:14 | sean-k-mooney[m] | that would now break | |
| 20:32:26 | efried | ^^ this. | |
| 20:32:51 | jaypipes | I really don't care. | |
| 20:32:51 | efried | With the granular syntax as designed, we have a way to ask for this ^ that will work in *any* scenario. | |
| 20:33:10 | efried | namely: resources1=VCPU:1,...,resources16=VCPU:1 | |
| 20:33:13 | sean-k-mooney[m] | efried: well diskGB only breaks in some cases | |
| 20:33:24 | efried | I'll grant you that I don't want the admin/operator to have to say that. | |
| 20:33:28 | jaypipes | efried: that is just over-engineering IMHO. | |
| 20:33:47 | jaypipes | for a use case that just isn't particularly attractive to me. | |
| 20:33:53 | openstackgerrit | Merged openstack/nova-specs master: tox.ini: remove the stale 'minversion = 1.4' https://review.openstack.org/530776 | |
| 20:34:46 | sean-k-mooney[m] | efried: that has the opisite problem | |
| 20:35:04 | efried | sean-k-mooney[m]: Not if separate granular groups can land on the same RP. | |
| 20:35:09 | sean-k-mooney[m] | now i cant say thes have to come from different RPs | |
| 20:35:15 | efried | correct. | |
| 20:35:20 | efried | without traits. | |
| 20:36:50 | sean-k-mooney[m] | efried: even with traits | |
| 20:36:58 | sean-k-mooney[m] | traits would artifically nanorow your selection | |
| 20:37:08 | sean-k-mooney[m] | you basically need a way to express "use_same_provider=True/False" relationships between Resouce1... | |
| 20:37:23 | sean-k-mooney[m] | e.g. Resouce1,Resouce2:use_same_provider=True | |
| 20:38:03 | efried | sean-k-mooney[m]: Agree we don't want to restrict artificially via traits. Which is why that's not a solution, just a side effect. | |
| 20:38:06 | jaypipes | oh wait, no, no they don't. at all. | |
| 20:38:24 | efried | jaypipes: kubernetes is just a babe. Give it a couple of years. | |
| 20:38:38 | efried | It'll either be dead, or supporting this level of crazy. | |
| 20:38:48 | jaypipes | efried: intel has been trying desperately for 2+ years to add this level of crazy to resource management in k8s. | |
| 20:38:51 | sean-k-mooney[m] | jaypipes: not yet. give intel time :P | |
| 20:38:55 | jaypipes | sorry sean-k-mooney[m], but it's true. | |
| 20:39:34 | sean-k-mooney[m] | jaypipes: haha i know one of our k8s teams sits 10 feet form me | |
| 20:40:02 | jaypipes | connor doyle? | |
| 20:42:07 | sean-k-mooney[m] | jaypipes: am i dont recognise that name but they have been working on multus ant the multi nic support + cpu pinning, hugepages and somthing else | |
| 20:43:40 | cfriesen | jaypipes: sean-k-mooney[m]: in the VCPU and 4KB pages case we currently let the instance float across the whole compute node....I'm of the opinion that we *shouldn't* let it, and *should* limit it to a single host NUMA node. | |
| 20:44:07 | mriedem | i saw in the latest k8s release notes that they now support cpu pinning and huge pages | |
| 20:44:08 | jaypipes | cfriesen: ok. nothing about the proposals would prevent that. | |
| 20:44:08 | efried | cfriesen: Fine by me, but you're going to bounce a lot of spawn requests that way. | |
| 20:44:09 | mriedem | good for them | |
| 20:44:35 | cfriesen | efried: if we don't restrict it, we have no idea how many 4KB pages are left on each host numa node. | |
| 20:44:54 | sean-k-mooney[m] | cfriesen: that breaks existing behavior where it cant fit in one numa node | |
| 20:45:03 | cfriesen | sean-k-mooney[m]: yes. and I think we have no option. | |
| 20:45:25 | efried | cfriesen: except splitting into individual pages, one per request group. | |
| 20:45:34 | efried | or... inventing something new. | |
| 20:46:15 | cfriesen | efried: I'm not sure we can do that with qemu. | |
| 20:46:56 | efried | cfriesen: I'm not talking about qemu splitting. I'm talking about the request being split. Then placement will give you back summed-up allocations per RP. | |
| 20:47:31 | efried | NUMA_0: PAGES=64, NUMA_1: PAGES=1024 or whatever | |
| 20:47:44 | efried | cfriesen: because btw, jaypipes and dansmith came down hard on the idea of placement tracking separate request groups in any way. | |
| 20:47:46 | cfriesen | efried: when you start up qemu and it's allowed to float across the whole host, we do not know how much it will consume from each host numa node | |
| 20:48:00 | sean-k-mooney[m] | jaypipes: so how would you feel about somthing like Resouce:VCPU:use_same_provider:true and then just define that all Rsource provider groups will not overlap with others | |
| 20:48:04 | efried | cfriesen: Oh, you have to let qemu have its head completely? Bogus. | |
| 20:48:43 | cfriesen | efried: at least the way we do it now, yes. for hugepages we map a file and tell it to use that, but for 4KB pages we just say "you're allowed to use up to X memory" | |
| 20:49:01 | sean-k-mooney[m] | efried: well no the said seperate request groups guarenteeed different RPs | |