| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-27 | |||
| 20:28:08 | jaypipes | oh, in that case then no, no providers would be returned (since none have 4 VCPU available) | |
| 20:28:15 | jaypipes | efried: ^ | |
| 20:28:22 | jaypipes | efried: and that is A-OK in my book. | |
| 20:28:30 | efried | jaypipes: Exactly. I get NoValidHosts. But there *were* hosts with available procs. | |
| 20:28:40 | jaypipes | no there were not. | |
| 20:28:50 | jaypipes | efried: there were no resource providers that had 4 VCPU. | |
| 20:29:03 | efried | But there were *hosts* that had 4 VCPu. | |
| 20:29:20 | jaypipes | efried: the host isn't the provider of the VCPU in your scenario, though. | |
| 20:29:25 | efried | why not just say you can't have the same RC in different RPs then? | |
| 20:30:04 | sean-k-mooney[m] | jaypipes: well in his case there were 3cpus on 1 numa node an 1 on the second | |
| 20:30:08 | efried | jaypipes: But I didn't ask for my instance to land on a RP. I asked for it to land on a host. Which has a tree of RPs. But I don't know (and shouldn't care in this case) where the VCPUs are in that tree. | |
| 20:30:26 | sean-k-mooney[m] | but there was not host with 1 RP with 4 VCPUS | |
| 20:30:26 | jaypipes | sean-k-mooney[m]: exactly. there were no providers that had 4 VCPU available. | |
| 20:31:14 | efried | tbc, I'm agreeing that VCPU=4 should give you no hits in this case. | |
| 20:31:34 | efried | I'm saying we need a way to express a request that *does* land this instance. | |
| 20:31:51 | jaypipes | efried: I pretty specifically remember you telling me that my proposed "sum the inventories for like resource classes within a provider tree" was absolutely the wrong way to handle nested providers. | |
| 20:31:56 | sean-k-mooney[m] | jaypipes: right so today. before placement i can have a host with 2 socket each with 10 cores and i can boot a vm with no numa topology with 16 cores | |
| 20:32:02 | jaypipes | efried: and I'm saying I don't care about that. | |
| 20:32:14 | sean-k-mooney[m] | that would now break | |
| 20:32:14 | efried | jaypipes: Yes, exactly, because the DISK_GB case breaks it unequivocally. | |
| 20:32:26 | efried | ^^ this. | |
| 20:32:51 | efried | With the granular syntax as designed, we have a way to ask for this ^ that will work in *any* scenario. | |
| 20:32:51 | jaypipes | I really don't care. | |
| 20:33:10 | efried | namely: resources1=VCPU:1,...,resources16=VCPU:1 | |
| 20:33:13 | sean-k-mooney[m] | efried: well diskGB only breaks in some cases | |
| 20:33:24 | efried | I'll grant you that I don't want the admin/operator to have to say that. | |
| 20:33:28 | jaypipes | efried: that is just over-engineering IMHO. | |
| 20:33:47 | jaypipes | for a use case that just isn't particularly attractive to me. | |
| 20:33:53 | openstackgerrit | Merged openstack/nova-specs master: tox.ini: remove the stale 'minversion = 1.4' https://review.openstack.org/530776 | |
| 20:34:46 | sean-k-mooney[m] | efried: that has the opisite problem | |
| 20:35:04 | efried | sean-k-mooney[m]: Not if separate granular groups can land on the same RP. | |
| 20:35:09 | sean-k-mooney[m] | now i cant say thes have to come from different RPs | |
| 20:35:15 | efried | correct. | |
| 20:35:20 | efried | without traits. | |
| 20:36:50 | sean-k-mooney[m] | efried: even with traits | |
| 20:36:58 | sean-k-mooney[m] | traits would artifically nanorow your selection | |
| 20:37:08 | sean-k-mooney[m] | you basically need a way to express "use_same_provider=True/False" relationships between Resouce1... | |
| 20:37:23 | sean-k-mooney[m] | e.g. Resouce1,Resouce2:use_same_provider=True | |
| 20:38:03 | efried | sean-k-mooney[m]: Agree we don't want to restrict artificially via traits. Which is why that's not a solution, just a side effect. | |
| 20:38:06 | jaypipes | oh wait, no, no they don't. at all. | |
| 20:38:24 | efried | jaypipes: kubernetes is just a babe. Give it a couple of years. | |
| 20:38:38 | efried | It'll either be dead, or supporting this level of crazy. | |
| 20:38:48 | jaypipes | efried: intel has been trying desperately for 2+ years to add this level of crazy to resource management in k8s. | |
| 20:38:51 | sean-k-mooney[m] | jaypipes: not yet. give intel time :P | |
| 20:38:55 | jaypipes | sorry sean-k-mooney[m], but it's true. | |
| 20:39:34 | sean-k-mooney[m] | jaypipes: haha i know one of our k8s teams sits 10 feet form me | |
| 20:40:02 | jaypipes | connor doyle? | |
| 20:42:07 | sean-k-mooney[m] | jaypipes: am i dont recognise that name but they have been working on multus ant the multi nic support + cpu pinning, hugepages and somthing else | |
| 20:43:40 | cfriesen | jaypipes: sean-k-mooney[m]: in the VCPU and 4KB pages case we currently let the instance float across the whole compute node....I'm of the opinion that we *shouldn't* let it, and *should* limit it to a single host NUMA node. | |
| 20:44:07 | mriedem | i saw in the latest k8s release notes that they now support cpu pinning and huge pages | |
| 20:44:08 | efried | cfriesen: Fine by me, but you're going to bounce a lot of spawn requests that way. | |
| 20:44:08 | jaypipes | cfriesen: ok. nothing about the proposals would prevent that. | |
| 20:44:09 | mriedem | good for them | |
| 20:44:35 | cfriesen | efried: if we don't restrict it, we have no idea how many 4KB pages are left on each host numa node. | |
| 20:44:54 | sean-k-mooney[m] | cfriesen: that breaks existing behavior where it cant fit in one numa node | |
| 20:45:03 | cfriesen | sean-k-mooney[m]: yes. and I think we have no option. | |
| 20:45:25 | efried | cfriesen: except splitting into individual pages, one per request group. | |
| 20:45:34 | efried | or... inventing something new. | |
| 20:46:15 | cfriesen | efried: I'm not sure we can do that with qemu. | |
| 20:46:56 | efried | cfriesen: I'm not talking about qemu splitting. I'm talking about the request being split. Then placement will give you back summed-up allocations per RP. | |
| 20:47:31 | efried | NUMA_0: PAGES=64, NUMA_1: PAGES=1024 or whatever | |
| 20:47:44 | efried | cfriesen: because btw, jaypipes and dansmith came down hard on the idea of placement tracking separate request groups in any way. | |
| 20:47:46 | cfriesen | efried: when you start up qemu and it's allowed to float across the whole host, we do not know how much it will consume from each host numa node | |
| 20:48:00 | sean-k-mooney[m] | jaypipes: so how would you feel about somthing like Resouce:VCPU:use_same_provider:true and then just define that all Rsource provider groups will not overlap with others | |
| 20:48:04 | efried | cfriesen: Oh, you have to let qemu have its head completely? Bogus. | |
| 20:48:43 | cfriesen | efried: at least the way we do it now, yes. for hugepages we map a file and tell it to use that, but for 4KB pages we just say "you're allowed to use up to X memory" | |
| 20:49:01 | sean-k-mooney[m] | efried: well no the said seperate request groups guarenteeed different RPs | |
| 20:49:03 | efried | cfriesen: Then wouldn't the hugepages be inventory on the compute RP? | |
| 20:49:15 | sean-k-mooney[m] | efried: that means the allocation candiates for those request groups will have to be reported sepreately | |
| 20:49:41 | cfriesen | efried: hugepages are fine, they imply that we're limited to a numa node. instances with "shared" cpus and 4KB pages are allowed to float across the whole compute node currently | |
| 20:49:41 | efried | sean-k-mooney[m]: Which I don't agree with. Which was already discussed and decided in the original spec in Q. | |
| 20:49:49 | sean-k-mooney[m] | cfriesen: for 4k we can map a file too if we want | |
| 20:50:12 | cfriesen | sean-k-mooney[m]: can we map multiple files for an instance with a single virtual numa node? | |
| 20:50:53 | sean-k-mooney[m] | cfriesen: we just dont but we can numa afinites 4k pages | |
| 20:51:26 | cfriesen | sean-k-mooney[m]: yes, but can we allocate 3GB of 4K pages from host numa node 0 and 1GB from host numa node 1 for an instance with a single virtual numa node? | |
| 20:52:26 | sean-k-mooney[m] | cfriesen: i think so. why would you want too | |
| 20:52:34 | sean-k-mooney[m] | to allow the memory to come form multiple host numa nodes | |
| 20:53:13 | cfriesen | sean-k-mooney[m]: If I have only 3GB memory free on one numa node and 1GB on the other, and I want to keep the current behaviour of letting the instance float across the whole compute node. | |
| 20:54:09 | jaypipes | sean-k-mooney[m]: again, I think that granular request groups should mean that the resources in each request group are provided by different resource providers. | |
| 20:54:26 | sean-k-mooney[m] | i would have to check. we added the abiltiy to use file desciptor memory by setting the souce elemet of this https://libvirt.org/formatdomain.html#elementsMemoryBacking | |
| 20:54:29 | jaypipes | sean-k-mooney[m]: I do not care about the use case of "general spreadability". | |
| 20:54:32 | efried | jaypipes: but only if they're the same resource class | |
| 20:54:38 | jaypipes | efried: yes. | |
| 20:54:54 | sean-k-mooney[m] | i know that backing file can be affinites but i dont know if we can create two backing files attach to the same guest numa node | |
| 20:55:14 | efried | jaypipes: That's gonna be tough to implement, just for starters. | |
| 20:55:17 | jaypipes | efried: though I don't see a reason why you would separate request groups where one request group does *not* contain a resource class... | |
| 20:55:33 | efried | jaypipes: So that they aren't forced to land on the *same* RP. | |
| 20:55:34 | sean-k-mooney[m] | cfriesen: i think it would work at the qemu level but i need to dig deeper | |
| 20:55:50 | jaypipes | efried: say wha? | |
| 20:56:02 | cfriesen | sean-k-mooney[m]: unless we can do that, we can't let a single-numa-node guest use memory from multiple host numa nodes. | |
| 20:56:05 | sean-k-mooney[m] | jaypipes: ya i actully prefer that each group is a different RP. that is what i had originally assumed | |
| 20:56:32 | jaypipes | efried: I mean, I don't see a use case for doing, for example, this: resources1=VCPU:1,required2=HW_NIC_OFFLOAD_GENEVE | |
| 20:56:33 | efried | jaypipes: It wouldn't make sense for me to say DISK_GB:1024,VF:1. Maybe I misunderstood your statement. | |
| 20:56:40 | jaypipes | efried: that doesn't make sense to me. | |
| 20:57:42 | sean-k-mooney[m] | cfriesen: well thats the thing they are not really singel numa guests | |
| 20:57:45 | efried | jaypipes: resources1=VF:1,BW:200&required1=PHYSNET_A&resources2=VF:1,BW:300&required2=REALLY_FAST | |
| 20:58:06 | sean-k-mooney[m] | they are guest that did no specify a numa topology so there is no reason nova could not make them multi numa guests | |
| 20:58:12 | jaypipes | efried: ok, and? | |
| 20:58:29 | cfriesen | sean-k-mooney[m]: ooh, fun. that won't surprise anyone. :) | |
| 20:58:29 | jaypipes | efried: that makes total sense to me. | |