| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-27 | |||
| 20:38:55 | jaypipes | sorry sean-k-mooney[m], but it's true. | |
| 20:39:34 | sean-k-mooney[m] | jaypipes: haha i know one of our k8s teams sits 10 feet form me | |
| 20:40:02 | jaypipes | connor doyle? | |
| 20:42:07 | sean-k-mooney[m] | jaypipes: am i dont recognise that name but they have been working on multus ant the multi nic support + cpu pinning, hugepages and somthing else | |
| 20:43:40 | cfriesen | jaypipes: sean-k-mooney[m]: in the VCPU and 4KB pages case we currently let the instance float across the whole compute node....I'm of the opinion that we *shouldn't* let it, and *should* limit it to a single host NUMA node. | |
| 20:44:07 | mriedem | i saw in the latest k8s release notes that they now support cpu pinning and huge pages | |
| 20:44:08 | jaypipes | cfriesen: ok. nothing about the proposals would prevent that. | |
| 20:44:08 | efried | cfriesen: Fine by me, but you're going to bounce a lot of spawn requests that way. | |
| 20:44:09 | mriedem | good for them | |
| 20:44:35 | cfriesen | efried: if we don't restrict it, we have no idea how many 4KB pages are left on each host numa node. | |
| 20:44:54 | sean-k-mooney[m] | cfriesen: that breaks existing behavior where it cant fit in one numa node | |
| 20:45:03 | cfriesen | sean-k-mooney[m]: yes. and I think we have no option. | |
| 20:45:25 | efried | cfriesen: except splitting into individual pages, one per request group. | |
| 20:45:34 | efried | or... inventing something new. | |
| 20:46:15 | cfriesen | efried: I'm not sure we can do that with qemu. | |
| 20:46:56 | efried | cfriesen: I'm not talking about qemu splitting. I'm talking about the request being split. Then placement will give you back summed-up allocations per RP. | |
| 20:47:31 | efried | NUMA_0: PAGES=64, NUMA_1: PAGES=1024 or whatever | |
| 20:47:44 | efried | cfriesen: because btw, jaypipes and dansmith came down hard on the idea of placement tracking separate request groups in any way. | |
| 20:47:46 | cfriesen | efried: when you start up qemu and it's allowed to float across the whole host, we do not know how much it will consume from each host numa node | |
| 20:48:00 | sean-k-mooney[m] | jaypipes: so how would you feel about somthing like Resouce:VCPU:use_same_provider:true and then just define that all Rsource provider groups will not overlap with others | |
| 20:48:04 | efried | cfriesen: Oh, you have to let qemu have its head completely? Bogus. | |
| 20:48:43 | cfriesen | efried: at least the way we do it now, yes. for hugepages we map a file and tell it to use that, but for 4KB pages we just say "you're allowed to use up to X memory" | |
| 20:49:01 | sean-k-mooney[m] | efried: well no the said seperate request groups guarenteeed different RPs | |
| 20:49:03 | efried | cfriesen: Then wouldn't the hugepages be inventory on the compute RP? | |
| 20:49:15 | sean-k-mooney[m] | efried: that means the allocation candiates for those request groups will have to be reported sepreately | |
| 20:49:41 | efried | sean-k-mooney[m]: Which I don't agree with. Which was already discussed and decided in the original spec in Q. | |
| 20:49:41 | cfriesen | efried: hugepages are fine, they imply that we're limited to a numa node. instances with "shared" cpus and 4KB pages are allowed to float across the whole compute node currently | |
| 20:49:49 | sean-k-mooney[m] | cfriesen: for 4k we can map a file too if we want | |
| 20:50:12 | cfriesen | sean-k-mooney[m]: can we map multiple files for an instance with a single virtual numa node? | |
| 20:50:53 | sean-k-mooney[m] | cfriesen: we just dont but we can numa afinites 4k pages | |
| 20:51:26 | cfriesen | sean-k-mooney[m]: yes, but can we allocate 3GB of 4K pages from host numa node 0 and 1GB from host numa node 1 for an instance with a single virtual numa node? | |
| 20:52:26 | sean-k-mooney[m] | cfriesen: i think so. why would you want too | |
| 20:52:34 | sean-k-mooney[m] | to allow the memory to come form multiple host numa nodes | |
| 20:53:13 | cfriesen | sean-k-mooney[m]: If I have only 3GB memory free on one numa node and 1GB on the other, and I want to keep the current behaviour of letting the instance float across the whole compute node. | |
| 20:54:09 | jaypipes | sean-k-mooney[m]: again, I think that granular request groups should mean that the resources in each request group are provided by different resource providers. | |
| 20:54:26 | sean-k-mooney[m] | i would have to check. we added the abiltiy to use file desciptor memory by setting the souce elemet of this https://libvirt.org/formatdomain.html#elementsMemoryBacking | |
| 20:54:29 | jaypipes | sean-k-mooney[m]: I do not care about the use case of "general spreadability". | |
| 20:54:32 | efried | jaypipes: but only if they're the same resource class | |
| 20:54:38 | jaypipes | efried: yes. | |
| 20:54:54 | sean-k-mooney[m] | i know that backing file can be affinites but i dont know if we can create two backing files attach to the same guest numa node | |
| 20:55:14 | efried | jaypipes: That's gonna be tough to implement, just for starters. | |
| 20:55:17 | jaypipes | efried: though I don't see a reason why you would separate request groups where one request group does *not* contain a resource class... | |
| 20:55:33 | efried | jaypipes: So that they aren't forced to land on the *same* RP. | |
| 20:55:34 | sean-k-mooney[m] | cfriesen: i think it would work at the qemu level but i need to dig deeper | |
| 20:55:50 | jaypipes | efried: say wha? | |
| 20:56:02 | cfriesen | sean-k-mooney[m]: unless we can do that, we can't let a single-numa-node guest use memory from multiple host numa nodes. | |
| 20:56:05 | sean-k-mooney[m] | jaypipes: ya i actully prefer that each group is a different RP. that is what i had originally assumed | |
| 20:56:32 | jaypipes | efried: I mean, I don't see a use case for doing, for example, this: resources1=VCPU:1,required2=HW_NIC_OFFLOAD_GENEVE | |
| 20:56:33 | efried | jaypipes: It wouldn't make sense for me to say DISK_GB:1024,VF:1. Maybe I misunderstood your statement. | |
| 20:56:40 | jaypipes | efried: that doesn't make sense to me. | |
| 20:57:42 | sean-k-mooney[m] | cfriesen: well thats the thing they are not really singel numa guests | |
| 20:57:45 | efried | jaypipes: resources1=VF:1,BW:200&required1=PHYSNET_A&resources2=VF:1,BW:300&required2=REALLY_FAST | |
| 20:58:06 | sean-k-mooney[m] | they are guest that did no specify a numa topology so there is no reason nova could not make them multi numa guests | |
| 20:58:12 | jaypipes | efried: ok, and? | |
| 20:58:29 | jaypipes | efried: that makes total sense to me. | |
| 20:58:29 | cfriesen | sean-k-mooney[m]: ooh, fun. that won't surprise anyone. :) | |
| 20:58:31 | efried | Contrived example. I don't care which physnet that second VF is on. I just want it to be fast. Why can't it land on the same RP as the first one? | |
| 20:58:56 | cfriesen | sean-k-mooney[m]: don't forget we'd have to preserve the numa topology over live migration | |
| 20:59:15 | sean-k-mooney[m] | cfriesen: if we made them multi numa guest when there resouces were split across host numa nodes it would fix the accounting issue and improve the performance of the guest | |
| 20:59:28 | jaypipes | efried: because it's just crazy to reason about for the user, frankly. | |
| 20:59:29 | sean-k-mooney[m] | cfriesen: your relying on an implentation deatil that i vift-driver specific | |
| 20:59:45 | cfriesen | sean-k-mooney[m]: it could also make the guest slower if it's OS isn't numa-aware. | |
| 21:00:10 | jaypipes | efried: if a user requests two groups of a resource, the user expects those groups to be separate. | |
| 21:00:25 | jaypipes | efried: and "separate" means "provided by different providers" in my and dansmith's book. | |
| 21:00:33 | sean-k-mooney[m] | cfriesen: the guest os | |
| 21:00:46 | cfriesen | sean-k-mooney[m]: yep | |
| 21:00:48 | efried | jaypipes: disagree completely. IMO it's way easier for the user to reason about the groups individually, without regard for their interrelations. | |
| 21:01:06 | sean-k-mooney[m] | cfriesen: how would it make it slow when before it would have been spread on the phyical host but it never would have knonw | |
| 21:01:43 | jaypipes | efried: I'm afraid we'll just have to agree to disagree. | |
| 21:01:48 | efried | jaypipes: Fact is, either way we decide this will leave a hole, cases we can't express. | |
| 21:01:55 | cfriesen | sean-k-mooney[m]: code running in the guest on two separate cpus that wants to share a lot of memory between the two CPUs. now you've got cross-NUMA latencies | |
| 21:01:56 | jaypipes | efried: true. | |
| 21:02:43 | efried | jaypipes: The way it's currently written is simpler, more flexible, and easier to implement; and IMO easier to reason about. But that may just be because it's what I've had in my head since early Q. | |
| 21:03:18 | cfriesen | sean-k-mooney[m]: you're correct that it's no worse than it is now | |
| 21:03:18 | sean-k-mooney[m] | cfriesen: yes and today that can happen too. the guest sees 1 numa node but on the host ist ram can be allocate form multiple because it floats | |
| 21:03:28 | jaypipes | My brain is, frankly, pretty fried. | |
| 21:03:32 | efried | I've thought through all the things you can't express with it, and it boils down to just one thing: you can't express "separate these request groups". | |
| 21:03:41 | jaypipes | I really need to eat and refresh my brain juices. | |
| 21:03:54 | efried | ...but adding that feature would be relatively straightforward with the addition of some new syntax. | |
| 21:04:16 | efried | Yeah, for my part I really ought to go review some more specs that aren't already approved. | |
| 21:06:19 | sean-k-mooney[m] | cfriesen: :) so i was suggesting was if you did not spcify a numa topology then we leave it up to the virt-driver but it then has to reflect the topolgy to the guest. it would be no wose then today but actully fix the 4k pages issue and technically i dont think we are breaking any guartees as i dont hink we define that if numa_nodes is not set its 1 numa node | |
| 21:07:41 | sean-k-mooney[m] | cfriesen: anyway that an issue for another day | |
| 21:11:22 | openstackgerrit | Hongbin Lu proposed openstack/nova-specs master: Choose default network on ambiguity https://review.openstack.org/520247 | |
| 21:11:50 | openstackgerrit | Ed Leafe proposed openstack/nova-specs master: Add Generation to Consumers https://review.openstack.org/556971 | |
| 21:12:09 | edleafe | cdent: efried: jaypipes: ^^ updated with your suggestions | |
| 21:14:38 | efried | edleafe: Check yer whitespace | |
| 21:15:46 | edleafe | efried: ugh, do I have to? <whine> | |
| 21:15:55 | efried | edleafe: But hold, there's another typo | |
| 21:16:02 | efried | edleafe: And another thing that needs to be said. | |
| 21:16:54 | edleafe | well, I wasn't gonna push a rev for whitespace until everyone had a crack at it | |
| 21:20:23 | efried | edleafe: Never mind that last thing. I was gonna ask if the generation in PUT /allocations/{c} was going to quit being ignored. But that's a RP generation, and not really relevant here. | |
| 21:21:04 | efried | edleafe: Consider me cracked. | |
| 21:22:24 | sean-k-mooney[m] | you know that feeling when you find the souce of a bug you hit and you wish you had not... | |
| 21:23:04 | sean-k-mooney[m] | here we convert form the disk bus the use asked for to a prefix https://github.com/openstack/nova/blob/ef0ce4d692d28a7f5a0079e24acdbfe7d2767e8b/nova/virt/libvirt/blockinfo.py#L124-L143 | |
| 21:23:15 | cdent | cracked | |
| 21:23:23 | sean-k-mooney[m] | here we convert form the prefix to the disk bus https://github.com/openstack/nova/blob/ef0ce4d692d28a7f5a0079e24acdbfe7d2767e8b/nova/virt/libvirt/blockinfo.py#L297-L304 | |
| 21:23:54 | sean-k-mooney[m] | we map sata scis and usb to sd then hard code sd = scsi | |
| 21:24:40 | edleafe | efried: I always have | |
| 21:24:40 | sean-k-mooney[m] | that mean on kvm you can request sata or usb disk which is why i had to go to IDE to fix my centos image | |
| 21:25:10 | sean-k-mooney[m] | *cant request | |