| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-01-29 | |||
| 18:09:09 | sean-k-mooney | stephenfin: im thinking 3 layers like this | |
| 18:09:10 | sean-k-mooney | https://etherpad.openstack.org/p/mem_page_size_and_placement | |
| 18:14:39 | sean-k-mooney | efried: bauzas ^ | |
| 18:14:41 | efried | sean-k-mooney: step_size=4 doesn't make sense unless we use MEMORY_KB | |
| 18:15:04 | sean-k-mooney | well actuly i gues it would be 1024 | |
| 18:15:20 | sean-k-mooney | since we limit flavor to 1mb granuarity | |
| 18:15:39 | efried | tbc, it's not really a problem to introduce new RCs for this, since we're doing the translation under the covers and only allowing NUMA-modeled VMs on NUMA-modeled hosts. | |
| 18:15:46 | sean-k-mooney | its only there becasue you can say mem_page_size=4 or 4k today | |
| 18:15:53 | efried | okay, that's fine. | |
| 18:16:33 | efried | The three-tiered approach works IF you always get exactly one page size | |
| 18:16:35 | sean-k-mooney | this will allow all the sentinels to wrok and we can remove the hugepage page tracking from the numa toplogy filter/resouce tracker | |
| 18:16:51 | sean-k-mooney | efried: yes today we only allow 1 page size | |
| 18:17:13 | sean-k-mooney | so if we dont enable more flexiblity then today we can make that assumtion a requirement | |
| 18:18:39 | efried | so if I have three different hugepage sizes on the same numa cell, say 1G, 2G, 4G, then memory=8G,page_size=large will only ever get me one of | |
| 18:18:40 | efried | [2 x 4G] | |
| 18:18:40 | efried | [4 x 2G] | |
| 18:18:40 | efried | [8 x 1G] | |
| 18:18:40 | efried | but never e.g. [1 x 4G] + [4 x 1G] | |
| 18:18:47 | efried | right? | |
| 18:19:03 | sean-k-mooney | yes | |
| 18:19:10 | efried | cool, then this works, I like it. | |
| 18:19:13 | sean-k-mooney | libvirt support mixing and we intentionaly do not | |
| 18:19:49 | sean-k-mooney | i propsoed this in the past and the main push back is an extra layer adds well an extra layer | |
| 18:19:49 | efried | shall I write it up in the spec comments? | |
| 18:20:06 | efried | yeah, the layer doesn't bother me. It's totally abstracted from the user. | |
| 18:20:09 | sean-k-mooney | sure that would be awsome | |
| 18:20:16 | efried | cool, on it. | |
| 18:21:04 | sean-k-mooney | well when i first propsoed this we did not have the abitiy to query nested rps because i first brough up this design 3-4 releases ago | |
| 18:21:34 | sean-k-mooney | so people were more conserend about 3 level when 2 level did not work | |
| 18:21:56 | efried | with placement today, this totally works. | |
| 18:22:04 | efried | you can represent the affinity using same_subtree | |
| 18:22:13 | efried | so the extra layer doesn't break that. | |
| 18:22:30 | sean-k-mooney | ya | |
| 18:22:59 | sean-k-mooney | if we do it this way we can remove much of the logic form the NTF and numa resouce tracker | |
| 18:23:36 | sean-k-mooney | unlike cpu pinning hugepage just need to know how much of each page type is avaible per numa node | |
| 18:23:51 | sean-k-mooney | so placement with 3 level can model eveything we need to track | |
| 18:24:30 | sean-k-mooney | so we could remvoe all the mempage trackinging in the host numa topology blob and only compute that in memroy to update the palcement inventory | |
| 18:28:27 | melwitt | stephenfin: looks like your update covered most of what was there but looks like you left out the server names the test was filling in before? https://review.opendev.org/695220 | |
| 18:54:06 | sean-k-mooney | efried: fyi the ther node would be for vgpus althoug they would proably go a the bottem level of the tree beside the memory ones | |
| 18:54:27 | sean-k-mooney | * the other nodes below the RP | |
| 18:54:59 | efried | For now we're punting on devices, so we'll leave the VGPUs where they are and not support affinity. Later, I agree, the providers representing PGPUs would be underneath the NUMA nodes, parallel to the memory providers. | |
| 18:55:11 | sean-k-mooney | yep | |
| 18:55:46 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Repro gen conflict in COMPUTE_STATUS_DISABLED handling https://review.opendev.org/704865 | |
| 18:55:46 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Reduce gen conflict in COMPUTE_STATUS_DISABLED handling https://review.opendev.org/704866 | |
| 18:56:37 | sean-k-mooney | efried: i know we have customer request for numa aware gpus/vgpus but we agreed even internally that that should wait until after the memory/cpu case is done | |
| 18:56:52 | sean-k-mooney | so im fine with that | |
| 18:56:54 | efried | yeah, sure, we're not going to not do it, we're just not going to do it now. | |
| 18:57:03 | sean-k-mooney | yep | |
| 18:57:45 | efried | sean-k-mooney: I'm going to put some examples in the etherpad, but will need you to go through and edit my spelling on the extra specs. | |
| 18:58:09 | sean-k-mooney | hehe i will try | |
| 18:58:46 | efried | I figure if I ask you to edit the spelling of real tokens I have a better chance. | |
| 18:58:49 | efried | vs. English. | |
| 18:59:01 | sean-k-mooney | yes you do | |
| 18:59:06 | efried | ye gods, it just occurred to me: do you speak gaelic? | |
| 18:59:35 | efried | your writing in gaelic must be IMPOSSIBLE to understand, given that gaelic is impossible to read in the first freakin place. | |
| 19:00:06 | sean-k-mooney | yes and no. i went to an all irish play school and got an irish langage exemption becaue i coudl not write in irish | |
| 19:00:15 | sean-k-mooney | i could speak it and could kind of read it | |
| 19:00:16 | efried | "oh, let me just throw an extra consonant and an extra vowel in each syllable. Half of them will be silent. Which ones? Who knows? It's a surprise!" | |
| 19:00:50 | sean-k-mooney | hehe irish is interesting that way | |
| 19:01:18 | sean-k-mooney | espcially since spelling of a verb chagne with gender pluarlity and something the noun it is used ith | |
| 19:05:53 | sean-k-mooney | actully do you want that to be a numa instance example or not | |
| 19:06:02 | sean-k-mooney | i assume a numa one | |
| 19:06:38 | sean-k-mooney | a non numa one would not have hw:mem_page_size set at all and not triat | |
| 19:10:21 | efried | eandersson: can you write me a flavor example with two numa nodes where one has small pages and one has large? | |
| 19:10:34 | efried | whoops, eandersson disregard. ^ was for sean-k-mooney | |
| 19:10:38 | sean-k-mooney | no | |
| 19:10:44 | sean-k-mooney | pagesize is vm wide | |
| 19:10:47 | efried | (my IRC client has been dropping the first char of my messages lately) | |
| 19:11:00 | efried | okay, fine, can you write me a flavor with two numa nodes and an explicit page size? | |
| 19:11:24 | sean-k-mooney | yes ill add 2 | |
| 19:11:31 | efried | (turns out 'sean<tab>' minus the first character is 'ean<tab>') | |
| 19:12:35 | sean-k-mooney | hehe ill remove our disscution on the first example to clean up the description | |
| 19:16:43 | sean-k-mooney | you can also split the cpu asemetircly but that is overkill for the exampels i think | |
| 19:17:27 | sean-k-mooney | for the implict splitign its an error if the cpus and ram and not evenly devisable by the number of numa nodes | |
| 19:19:16 | efried | sean-k-mooney: I prefer leaving that third example as it was before, since it shows that we could land on either of the LARGE RPs. | |
| 19:19:25 | efried | ++ | |
| 19:19:25 | sean-k-mooney | ya | |
| 19:19:30 | sean-k-mooney | that is why i reverted it | |
| 19:19:47 | sean-k-mooney | any why i used large in the first place | |
| 19:19:59 | efried | sean-k-mooney: any other examples that we should throw in? | |
| 19:20:11 | sean-k-mooney | am maybe a non numa guests | |
| 19:20:58 | efried | yeah | |
| 19:21:17 | efried | to show that it would not land on the sample host at all. | |
| 19:22:28 | sean-k-mooney | yes | |
| 19:22:37 | sean-k-mooney | i was trying to figure out the not num host | |
| 19:22:40 | sean-k-mooney | *numa | |
| 19:23:44 | sean-k-mooney | if we did not add the forbiden trait | |
| 19:23:49 | sean-k-mooney | then it could actully land on that host | |
| 19:24:00 | sean-k-mooney | but we would have to pin it to a numa node | |
| 19:24:13 | sean-k-mooney | which we could do but we proably dont want too | |
| 19:24:28 | sean-k-mooney | so you can continue to create vms that span numa nodes if you dont care | |
| 19:24:30 | efried | right, that's what I was implying earlier when I said we could probably overlap, but should prevent that for now | |
| 19:24:40 | efried | oh, that wouldn't allow you to span numa nodes. | |
| 19:24:40 | sean-k-mooney | ya | |
| 19:24:52 | sean-k-mooney | ya it would not you are right | |
| 19:24:56 | efried | oh, I see, you mean your proc from one and your mem from another | |
| 19:24:57 | sean-k-mooney | not without can split | |
| 19:25:20 | sean-k-mooney | you could have procs and memory span ya | |
| 19:25:33 | sean-k-mooney | but can_split would be need to have any one resouce span numa nodes | |
| 19:25:42 | efried | yeah, actually, the non-NUMA example can be simplified if we just use a granular group for the proc & mem.... | |