| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-01-29 | |||
| 17:17:39 | sean-k-mooney | yes because we support hw:mem_page_size=large | |
| 17:17:41 | stephenfin | PAGES_1M, surely? | |
| 17:17:47 | efried | I hate that. | |
| 17:17:53 | stephenfin | PAGES_1M, PAGES_2K, PAGES_1G? | |
| 17:17:54 | sean-k-mooney | and that cant be modeld as pages_2M and pages _1G | |
| 17:17:55 | efried | unless 1M is the only size we can ever have. | |
| 17:18:00 | efried | No | |
| 17:18:13 | efried | we ought to be able to do that with MEMORY_MB with appropriate step_size. | |
| 17:18:33 | sean-k-mooney | my point is we cant make mem_page_size=large work if the resouce class has the size info | |
| 17:18:36 | efried | then you can ask for so many MB, and you'll get the right number of pages chopped up according to that step_size. | |
| 17:18:48 | efried | pretty sure that's what step_size exists for. | |
| 17:18:56 | sean-k-mooney | the step_zize and min/max allocation works | |
| 17:19:09 | stephenfin | how does one determine the step size? | |
| 17:19:15 | efried | it's the page size | |
| 17:19:16 | sean-k-mooney | i have talked about that before | |
| 17:19:24 | efried | the virt driver knows how big a page is, yah? | |
| 17:19:25 | stephenfin | large isn't a page size | |
| 17:19:31 | sean-k-mooney | but we cant explctly request 1G or 2mb pages in that case | |
| 17:19:31 | efried | no, 'large' isn't a thing. | |
| 17:19:31 | stephenfin | it's anything bigger than 4k | |
| 17:19:39 | sean-k-mooney | efried: yes it is | |
| 17:19:43 | stephenfin | which could be 1G on half the compute nodes | |
| 17:19:57 | sean-k-mooney | hw:mem_page_size=large is the recommend way to enable hugepages | |
| 17:20:02 | efried | wait, so there are flavors that care how many pages they get, but not how much memory that ends up being???? | |
| 17:20:16 | sean-k-mooney | yes | |
| 17:20:20 | stephenfin | no | |
| 17:20:23 | sean-k-mooney | well not pages | |
| 17:20:26 | sean-k-mooney | but page size | |
| 17:20:37 | stephenfin | they care about how much memory they get but not the exact size of the pages | |
| 17:20:42 | stephenfin | just so long as it's != 4k | |
| 17:20:58 | stephenfin | 2M, 1G, 8G (on POWER) - it's all fair game | |
| 17:21:13 | sean-k-mooney | stephenfin: that is not alwasy true | |
| 17:21:23 | sean-k-mooney | many people do care about the page size | |
| 17:21:23 | efried | Did we ever implement required=in:T1,T2,T3? | |
| 17:21:29 | sean-k-mooney | no | |
| 17:21:35 | efried | so | |
| 17:21:38 | sean-k-mooney | if we did we could use that | |
| 17:22:01 | sean-k-mooney | although no it would not work because reqired in was for traits | |
| 17:22:05 | efried | if people care about specific page sizes, we have a trait that says PAGE_SIZES_HERE_ARE_4K. | |
| 17:22:05 | efried | But if people care about 'large', where that's allowed to mean "bigger than X", that doesn't work. | |
| 17:22:06 | stephenfin | right, and those people are probably explicitly saying e.g. 'hw:mem_page_size=2k' | |
| 17:22:13 | sean-k-mooney | yes | |
| 17:22:21 | stephenfin | or know that their datacenter is configured with only 2k or 1G pages | |
| 17:22:25 | efried | (is bauzas still listening btw?) | |
| 17:22:32 | stephenfin | bauzas is on kid duty | |
| 17:22:35 | stephenfin | but has scrollback | |
| 17:22:35 | efried | k | |
| 17:22:38 | efried | yuh | |
| 17:23:24 | stephenfin | efried: Still not sure how we do the step_size determination though | |
| 17:23:30 | sean-k-mooney | https://etherpad.openstack.org/p/mem_page_size_and_placement | |
| 17:23:37 | sean-k-mooney | lets go there too | |
| 17:23:39 | efried | stephenfin: Doesn't the virt driver know how big its pages are? | |
| 17:23:44 | sean-k-mooney | yes | |
| 17:23:51 | efried | that's the step_size. | |
| 17:23:52 | sean-k-mooney | and a host can have multiple pages sizes | |
| 17:24:03 | efried | in the same numa cell? | |
| 17:24:15 | sean-k-mooney | a host can have 4k 2mb and 1g in the same cell yes | |
| 17:24:16 | gibi | stephenfin: having a single representation is a positive thing for me. and I accept that it will be python instead of yaml | |
| 17:24:18 | stephenfin | But you don't know what virt driver you're going to use before you query placement | |
| 17:24:28 | efried | stephenfin: step_size is part of the model, not part of the query. | |
| 17:24:30 | sean-k-mooney | yep which is why large is a thing | |
| 17:24:50 | sean-k-mooney | efried: yep that was why step size did not work when i discussed this in the past | |
| 17:25:04 | efried | because you can have different size pages in the same cell. | |
| 17:25:07 | sean-k-mooney | we cant say 10G of ram in 1G increments | |
| 17:25:24 | sean-k-mooney | at least not with just stepsize | |
| 17:25:30 | stephenfin | efried: you can, and not everything the same size | |
| 17:25:47 | efried | okay, then there's no way this works without an abstraction and/or simplification. | |
| 17:25:55 | efried | so we need to decide what to cut | |
| 17:26:05 | stephenfin | How would you say "give me a compute node that can handle an 8GB instance with 1G pages" | |
| 17:26:06 | efried | because we're not going to make resource classes for PAGE_$size | |
| 17:26:23 | stephenfin | I don't see how we can avoid it | |
| 17:26:36 | stephenfin | a 4k page != a 2M page | |
| 17:26:45 | efried | stephenfin: because then you *can't* say "give me an instance with 8GB" | |
| 17:27:00 | efried | which is surely the more common use case? | |
| 17:27:06 | stephenfin | sure you can | |
| 17:27:13 | stephenfin | we just translate the request | |
| 17:27:24 | efried | you would have to translate the request into multiple GET /a_c queries. | |
| 17:27:34 | efried | because you can build 8GB a zillion different ways | |
| 17:27:46 | efried | AND it requires discovery of which page-size-resource-classes exist in your deployment. | |
| 17:27:51 | efried | that's a non-starter, sorry. | |
| 17:28:01 | stephenfin | No, you can't consume hugepages unless you explicitly request them | |
| 17:28:08 | sean-k-mooney | so by default if you dont specify hw:mem_page_size it has to be the smalles page size | |
| 17:28:19 | sean-k-mooney | efried: that is why i want mempage_small and mempage_large to be different resoce classes | |
| 17:28:20 | efried | but "huge" isn't a discrete number? | |
| 17:28:39 | efried | so again, if you do that, you can't ask for 8GB | |
| 17:28:43 | stephenfin | hugepages are offlimits for instances without 'hw:mem_page_size' | |
| 17:28:43 | stephenfin | right, sean-k-mooney? | |
| 17:28:54 | efried | because you have to ask for a number of pages, but you don't know how big those pages are. | |
| 17:29:02 | stephenfin | yeah, that one is a problem | |
| 17:29:10 | stephenfin | because it's intentionally abstract | |
| 17:29:30 | sean-k-mooney | yes | |
| 17:29:32 | sean-k-mooney | if you dont set hw:mem_page_size you will only use 4k memory | |
| 17:29:42 | stephenfin | right | |
| 17:29:59 | stephenfin | so for non-hugepages instances, there will only ever be a single GET /a_c query | |
| 17:30:07 | efried | um | |
| 17:30:12 | stephenfin | for MEMORY_MB or PAGES_4K or whatever we call it | |
| 17:30:13 | sean-k-mooney | efried: its that way because people wanted to change for hugepages and people did not want all instacne to have a numa toplogy by defualt | |
| 17:30:13 | efried | there will only ever be a single GET /a_c query. | |
| 17:30:20 | efried | ever ever | |
| 17:30:46 | stephenfin | not for pinned instances, at the moment | |