| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-01-29 | |||
| 17:25:47 | efried | okay, then there's no way this works without an abstraction and/or simplification. | |
| 17:25:55 | efried | so we need to decide what to cut | |
| 17:26:05 | stephenfin | How would you say "give me a compute node that can handle an 8GB instance with 1G pages" | |
| 17:26:06 | efried | because we're not going to make resource classes for PAGE_$size | |
| 17:26:23 | stephenfin | I don't see how we can avoid it | |
| 17:26:36 | stephenfin | a 4k page != a 2M page | |
| 17:26:45 | efried | stephenfin: because then you *can't* say "give me an instance with 8GB" | |
| 17:27:00 | efried | which is surely the more common use case? | |
| 17:27:06 | stephenfin | sure you can | |
| 17:27:13 | stephenfin | we just translate the request | |
| 17:27:24 | efried | you would have to translate the request into multiple GET /a_c queries. | |
| 17:27:34 | efried | because you can build 8GB a zillion different ways | |
| 17:27:46 | efried | AND it requires discovery of which page-size-resource-classes exist in your deployment. | |
| 17:27:51 | efried | that's a non-starter, sorry. | |
| 17:28:01 | stephenfin | No, you can't consume hugepages unless you explicitly request them | |
| 17:28:08 | sean-k-mooney | so by default if you dont specify hw:mem_page_size it has to be the smalles page size | |
| 17:28:19 | sean-k-mooney | efried: that is why i want mempage_small and mempage_large to be different resoce classes | |
| 17:28:20 | efried | but "huge" isn't a discrete number? | |
| 17:28:39 | efried | so again, if you do that, you can't ask for 8GB | |
| 17:28:43 | stephenfin | hugepages are offlimits for instances without 'hw:mem_page_size' | |
| 17:28:43 | stephenfin | right, sean-k-mooney? | |
| 17:28:54 | efried | because you have to ask for a number of pages, but you don't know how big those pages are. | |
| 17:29:02 | stephenfin | yeah, that one is a problem | |
| 17:29:10 | stephenfin | because it's intentionally abstract | |
| 17:29:30 | sean-k-mooney | yes | |
| 17:29:32 | sean-k-mooney | if you dont set hw:mem_page_size you will only use 4k memory | |
| 17:29:42 | stephenfin | right | |
| 17:29:59 | stephenfin | so for non-hugepages instances, there will only ever be a single GET /a_c query | |
| 17:30:07 | efried | um | |
| 17:30:12 | stephenfin | for MEMORY_MB or PAGES_4K or whatever we call it | |
| 17:30:13 | sean-k-mooney | efried: its that way because people wanted to change for hugepages and people did not want all instacne to have a numa toplogy by defualt | |
| 17:30:13 | efried | there will only ever be a single GET /a_c query. | |
| 17:30:20 | efried | ever ever | |
| 17:30:46 | stephenfin | not for pinned instances, at the moment | |
| 17:30:53 | stephenfin | but that's a temporary thing | |
| 17:30:55 | stephenfin | agreed | |
| 17:30:56 | sean-k-mooney | so i was proposeing useing MEMORY_MB for non numa small pages | |
| 17:31:01 | sean-k-mooney | e.g. on the root RP | |
| 17:31:04 | efried | sean-k-mooney: we're simplifying the "all instances have a numa topology" aspect by saying that NUMA-aware requests only go to NUMA-modeled hosts and vice versa. | |
| 17:31:22 | stephenfin | sean-k-mooney: agree on the first part. disagree on the second | |
| 17:31:35 | stephenfin | it's on the root RP if that compute node is configured to be NUMA'y | |
| 17:31:36 | sean-k-mooney | well that is all hosts | |
| 17:31:42 | efried | The small pages are still affined, yah? | |
| 17:31:50 | efried | So they should be modeled on the NUMA RPs | |
| 17:31:54 | sean-k-mooney | there are no non numa host for the better part of a decade | |
| 17:31:56 | stephenfin | efried: If you're using guest NUMA, they should be | |
| 17:32:11 | stephenfin | sean-k-mooney: No, but there are people who pretend they don't exist | |
| 17:32:14 | stephenfin | *they exist | |
| 17:32:16 | sean-k-mooney | efried: they have an affinty but they are allocated by the kernel | |
| 17:32:17 | efried | yeah, so given a strict NUMA-or-not split, I don't see a reason to keep any memory on the root | |
| 17:32:24 | sean-k-mooney | and they can move between numa nodes | |
| 17:32:50 | stephenfin | by creating e.g. 128GB instances on a dual socket compute node with only 64GB RAM per node | |
| 17:33:00 | sean-k-mooney | so unless we pinn the memory to a numa node when we creat the vm the kernel manages where the pages are paged form | |
| 17:33:16 | sean-k-mooney | for hugepages we allways pin them to a numa node | |
| 17:33:37 | efried | and why would we not pin the small pages? | |
| 17:33:47 | efried | wait | |
| 17:33:50 | efried | let me turn that around: | |
| 17:33:59 | efried | do we ever want to make sure we *do* pin the small pages? | |
| 17:34:24 | sean-k-mooney | we have an option for that today | |
| 17:34:33 | sean-k-mooney | if you set hw:mem_page_size=small | |
| 17:34:41 | efried | k, then we should model them in the numa cells, and always pin them. | |
| 17:34:45 | sean-k-mooney | if you set hw:mem_page_size it will be pinned | |
| 17:34:58 | stephenfin | If we're modelling NUMA in placement, then yes. We want to make sure that the memory placement allocated came from the actual provider (the host NUMA node) | |
| 17:35:14 | efried | that will mean there are a few cases where you didn't care about that and you don't land on a host whose small pages are sparse and spread across cells. But we've already said we're accepting that. | |
| 17:35:15 | sean-k-mooney | efried: we could but the implciation of that are 1 of the follow two things need to happen | |
| 17:35:40 | stephenfin | otherwise placements says "here's 8GB from <RP mapping to host NUMA node #1>", but on the host that memory comes from node #0 | |
| 17:35:57 | stephenfin | and placement doesn't reflect reality | |
| 17:36:07 | sean-k-mooney | 1 all vms must be numa vms and there fro cannot span host numa nodes unless you set hw:numa_nodes>1 | |
| 17:36:17 | efried | this ^ | |
| 17:36:31 | sean-k-mooney | or we have to allow a singel numa node guest to sapn host numa nodes based on the placement allcoation | |
| 17:36:38 | efried | nope | |
| 17:36:42 | stephenfin | alternative | |
| 17:36:45 | efried | Your VM is a numa VM | |
| 17:36:49 | stephenfin | let people turn off the NUMA reporting | |
| 17:36:54 | stephenfin | make everything flat again | |
| 17:36:54 | efried | If it's not a numa VM, it doesn't land on a NUMA host | |
| 17:36:59 | efried | yes, that ^ | |
| 17:37:01 | sean-k-mooney | ok i have argued for your vm is a numa vm for years | |
| 17:37:12 | stephenfin | Yeah, I also want that | |
| 17:37:12 | efried | sorry, you're misunderstanding. | |
| 17:37:26 | stephenfin | I know what efried is going to say | |
| 17:37:36 | stephenfin | the NUMA reporting has to be turned on? | |
| 17:37:38 | sean-k-mooney | ok no i get what your saying | |
| 17:37:46 | sean-k-mooney | but to not land on a numa host i think we need to have 2 resouce classes | |
| 17:37:47 | stephenfin | defaults to off | |
| 17:38:00 | efried | We force deployers to split their data center. | |
| 17:38:00 | efried | N hosts are NUMA-modeled. All and only VMs with a NUMA topo land on those hosts. | |
| 17:38:00 | efried | M hosts are flat. All and only *non* NUMA topo VMs land on those hosts. | |
| 17:38:36 | sean-k-mooney | so in pricipal you shoudl be doing that partioning alreday | |
| 17:38:37 | sean-k-mooney | for reason i wont go into today | |
| 17:38:41 | efried | exactly. | |
| 17:38:50 | stephenfin | she borked. | |
| 17:38:57 | stephenfin | (if you don't do that partitioning) | |
| 17:39:01 | sean-k-mooney | elequently said | |
| 17:39:04 | efried | there may be a way we can support a small amount of overlap in the future, but for the moment I think it wouldn't be a terrible idea to add (under the covers) a required or forbidden trait that makes sure you land on the right kind of host. | |
| 17:39:27 | efried | like the NUMA_ROOT trait on the provider from which you're requesting $resource. | |
| 17:39:43 | efried | so let's get back to pages. | |
| 17:39:44 | stephenfin | that might be needed to handle the upgrade impact | |
| 17:39:59 | efried | How many "abstract" page sizes are there? | |
| 17:40:07 | efried | small/large/huge? | |