| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-01-29 | |||
| 17:32:50 | stephenfin | by creating e.g. 128GB instances on a dual socket compute node with only 64GB RAM per node | |
| 17:33:00 | sean-k-mooney | so unless we pinn the memory to a numa node when we creat the vm the kernel manages where the pages are paged form | |
| 17:33:16 | sean-k-mooney | for hugepages we allways pin them to a numa node | |
| 17:33:37 | efried | and why would we not pin the small pages? | |
| 17:33:47 | efried | wait | |
| 17:33:50 | efried | let me turn that around: | |
| 17:33:59 | efried | do we ever want to make sure we *do* pin the small pages? | |
| 17:34:24 | sean-k-mooney | we have an option for that today | |
| 17:34:33 | sean-k-mooney | if you set hw:mem_page_size=small | |
| 17:34:41 | efried | k, then we should model them in the numa cells, and always pin them. | |
| 17:34:45 | sean-k-mooney | if you set hw:mem_page_size it will be pinned | |
| 17:34:58 | stephenfin | If we're modelling NUMA in placement, then yes. We want to make sure that the memory placement allocated came from the actual provider (the host NUMA node) | |
| 17:35:14 | efried | that will mean there are a few cases where you didn't care about that and you don't land on a host whose small pages are sparse and spread across cells. But we've already said we're accepting that. | |
| 17:35:15 | sean-k-mooney | efried: we could but the implciation of that are 1 of the follow two things need to happen | |
| 17:35:40 | stephenfin | otherwise placements says "here's 8GB from <RP mapping to host NUMA node #1>", but on the host that memory comes from node #0 | |
| 17:35:57 | stephenfin | and placement doesn't reflect reality | |
| 17:36:07 | sean-k-mooney | 1 all vms must be numa vms and there fro cannot span host numa nodes unless you set hw:numa_nodes>1 | |
| 17:36:17 | efried | this ^ | |
| 17:36:31 | sean-k-mooney | or we have to allow a singel numa node guest to sapn host numa nodes based on the placement allcoation | |
| 17:36:38 | efried | nope | |
| 17:36:42 | stephenfin | alternative | |
| 17:36:45 | efried | Your VM is a numa VM | |
| 17:36:49 | stephenfin | let people turn off the NUMA reporting | |
| 17:36:54 | stephenfin | make everything flat again | |
| 17:36:54 | efried | If it's not a numa VM, it doesn't land on a NUMA host | |
| 17:36:59 | efried | yes, that ^ | |
| 17:37:01 | sean-k-mooney | ok i have argued for your vm is a numa vm for years | |
| 17:37:12 | stephenfin | Yeah, I also want that | |
| 17:37:12 | efried | sorry, you're misunderstanding. | |
| 17:37:26 | stephenfin | I know what efried is going to say | |
| 17:37:36 | stephenfin | the NUMA reporting has to be turned on? | |
| 17:37:38 | sean-k-mooney | ok no i get what your saying | |
| 17:37:46 | sean-k-mooney | but to not land on a numa host i think we need to have 2 resouce classes | |
| 17:37:47 | stephenfin | defaults to off | |
| 17:38:00 | efried | We force deployers to split their data center. | |
| 17:38:00 | efried | N hosts are NUMA-modeled. All and only VMs with a NUMA topo land on those hosts. | |
| 17:38:00 | efried | M hosts are flat. All and only *non* NUMA topo VMs land on those hosts. | |
| 17:38:36 | sean-k-mooney | so in pricipal you shoudl be doing that partioning alreday | |
| 17:38:37 | sean-k-mooney | for reason i wont go into today | |
| 17:38:41 | efried | exactly. | |
| 17:38:50 | stephenfin | she borked. | |
| 17:38:57 | stephenfin | (if you don't do that partitioning) | |
| 17:39:01 | sean-k-mooney | elequently said | |
| 17:39:04 | efried | there may be a way we can support a small amount of overlap in the future, but for the moment I think it wouldn't be a terrible idea to add (under the covers) a required or forbidden trait that makes sure you land on the right kind of host. | |
| 17:39:27 | efried | like the NUMA_ROOT trait on the provider from which you're requesting $resource. | |
| 17:39:43 | efried | so let's get back to pages. | |
| 17:39:44 | stephenfin | that might be needed to handle the upgrade impact | |
| 17:39:59 | efried | How many "abstract" page sizes are there? | |
| 17:40:07 | efried | small/large/huge? | |
| 17:40:14 | stephenfin | we can't expect users to go set the "this is NUMA node" flag ahead of time | |
| 17:40:18 | stephenfin | efried: small, large | |
| 17:40:29 | stephenfin | hey, I documented that for my validator | |
| 17:40:30 | efried | stephenfin: totally not, that happens under the covers, now and forever, if it happens at all. | |
| 17:40:45 | efried | okay, and how many *discrete* page sizes are there within those buckets? | |
| 17:40:45 | sean-k-mooney | small/large/any or a specifc page size expressed as an integer with an optional suffix | |
| 17:41:04 | stephenfin | https://review.opendev.org/#/c/704643/2/nova/api/validation/extra_specs/hw.py@113 | |
| 17:41:09 | efried | \o/ | |
| 17:41:12 | stephenfin | noting sean-k-mooney's comment | |
| 17:41:15 | sean-k-mooney | the integuer is unbounded but in pratice about 12ish | |
| 17:41:32 | stephenfin | efried: architecture dependent | |
| 17:41:40 | stephenfin | Intel supports 2M and 1G | |
| 17:41:47 | stephenfin | *x86 | |
| 17:41:49 | efried | lovely. Is there overlap on what's considered small and large? | |
| 17:41:57 | stephenfin | POWER supports 8G, iirc | |
| 17:41:57 | sean-k-mooney | efried: jay put up a patch to os resouces for the common ones a while ago | |
| 17:42:01 | stephenfin | Who knows what ARM supports | |
| 17:42:04 | efried | that is, are there discrete values that are considered small on some systems and large on others? | |
| 17:42:10 | stephenfin | ALL the pagesizes | |
| 17:42:22 | sean-k-mooney | efried: in practice not really but technically there can be | |
| 17:42:30 | stephenfin | sean-k-mooney: 4k is a small page _everywhere_, right? | |
| 17:42:32 | sean-k-mooney | small is almost always 4k | |
| 17:42:38 | sean-k-mooney | no | |
| 17:42:40 | efried | those are different statements | |
| 17:42:54 | sean-k-mooney | small is the native pagesize of the host | |
| 17:43:01 | sean-k-mooney | and the smallest in the available set | |
| 17:43:25 | sean-k-mooney | it is almost alwasy 4k and raely 16k or 64k | |
| 17:43:47 | stephenfin | sean-k-mooney: The internet tells me 4k is hardcoded as the default page size in Linux | |
| 17:43:57 | stephenfin | https://unix.stackexchange.com/a/128218 | |
| 17:44:18 | sean-k-mooney | yes on x86, aarch64 and power9 | |
| 17:44:41 | stephenfin | okay, we don't need to care about anything else, realistically | |
| 17:44:56 | sean-k-mooney | that why i said its almost alwasy 4k | |
| 17:45:01 | stephenfin | we can stick a giant TODO in somewhere in case someone wants to run openstack on obscure architecture | |
| 17:45:03 | sean-k-mooney | we can proably treat it as such | |
| 17:45:07 | stephenfin | agreed | |
| 17:45:15 | stephenfin | s/TODO/NOTE/ | |
| 17:45:36 | stephenfin | efried: what are you referring to? | |
| 17:45:44 | sean-k-mooney | large is defiend as any pagesize that is not the same as small on the plathform | |
| 17:45:49 | efried | okay, so I'm afraid the compromise we need to make is this: | |
| 17:45:49 | efried | Represent all the memory as MEMORY_MB with a step_size that's the least common denominator of all the page sizes. | |
| 17:45:49 | efried | Supply traits saying things like I_HAVE_$SIZE_PAGES_HERE, where $SIZE can include both discrete and abstract sizes. | |
| 17:45:49 | efried | Use ^ where possible to make the placement pass a bit better | |
| 17:45:49 | efried | but accept that it's not going to be perfect, and that we have to do the rest in the NTF. | |
| 17:46:12 | efried | ...which may entail bouncing a host late. | |
| 17:46:53 | stephenfin | I don't understand how step_size would work | |
| 17:46:54 | sean-k-mooney | efried: if we do that we need to keep the hugepage code in the resouce tracker and numa toplogy filter for ever | |
| 17:47:24 | efried | I'm afraid that may be unavoidable, unless we make big changes in placement, or people agree to stop needing that shit. | |
| 17:47:29 | sean-k-mooney | efried: if we had a placement exteion weree we could pass a step size in the querry that would help | |
| 17:47:39 | stephenfin | I have a host that have 16 1GB hugepages and the rest are normal small pages | |
| 17:47:45 | efried | stephenfin: example, if we have 4K, 1M, and 2G pages, the step_size has to be 4K | |
| 17:47:59 | stephenfin | you will always use 4k so | |