| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-01-29 | |||
| 16:41:53 | bauzas | either way, I know what to do, but I have to look at the target differences in tox.ini | |
| 16:42:32 | efried | bauzas: would it be productive for me to read the current PS of the numa topo spec, or wait til you rev it? | |
| 16:43:06 | bauzas | efried: I'm wraping my head around the group_policy issue, but your thoughts could be helpful | |
| 16:43:59 | bauzas | efried: and we have a disagreement on the memory modeling with NUMA with sean-k-mooney, your opinion could help us finding a consensus | |
| 16:44:25 | bauzas | efried: to answer your question, yeah comments would be appreciated on the current rev | |
| 16:44:32 | efried | okay, I'll give it a read. My view on group_policy is that we should be ignoring it. It doesn't really have a place with granular groups and the other knobs we put in in recent microversions. | |
| 16:44:59 | efried | I at least proposed (though I don't remember if we actually pulled the trigger on this) making it no longer required in a recent microversion. | |
| 16:45:09 | sean-k-mooney | bauzas: sorry can we pick this up after the internal call | |
| 16:45:19 | sean-k-mooney | e.g. in 15 mins | |
| 16:45:29 | bauzas | sean-k-mooney: yeah, and I even wanted to discuss it during the internal call :D | |
| 16:45:42 | bauzas | (but I'm superseded by other topics :) ) | |
| 16:45:52 | sean-k-mooney | ya i saw | |
| 16:47:14 | efried | I won't be much help on the memory modeling, probably, as I really don't understand that level of detail of NUMA itself. But I'll give it a shot. | |
| 16:48:38 | bauzas | tbh, me too | |
| 16:49:13 | bauzas | the big question is should we iteratively model huge pages and have memory split now, or care about the whole now ? | |
| 16:51:02 | efried | I'll have to refresh my memory (heh) on whether we did things to support cross-provider accumulation of resources, e.g. so you could still land on a NUMA-modeled host with 128/128 if you asked for 256. | |
| 16:52:10 | efried | I seem to recall we tried to KISS by saying you land on a NUMA-modeled host by asking for NUMA-modeled resources, and the converse, but that may have only been a point in time in the discussion, not where we landed. | |
| 16:52:15 | efried | I'll have to swap this all back in. | |
| 16:53:05 | bauzas | efried: yeah that's what I recall too | |
| 17:03:35 | openstackgerrit | Vladyslav Drok proposed openstack/nova master: Fix volume attachment rollback https://review.opendev.org/704847 | |
| 17:07:39 | efried | nts: we did abandon "can_split" https://review.opendev.org/#/c/658510/ | |
| 17:08:03 | sean-k-mooney | yes we did | |
| 17:09:29 | stephenfin | sean-k-mooney: for that memory issue, is there a reason we need to continue to support floating memory when NUMA is used? | |
| 17:09:49 | stephenfin | do we do strict NUMA affinity when using a NUMA topology at the moment? | |
| 17:09:52 | sean-k-mooney | that depends. yes but the question is do we care | |
| 17:10:06 | stephenfin | for memory, that is | |
| 17:10:13 | sean-k-mooney | we would have to pin the memoyy based on the placemetn allocation of all vms | |
| 17:10:18 | stephenfin | I think we don't unless we explicitly request a pagesize, yeah? | |
| 17:10:34 | sean-k-mooney | but for vms with out a numa toplogy we could pin that vm aross host numa nodes | |
| 17:11:00 | stephenfin | okay, so that leads to my question: do we need to support that model? | |
| 17:11:10 | stephenfin | VMs with a NUMA topology alongside those without? | |
| 17:11:13 | efried | Again, I think we decided it was acceptable to enforce a segregated data center | |
| 17:11:15 | sean-k-mooney | if we want to support vms without a numa toplogy yes | |
| 17:11:21 | efried | where some hosts are NUMA-aware and some are not. | |
| 17:11:26 | stephenfin | sean-k-mooney: we could have a knob | |
| 17:11:32 | stephenfin | use_numa_in_placement | |
| 17:11:38 | efried | And if you ask for a NUMA topo, you land on the former, always, exclusively. And the converse. | |
| 17:11:46 | stephenfin | efried: right, thing we're talking about the same thing | |
| 17:11:48 | stephenfin | *think | |
| 17:11:54 | sean-k-mooney | i would prefer to handel it via resouce classes | |
| 17:12:05 | sean-k-mooney | so if you report memory_mb its non numa aware memory | |
| 17:12:12 | efried | Except for the upgrade issue, I think it wouldn't matter. | |
| 17:12:17 | sean-k-mooney | if you reprot mempage_small its numa aware | |
| 17:12:31 | efried | Upgrade is my only concern about doing MEMORY_MB now and pages later. | |
| 17:12:40 | efried | because now you have flavors that straddle both models | |
| 17:12:52 | stephenfin | sean-k-mooney: right, but that's duplication | |
| 17:12:57 | efried | but they are mutually exclusive as to the hosts they work with. | |
| 17:12:58 | sean-k-mooney | same for the cpu if you use VCPU its not numa aware and on the root provider if it SCPU its numa aware shared cpu | |
| 17:13:03 | stephenfin | and instances will have to consume both | |
| 17:13:08 | stephenfin | assuming we report both | |
| 17:13:11 | sean-k-mooney | stephenfin: no the host would only report one or the other | |
| 17:13:18 | efried | We shouldn't have the same resource represented by two different resource classes. We've nacked that idea many times in the past. | |
| 17:13:24 | stephenfin | what determines what type we report? | |
| 17:13:32 | sean-k-mooney | we did it with pcpus and vcpus | |
| 17:14:01 | sean-k-mooney | but if we say that then MEMORY_MB is not the correct baseline for memory | |
| 17:14:02 | efried | eh? No, the PCPU and VCPU counts represent different physical processors. | |
| 17:14:27 | stephenfin | oh, so you're saying instead of having a "turn on NUMA" flag, we have two flags: one for non-NUMA memory and one for NUMA memory? | |
| 17:14:28 | efried | yeah, I think we ought to bite the bullet on memory. If we're ever going to want it to be something other than MEMORY_MB, we should do that now. | |
| 17:14:29 | sean-k-mooney | they represetn different pools of hardware threads | |
| 17:14:40 | sean-k-mooney | but they are both the resouces | |
| 17:14:47 | sean-k-mooney | we just treat them as if they are different | |
| 17:14:48 | efried | my point is, they don't overlap | |
| 17:15:19 | sean-k-mooney | right so i think we should be modeling memory as mempages of a specifc size | |
| 17:15:34 | sean-k-mooney | we already do in the resouce tracker in the host numa toplogy blob | |
| 17:15:39 | efried | If we have N pages that total to M mb of ram, we should *either* model PAGES=N *or* MEMORY_MB=M, but *never* both at the same time. | |
| 17:15:48 | efried | I think we're agreeing on that, just want to be clear. | |
| 17:15:59 | stephenfin | Yup | |
| 17:16:13 | sean-k-mooney | yep although its slightly more complicated | |
| 17:16:26 | efried | sean-k-mooney: I also want us to make sure we're not tying ourselves to libvirt unless that's inherently unavoidable. | |
| 17:16:30 | sean-k-mooney | but in pricipal yes always one or the other | |
| 17:16:40 | efried | I.e. are there other drivers that don't think of their NUMA-based memory in terms of pages? | |
| 17:16:41 | sean-k-mooney | yes i want to add an abstration | |
| 17:16:43 | stephenfin | maybe we should make like the 90s and teach placement about OOP | |
| 17:16:53 | efried | Or that think of them as pages of a different size? | |
| 17:16:54 | sean-k-mooney | so mempages_small and mempages_larage | |
| 17:16:56 | stephenfin | 1GB_PAGES is_a PAGES | |
| 17:17:08 | efried | hahaha, you're a funny guy. | |
| 17:17:14 | sean-k-mooney | with 1G or 2MB large ppages modeld as a trait | |
| 17:17:25 | efried | NO | |
| 17:17:39 | sean-k-mooney | yes because we support hw:mem_page_size=large | |
| 17:17:41 | stephenfin | PAGES_1M, surely? | |
| 17:17:47 | efried | I hate that. | |
| 17:17:53 | stephenfin | PAGES_1M, PAGES_2K, PAGES_1G? | |
| 17:17:54 | sean-k-mooney | and that cant be modeld as pages_2M and pages _1G | |
| 17:17:55 | efried | unless 1M is the only size we can ever have. | |
| 17:18:00 | efried | No | |
| 17:18:13 | efried | we ought to be able to do that with MEMORY_MB with appropriate step_size. | |
| 17:18:33 | sean-k-mooney | my point is we cant make mem_page_size=large work if the resouce class has the size info | |
| 17:18:36 | efried | then you can ask for so many MB, and you'll get the right number of pages chopped up according to that step_size. | |
| 17:18:48 | efried | pretty sure that's what step_size exists for. | |
| 17:18:56 | sean-k-mooney | the step_zize and min/max allocation works | |
| 17:19:09 | stephenfin | how does one determine the step size? | |
| 17:19:15 | efried | it's the page size | |
| 17:19:16 | sean-k-mooney | i have talked about that before | |
| 17:19:24 | efried | the virt driver knows how big a page is, yah? | |
| 17:19:25 | stephenfin | large isn't a page size | |
| 17:19:31 | sean-k-mooney | but we cant explctly request 1G or 2mb pages in that case | |
| 17:19:31 | efried | no, 'large' isn't a thing. | |
| 17:19:31 | stephenfin | it's anything bigger than 4k | |
| 17:19:39 | sean-k-mooney | efried: yes it is | |