| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-09-17 | |||
| 15:51:32 | dansmith | I think we're talking about how to reshape from the controller (or not) | |
| 15:51:45 | bauzas | mriedem's patch ? | |
| 15:51:52 | bauzas | it's simple in my mind | |
| 15:51:59 | bauzas | we only support one type | |
| 15:52:18 | bauzas | reshaping will just create a child with the exact same inventory | |
| 15:52:27 | bauzas | and move allocations against it | |
| 15:52:40 | sean-k-mooney | bauzas: do we today supprot 2+ pgpus on a host | |
| 15:52:58 | bauzas | sean-k-mooney: yup, but they are hidden for libvirt | |
| 15:53:04 | mriedem | bauzas: and that's what i've implemented | |
| 15:53:10 | bauzas | mriedem: I know | |
| 15:53:13 | bauzas | hence my +1 | |
| 15:53:17 | bauzas | so i don't get the concerns | |
| 15:53:27 | bauzas | having multiple cards isn't a problem | |
| 15:53:40 | bauzas | or I'm missing a crucial point | |
| 15:53:52 | bauzas | I mean, I can be wrong but I don't get the problem | |
| 15:54:00 | dansmith | bauzas: presumably we want to reshape inventories onto providers that represent actual cards, right? | |
| 15:54:09 | bauzas | dansmith: no | |
| 15:54:10 | sean-k-mooney | bauzas: i would have assumed that as part of reshaper we would have had to devide that compinded inventory across the pGPUs no ? | |
| 15:54:11 | dansmith | not just reshape onto a provider that matches the one type we support today? | |
| 15:54:21 | dansmith | bauzas: why not? | |
| 15:54:46 | bauzas | dansmith: no, because once we support multiple types, they will still hide the fact that each type can be supported on N possible pGPUs | |
| 15:55:02 | dansmith | how is that helpful? | |
| 15:55:04 | dansmith | meaning, | |
| 15:55:14 | dansmith | how does that help us represent the physical numa affinity? | |
| 15:55:15 | sean-k-mooney | bauzas: you can hide that but you can also get the info via libvirt and expose it | |
| 15:55:34 | dansmith | and, presumably a guest that wants 4 vgpus won't (presumably) be happy with 2 from 2 separate cards? | |
| 15:56:00 | sean-k-mooney | dansmith: actully i think 4vgpus form 2 cards should be ok | |
| 15:56:21 | bauzas | dansmith: ah shit, for NUMA affinity, topology is important | |
| 15:56:28 | dansmith | sean-k-mooney: not if they look to be on the same NUMA node but aren't, right? | |
| 15:56:36 | bauzas | shit shit shit | |
| 15:56:40 | dansmith | ... | |
| 15:56:45 | sean-k-mooney | dansmith: right if they are split across numa nodes that would be bad | |
| 15:56:45 | dansmith | isn't this the whole discussion we had in denver? | |
| 15:56:53 | dansmith | sean-k-mooney: right | |
| 15:57:10 | dansmith | I'm confused about why we had that discussion and are now having it again :) | |
| 15:57:15 | sean-k-mooney | dansmith: but in general they will appear to be 4 different cards to the guest driver so it wont be able to tell the difference bar numa effects | |
| 15:57:20 | dansmith | but I have to jump on a call in a few minutes | |
| 15:57:26 | dansmith | sean-k-mooney: sure | |
| 15:57:29 | bauzas | dansmith: in https://specs.openstack.org/openstack/nova-specs/specs/queens/implemented/add-support-for-vgpu.html#proposed-change we always thought on GPU types | |
| 15:58:02 | bauzas | dansmith: yes we barely discussed on NUMA affinity for GPUs | |
| 15:58:12 | dansmith | barely/ | |
| 15:58:15 | bauzas | dansmith: but in my mind, those were GPU types, and not cards | |
| 15:58:27 | dansmith | what was the whole reason to talk about the reshape happening on the compute node, AFAIK :) | |
| 15:58:55 | dansmith | anyway, /me -> call | |
| 15:59:20 | johnthetubaguy | +1 | |
| 15:59:47 | bauzas | mriedem: I just feel I'll necessarly have to help you on the reshaper patch | |
| 16:00:25 | bauzas | but I first need to consider whether a child being a pGPU and not a GPU type would change | |
| 16:02:00 | sean-k-mooney | bauzas: i think the pGPUs would be childeren of the numa nodes with inventoies of vgpu(types) | |
| 16:02:30 | sean-k-mooney | bauzas: we might also want to group pgpus into nvlink aggretates at some point but not now | |
| 16:03:58 | bauzas | sean-k-mooney: shhhhhhhhhhhht | |
| 16:04:00 | mriedem | bauzas: clearly. i only shat this out b/c it seemed trivial. i never made the link between the numa node stuff in the vgpu reshaper discussions. | |
| 16:04:12 | bauzas | sean-k-mooney: some customer could hear us and ask for this feature | |
| 16:04:27 | bauzas | mriedem: and my brain fucked on it, I'm sorry | |
| 16:05:09 | bauzas | that's where I think it's crucial to see the topology now and in the future | |
| 16:05:14 | bauzas | in order to prevent reshapes | |
| 16:05:27 | bauzas | we could implement chidren as GPU types and do reshapes later | |
| 16:05:38 | bauzas | or we could reshape it now and do the necessary work now | |
| 16:05:53 | bauzas | I think given FFU pain, the less reshapes we have, the better it will be | |
| 16:06:10 | bauzas | mriedem: permission to play with your change ? | |
| 16:06:36 | mriedem | bauzas: sure | |
| 16:06:46 | sean-k-mooney | bauzas: /window splitv | |
| 16:06:58 | sean-k-mooney | ... ignore that | |
| 16:07:37 | bauzas | call me vim | |
| 16:08:42 | sean-k-mooney | bauzas: can you do that in vim? i was setting back up my irc client "weechat" | |
| 16:08:57 | bauzas | you can split for sure | |
| 16:10:44 | bauzas | mriedem: I'm just afraid of Xen having different RP structure | |
| 16:11:02 | bauzas | I know we said the drivers are responsible, but that makes us differ | |
| 16:11:37 | sean-k-mooney | bauzas: well im sure hyperv and powervm will likely have different stutures too | |
| 16:13:27 | sean-k-mooney | bauzas: is your consern that you would not know the numa affinty of xen. xen report and supports numa affinity and cpu pinning via libvirt. we just dnot have the supprot in the nova driver | |
| 16:15:03 | bauzas | sean-k-mooney: no, I'm talking of XenServer driver not reporting GPUs like libvirt | |
| 16:15:54 | sean-k-mooney | bauzas: ah ok. i think that is ok provided that does not require use to change the flaovr in some way | |
| 16:16:06 | bauzas | right | |
| 16:16:40 | bauzas | anyway, now /me has to figure out the best way to lookup which GPU is attached to each running instance | |
| 16:22:31 | openstackgerrit | Matt Riedemann proposed openstack/nova master: libvirt: implement reshaper for vgpu https://review.openstack.org/599208 | |
| 16:36:44 | openstackgerrit | John Garbutt proposed openstack/nova-specs master: Add Unified Limits Spec https://review.openstack.org/602201 | |
| 16:41:33 | openstackgerrit | John Garbutt proposed openstack/nova-specs master: Add Unified Limits Spec https://review.openstack.org/602201 | |
| 16:43:41 | johnthetubaguy | melwitt alex_xu I have attempted to capture the unified limits discussion here in here, still not complete, but getting closer: https://review.openstack.org/602201 | |
| 16:50:34 | bauzas | mriedem: thanks for the last update on libvirt reshape, I now see your TODO(sbauza) and the direction it goes | |
| 16:50:56 | bauzas | mriedem: starting from now, I'll test this by changing to create a child per pGPU | |
| 17:46:19 | openstack | Launchpad bug 1792985 in OpenStack Compute (nova) "strict NUMA memory allocation for 4K pages leads to OOM-killer" [Undecided,New] | |
| 17:46:19 | cfriesen | sean-k-mooney: bauzas: curious what you think of https://bugs.launchpad.net/nova/+bug/1792985. Huge bug write-up for what would be a small code change. | |
| 17:53:34 | sean-k-mooney | cfriesen: we have spoken about that in the past | |
| 17:53:50 | sean-k-mooney | cfriesen: its because the reserved memory is global not per numa node | |
| 17:54:28 | cfriesen | sean-k-mooney: I'm talking about actual instance memory, so reserved shouldn't be involved | |
| 17:54:53 | cfriesen | sean-k-mooney: for a floating instance, it can allocate memory from any numa node | |
| 17:54:55 | sean-k-mooney | cfriesen: reserved should because it will prevent schduing to that node so we dont exaust the memory on one numa node | |
| 17:55:15 | sean-k-mooney | cfriesen: when you set a mem page size it created a numa node of 1 | |
| 17:55:30 | cfriesen | sean-k-mooney: if you don't specify a page size, there is no instance numa_topology | |
| 17:55:32 | sean-k-mooney | cfriesen: so its now confied to float over one numa node | |
| 17:56:11 | sean-k-mooney | yes but if you dont specify a page size of 4 k we dont set the numa tune element at all unless you specify some else that created the numa topologyy | |
| 17:57:20 | cfriesen | sean-k-mooney: the scenario is this: one floating instance that can allocate from any numa node. one instance (with pci device maybe) that is pinned to a single numa node and can't allocate because the other instance consumed all the memory from that node. | |
| 17:57:38 | sean-k-mooney | cfriesen: this xml | |
| 17:57:40 | sean-k-mooney | <numatune> | |
| 17:57:43 | sean-k-mooney | <memory mode='strict' nodeset='0'/> | |
| 17:57:45 | sean-k-mooney | <memnode cellid='0' mode='strict' nodeset='0'/> | |
| 17:57:47 | sean-k-mooney | </numatune> | |
| 17:58:01 | sean-k-mooney | will not be genergated unless the guest has a numa toployt of 1 either implictly or implcitly | |
| 17:58:10 | sean-k-mooney | *explictly | |
| 17:58:14 | cfriesen | sean-k-mooney: yes, that's the one with the pci device | |
| 17:58:21 | cfriesen | which is pinned to a numa node | |