Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-09-17
15:56:28 dansmith sean-k-mooney: not if they look to be on the same NUMA node but aren't, right?
15:56:36 bauzas shit shit shit
15:56:40 dansmith ...
15:56:45 dansmith isn't this the whole discussion we had in denver?
15:56:45 sean-k-mooney dansmith: right if they are split across numa nodes that would be bad
15:56:53 dansmith sean-k-mooney: right
15:57:10 dansmith I'm confused about why we had that discussion and are now having it again :)
15:57:15 sean-k-mooney dansmith: but in general they will appear to be 4 different cards to the guest driver so it wont be able to tell the difference bar numa effects
15:57:20 dansmith but I have to jump on a call in a few minutes
15:57:26 dansmith sean-k-mooney: sure
15:57:29 bauzas dansmith: in https://specs.openstack.org/openstack/nova-specs/specs/queens/implemented/add-support-for-vgpu.html#proposed-change we always thought on GPU types
15:58:02 bauzas dansmith: yes we barely discussed on NUMA affinity for GPUs
15:58:12 dansmith barely/
15:58:15 bauzas dansmith: but in my mind, those were GPU types, and not cards
15:58:27 dansmith what was the whole reason to talk about the reshape happening on the compute node, AFAIK :)
15:58:55 dansmith anyway, /me -> call
15:59:20 johnthetubaguy +1
15:59:47 bauzas mriedem: I just feel I'll necessarly have to help you on the reshaper patch
16:00:25 bauzas but I first need to consider whether a child being a pGPU and not a GPU type would change
16:02:00 sean-k-mooney bauzas: i think the pGPUs would be childeren of the numa nodes with inventoies of vgpu(types)
16:02:30 sean-k-mooney bauzas: we might also want to group pgpus into nvlink aggretates at some point but not now
16:03:58 bauzas sean-k-mooney: shhhhhhhhhhhht
16:04:00 mriedem bauzas: clearly. i only shat this out b/c it seemed trivial. i never made the link between the numa node stuff in the vgpu reshaper discussions.
16:04:12 bauzas sean-k-mooney: some customer could hear us and ask for this feature
16:04:27 bauzas mriedem: and my brain fucked on it, I'm sorry
16:05:09 bauzas that's where I think it's crucial to see the topology now and in the future
16:05:14 bauzas in order to prevent reshapes
16:05:27 bauzas we could implement chidren as GPU types and do reshapes later
16:05:38 bauzas or we could reshape it now and do the necessary work now
16:05:53 bauzas I think given FFU pain, the less reshapes we have, the better it will be
16:06:10 bauzas mriedem: permission to play with your change ?
16:06:36 mriedem bauzas: sure
16:06:46 sean-k-mooney bauzas: /window splitv
16:06:58 sean-k-mooney ... ignore that
16:07:37 bauzas call me vim
16:08:42 sean-k-mooney bauzas: can you do that in vim? i was setting back up my irc client "weechat"
16:08:57 bauzas you can split for sure
16:10:44 bauzas mriedem: I'm just afraid of Xen having different RP structure
16:11:02 bauzas I know we said the drivers are responsible, but that makes us differ
16:11:37 sean-k-mooney bauzas: well im sure hyperv and powervm will likely have different stutures too
16:13:27 sean-k-mooney bauzas: is your consern that you would not know the numa affinty of xen. xen report and supports numa affinity and cpu pinning via libvirt. we just dnot have the supprot in the nova driver
16:15:03 bauzas sean-k-mooney: no, I'm talking of XenServer driver not reporting GPUs like libvirt
16:15:54 sean-k-mooney bauzas: ah ok. i think that is ok provided that does not require use to change the flaovr in some way
16:16:06 bauzas right
16:16:40 bauzas anyway, now /me has to figure out the best way to lookup which GPU is attached to each running instance
16:22:31 openstackgerrit Matt Riedemann proposed openstack/nova master: libvirt: implement reshaper for vgpu https://review.openstack.org/599208
16:36:44 openstackgerrit John Garbutt proposed openstack/nova-specs master: Add Unified Limits Spec https://review.openstack.org/602201
16:41:33 openstackgerrit John Garbutt proposed openstack/nova-specs master: Add Unified Limits Spec https://review.openstack.org/602201
16:43:41 johnthetubaguy melwitt alex_xu I have attempted to capture the unified limits discussion here in here, still not complete, but getting closer: https://review.openstack.org/602201
16:50:34 bauzas mriedem: thanks for the last update on libvirt reshape, I now see your TODO(sbauza) and the direction it goes
16:50:56 bauzas mriedem: starting from now, I'll test this by changing to create a child per pGPU
17:46:19 cfriesen sean-k-mooney: bauzas: curious what you think of https://bugs.launchpad.net/nova/+bug/1792985. Huge bug write-up for what would be a small code change.
17:46:19 openstack Launchpad bug 1792985 in OpenStack Compute (nova) "strict NUMA memory allocation for 4K pages leads to OOM-killer" [Undecided,New]
17:53:34 sean-k-mooney cfriesen: we have spoken about that in the past
17:53:50 sean-k-mooney cfriesen: its because the reserved memory is global not per numa node
17:54:28 cfriesen sean-k-mooney: I'm talking about actual instance memory, so reserved shouldn't be involved
17:54:53 cfriesen sean-k-mooney: for a floating instance, it can allocate memory from any numa node
17:54:55 sean-k-mooney cfriesen: reserved should because it will prevent schduing to that node so we dont exaust the memory on one numa node
17:55:15 sean-k-mooney cfriesen: when you set a mem page size it created a numa node of 1
17:55:30 cfriesen sean-k-mooney: if you don't specify a page size, there is no instance numa_topology
17:55:32 sean-k-mooney cfriesen: so its now confied to float over one numa node
17:56:11 sean-k-mooney yes but if you dont specify a page size of 4 k we dont set the numa tune element at all unless you specify some else that created the numa topologyy
17:57:20 cfriesen sean-k-mooney: the scenario is this: one floating instance that can allocate from any numa node. one instance (with pci device maybe) that is pinned to a single numa node and can't allocate because the other instance consumed all the memory from that node.
17:57:38 sean-k-mooney cfriesen: this xml
17:57:40 sean-k-mooney <numatune>
17:57:43 sean-k-mooney <memory mode='strict' nodeset='0'/>
17:57:45 sean-k-mooney <memnode cellid='0' mode='strict' nodeset='0'/>
17:57:47 sean-k-mooney </numatune>
17:58:01 sean-k-mooney will not be genergated unless the guest has a numa toployt of 1 either implictly or implcitly
17:58:10 sean-k-mooney *explictly
17:58:14 cfriesen sean-k-mooney: yes, that's the one with the pci device
17:58:21 cfriesen which is pinned to a numa node
17:58:40 sean-k-mooney right so the behavor is correct
17:59:01 cfriesen no, because the "floating" instance may have consumed all the memory from that node
17:59:11 sean-k-mooney the issue is we do not have a per numa reserved memory so we can exaust one numa node and trigger oom
17:59:28 cfriesen so either we restrict floating instances to a single numa node, or else we can't use "strict" for the "memory mode" line
18:00:07 sean-k-mooney cfriesen: we already no you cannot mix numa affined instances with non numa instances on a singel host for this exact reason
18:00:30 cfriesen sean-k-mooney: but you can, because you can specify a pci device which implicitly makes it numa-affined
18:00:46 sean-k-mooney cfriesen: we could fix this but we tell operators that this is not ok and use host aggreate to seperate these flavors today
18:01:03 sean-k-mooney cfriesen: you mean by neutron sriov ports
18:01:09 sean-k-mooney cfriesen: that is true
18:02:31 sean-k-mooney cfriesen: i would be ok with introduceing the hw:numa_mem_policy extra spec that was originally proposed for hyper v to allowchanging the mode but it messes up our accounting in placement if we do
18:04:23 sean-k-mooney that said another way to resolve this would be to allways specify a page size so your floating instances would be hw:mem_page_size=small or 4k
18:04:35 cfriesen sean-k-mooney: I was thinking we could literally just remove the "memory mode=strict" line. That would make us use the linux default, which is local allocation.
18:05:09 sean-k-mooney cfriesen: if we remove it for 4k why would we not allow it for any page size
18:05:09 cfriesen sean-k-mooney: explicitly specifying page size would limit them to a single numa node, which we had before but removed.
18:05:45 cfriesen sean-k-mooney: adding hugepages explicitly results in a numa-topology and being restricted to a single node.
18:06:04 sean-k-mooney cfriesen: in nova yes but qemu does not require that
18:06:42 cfriesen sean-k-mooney: I suspect that people using hugepages care about performance, while people using the default pagesize and floating instances don't. :)
18:07:08 sean-k-mooney cfriesen: so the defautl is stict and you opt out
18:07:40 sean-k-mooney cfriesen: that argument will never win with me because i argued that none of the implit behavior should have existied for the start
18:09:09 cfriesen sean-k-mooney: I'd rather have the default be something that doesn't lead to the OOM-killer. Then if you've engineered the cluster to avoid the problem you can opt-in to "strict"
18:10:01 sean-k-mooney cfriesen: prefer will break us one memory is reported in placement as we will nologer no where the memory was consumed form
18:10:23 cfriesen sean-k-mooney: we already don't know where it's consumed from. we just think we do. :)
18:10:36 cfriesen for the floating instances, at least
18:10:42 sean-k-mooney cfriesen: we know when we explcitly state a page size
18:10:47 cfriesen sean-k-mooney: yes
18:11:13 sean-k-mooney for floating instance we dont touch the numa toplogy blob.
18:11:17 sean-k-mooney so its fine
18:11:41 sean-k-mooney the numa toplogy blob heals based on the periodic task

Earlier   Later