| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-11-20 | |||
| 13:16:36 | mnaser | `numactl -N 0 -m 0 sysbench memory --threads=8 --memory-total-size=32G run` with NPS=4 was *WAY* faster than with NPS=1 | |
| 13:16:43 | sean-k-mooney | i might need to reasses | |
| 13:16:48 | sean-k-mooney | yes it would be | |
| 13:17:46 | sean-k-mooney | so basically we were under the impression that gen 2 would only have 1 numa node so we decided not to optimise for the up to 8 numa nodes per socket that can be exposed in gen 1 | |
| 13:18:11 | mnaser | gen 2 can even go up to 16 numa nodes, you can make it expose a numa node per each l3 cache | |
| 13:18:32 | sean-k-mooney | mnaser: so to your orgincal question giving the guest 2 numa nodes will imporve the guest performace in general | |
| 13:18:55 | mnaser | sean-k-mooney: https://softwarefactory-project.io/logs/45/16145/11/check/benchmark-combine/ba8ff3a/phoronix-results-cloud-fedora-30-vexxhost-vexxhost-nodepool-tripleo-0002241618/ in this case, much older hardware vs gen 2 epyc.. look at the difference in memory performance | |
| 13:19:01 | sean-k-mooney | so the trade of with that is it limits the vms signifcantly | |
| 13:19:11 | sean-k-mooney | or rather how you create your flavors | |
| 13:19:43 | sean-k-mooney | if you enable all 16 | |
| 13:20:02 | stephenfin | gibi: https://review.opendev.org/695191 | |
| 13:20:16 | sean-k-mooney | then you vms with a numa topology can only have 3-4 cpus per numa node due to how we do numa in nova | |
| 13:20:51 | stephenfin | gibi: Thank God I included https://review.opendev.org/#/c/692374/6/gate/live_migration/hooks/ceph.sh@11 :D | |
| 13:20:58 | mnaser | i see, i'm ok with 8 if they are living next to each other (and i guess giving the vm the proper awareness means it will know how to address things) | |
| 13:21:06 | sean-k-mooney | e.g. if you expose 1 numa node per ccx it give you the best performance but you are forst to either have multipl numa nodes for larger vms or use no numa features | |
| 13:21:16 | mnaser | i rather 8 because it gives me bigger memory slices and more cores (so i can have an entire VM live in a single NUMA node) | |
| 13:21:49 | sean-k-mooney | yes but that raises an intersting point | |
| 13:21:50 | gibi | stephenfin: deep down you knew that you need to check for python3 enabdled :) | |
| 13:22:13 | sean-k-mooney | we have talked about breaking the 1:1 mapping betwen numa nodes in the guest and the host | |
| 13:22:26 | sean-k-mooney | if we did that we could decuple this | |
| 13:22:39 | gibi | stephenfin: thanks for the fix | |
| 13:23:09 | sean-k-mooney | there are tradoffs but if it was a policy on the guest that would mean you could optimise the hardware config without limiting the slicing | |
| 13:26:10 | stephenfin | artom: Yeah, I started looking at it this morning before lunch. ngl, it's tough going /o\ | |
| 13:26:26 | mnaser | sean-k-mooney: right, which is odd because when i run `numastat -cv qemu-kvm` -- it seems like the VM lives in a single node | |
| 13:26:37 | mnaser | but looking at the performance benchmarks, the numbers are pretty bad | |
| 13:27:15 | mnaser | i mean, its not hosting the *entire* VM in a single NUMA node, but you can tell the memory allocated to the VM does live in that node | |
| 13:27:26 | mnaser | https://www.irccloud.com/pastebin/9h0uziTB/ | |
| 13:27:28 | sean-k-mooney | mnaser: also damb the intel chip got schooled | |
| 13:27:56 | mnaser | sean-k-mooney: it did in everything except memory bound stuff, i think qpi is just insanely fast | |
| 13:28:27 | sean-k-mooney | qpi was retired its no UPI | |
| 13:28:41 | sean-k-mooney | partly for licensing reasons but also upi is fater the qpi | |
| 13:29:30 | artom | stephenfin, I know :( | |
| 13:29:32 | sean-k-mooney | i thought the inifity fabric in zen processor was ment to be faster however but i dont think we have ever really seen benchmarks of the too fabrics | |
| 13:29:36 | artom | You efforts are appreciated | |
| 13:29:48 | artom | It's a big patch, I tried documenting what I thought was worth it | |
| 13:29:49 | mnaser | sean-k-mooney: the numbers i see make it seem pretty slow. | |
| 13:30:11 | artom | sean-k-mooney, so actually I think you have enough good points on the whitebox series to warrant a respin, if you don't mind looking at it again after | |
| 13:30:20 | sean-k-mooney | mnaser: could you provide us with the output form virsh capablitys by the way | |
| 13:30:41 | sean-k-mooney | i would be interseted in seeing what data we are feeding to nova for it to make decieions | |
| 13:31:01 | bauzas | gibi: I updated the placement audit command + added a FUP for testing the PlacementFixture | |
| 13:31:05 | sean-k-mooney | artom: sure ping me when its done | |
| 13:31:29 | sean-k-mooney | artom: there was noting i would block on but they were all minor imporvements | |
| 13:31:31 | gibi | bauzas: It is on my list for a proper code review because so far I only tried the tool but did not properly read the code | |
| 13:31:40 | bauzas | holy shit, pep8 | |
| 13:31:55 | mnaser | https://www.irccloud.com/pastebin/veCa0dq8/ | |
| 13:31:55 | bauzas | gibi: sure, no worries | |
| 13:32:09 | mnaser | sean-k-mooney: ^ | |
| 13:32:10 | bauzas | gibi: I provided a functest for verifying the deletion, it should work | |
| 13:33:22 | bauzas | gibi: given your notes, I think I can try to have better logs | |
| 13:34:10 | gibi | bauzas: the log improvement is not something I will -1 but If you have time then better logs are always welcome | |
| 13:34:22 | bauzas | gibi: I think it's important | |
| 13:34:32 | bauzas | so I would understand a -1 for this | |
| 13:35:45 | sean-k-mooney | i need to jump on a call but im seeing som intersting thing in that output notably the way the cache is reported | |
| 13:37:20 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/train: Don't delete compute node, when deleting service other than nova-compute https://review.opendev.org/695145 | |
| 13:37:21 | sean-k-mooney | mnaser: i dont know if you remember but i have spoke about the need to model cpus per cache node and cache nodes per numa node in the past at the denver ptg | |
| 13:39:24 | sean-k-mooney | mnaser: noone else seams to be on the call so ill expand on that | |
| 13:40:00 | sean-k-mooney | mnaser: cache banks 1 and 2 both part of numa node 1 in that config | |
| 13:40:28 | sean-k-mooney | it looks like the 24 core sku has 3 active cpus per ccx and hence per cache region | |
| 13:40:55 | sean-k-mooney | so your 8 core vm needs to be spread across 3 cache regions and 2 numa nodes | |
| 13:40:56 | mnaser | sean-k-mooney: it does, when i enable numa node per l3 cache, i end up with 6 threads in a numa node | |
| 13:41:43 | sean-k-mooney | oh ya i guess you could fit it on 2 cache regions and 1 numa node if you are using hyperthreads | |
| 13:42:28 | sean-k-mooney | mnaser: so rather then select core fomr the same numa node if we wanted to optimise in nova we would want to select cores in the same cache region first | |
| 13:42:49 | sean-k-mooney | we can actuly model the cache toploy in libvirt too and expose that to the guest | |
| 13:43:23 | sean-k-mooney | mnaser: it would be interesting to see how a 6 core vm faired against the 8 core vm | |
| 13:43:46 | mnaser | yeah, i read a bit about that. i'm wondering if performance wise, exposing numa node per l3 cache and then doing 2 numa nodes in a VM would result in good overall performance | |
| 13:44:05 | mnaser | you'd have 4 cores each sitting in a numa node and hopefully the guest is smart enough to understand the distances involved | |
| 13:44:49 | sean-k-mooney | yes although again nova does not today fully understand the hierachy | |
| 13:45:03 | sean-k-mooney | e.g. nova could select 2 numa nodes form differnet sockets | |
| 13:45:21 | sean-k-mooney | ideally it woudl select 2 form the same socket to miniums the numa distance | |
| 13:45:35 | sean-k-mooney | that is actully just an optimization of the current behavior | |
| 13:45:43 | sean-k-mooney | rahter then a change | |
| 13:45:49 | mnaser | sean-k-mooney: right but at least the guest will be aware there is possible latency | |
| 13:45:54 | sean-k-mooney | yep | |
| 13:45:55 | mnaser | yep, indeed, its not optimal, but its an improvement | |
| 13:46:46 | sean-k-mooney | so the reason i suggested disabling the multiple numa nodes in zen 1 was our pms insisting vnf dont understand numa | |
| 13:47:08 | sean-k-mooney | after 5+ years of eveyone explaiming what numa is to vnf vendors i dont really buy that | |
| 13:47:57 | mnaser | im wondering which would be better: NPS=4 + no NUMA per L3 + 1 numa node .. vs .. NPS=4 + NUMA per L3 + 2 numa nodes -- im thinking the latter will likely be much faster (or at least the VM will be 'smarter' at understanding the topology.. somewhat) | |
| 13:48:26 | mnaser | it might be unpredictable because the NUMA node might have a distance of 32, or 12, or 10 .. but at least it might know there is distance | |
| 13:48:40 | sean-k-mooney | the later i think would optimize the memory latency | |
| 13:50:21 | mnaser | im gathering some benchmark info | |
| 13:50:29 | mnaser | and then ill try the two numa node + l3 per numa | |
| 13:50:32 | sean-k-mooney | exposing NUMA per L3 should give the best performance it just requires your vms to have multiple numa nodes if they have more then 3-4 cores or >32-64G of ram | |
| 13:50:53 | mnaser | yep, most of the VMs will be 8 core / 8 gb | |
| 13:51:10 | mnaser | also FWIW those aren't dedicated cores | |
| 13:51:39 | sean-k-mooney | sure you just setting hw:numa_nodes=2 but not enabling pinning | |
| 13:51:48 | aarents | hi dansmith, can you confirm that the last update is fine for you ? https://review.opendev.org/#/c/670000 | |
| 13:51:57 | mnaser | yep, os that way im thinknig nova should do 4 cores per NUMA, 4gb in each numa node | |
| 13:52:03 | sean-k-mooney | if you do that you should also set hw:mem_page_size to either large or small | |
| 13:52:27 | sean-k-mooney | mnaser: yes it would | |
| 13:53:00 | sean-k-mooney | the reason for doing hw:mem_page_size by the way is you want to enable novas numa aware memory tracking | |
| 13:53:24 | sean-k-mooney | if your dont want to use hugepages set it to "small" | |
| 13:53:39 | sean-k-mooney | that will use 4k pages | |
| 13:53:46 | mnaser | sean-k-mooney: i think huge pages is soemthing id want, but i think transparent huge pages are enabled right now | |
| 13:54:26 | sean-k-mooney | right but you should still set hw:mem_page_size=small if you have a numa toplogy and are not using explcit hugepages | |
| 13:54:33 | sean-k-mooney | if you dont you can get OOM events | |
| 13:54:45 | mnaser | i think id probably want large anyways, host has 512G memory and plenty of reserved memoryt | |
| 13:55:12 | sean-k-mooney | ya if you dont allow memory oversubsription then there is no reason not to use hugepages | |
| 13:55:32 | mnaser | ya nope no oversubscription | |
| 13:58:52 | mnaser | epyc rome numbers https://www.irccloud.com/pastebin/3bgON0HX/ | |
| 13:59:18 | mnaser | intel (but much older hardware) https://www.irccloud.com/pastebin/6M7B4QLY/ | |