| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-11-20 | |||
| 13:05:24 | kashyap | stephenfin: Oh, certainly; that's how we do anyway - removing the dead constants after the bump. | |
| 13:05:33 | stephenfin | kashyap: so the only thing we'd have to clean up in that patch is anything that assumes the version is == the old minimum | |
| 13:05:44 | openstackgerrit | Mark Goddard proposed openstack/nova master: Add functional regression test for bug 1853009 https://review.opendev.org/695012 | |
| 13:05:44 | openstack | bug 1853009 in OpenStack Compute (nova) "Ironic node rebalance race can lead to missing compute nodes in DB" [Undecided,In progress] https://launchpad.net/bugs/1853009 - Assigned to Mark Goddard (mgoddard) | |
| 13:05:44 | openstackgerrit | Mark Goddard proposed openstack/nova master: Prevent deletion of a compute node belonging to another host https://review.opendev.org/694802 | |
| 13:05:44 | openstackgerrit | Mark Goddard proposed openstack/nova master: Clear rebalanced compute nodes from resource tracker https://review.opendev.org/695187 | |
| 13:05:44 | openstackgerrit | Mark Goddard proposed openstack/nova master: Invalidate provider tree when compute node disappears https://review.opendev.org/695188 | |
| 13:05:45 | openstackgerrit | Mark Goddard proposed openstack/nova master: Fix inactive session error in compute node creation https://review.opendev.org/695189 | |
| 13:06:17 | kashyap | stephenfin: That's what I did (or I thought I did) here: https://review.opendev.org/#/c/695056/2 | |
| 13:06:45 | kashyap | stephenfin: If you skimmed the commit, don't mistake the listed constants as the ones I've removed -- I just noted that they'll be removed in _future_ patches | |
| 13:10:24 | mnaser | sean-k-mooney: i'm trying to optimize a system to minimize memory latency. i have a total of 8 numa nodes, 4 inside every socket (distances numa<=>numa on same socket is 10, but cross numa in same socket is 12, cross socket+cross numa is 32) | |
| 13:11:07 | mnaser | so its really expensive to do cross-numa cross-socket (and even somewhat to do cross-numa in socket too). i have 12 threads inside a NUMA node too | |
| 13:12:08 | mnaser | i'm thinking if i give the VM 2 NUMA nodes (instead of none), will that end up increasing performance because hopefully the two pair of NUMA nodes end up being close by (rather than cross socket) | |
| 13:12:22 | stephenfin | gibi: okay, I can fix that now | |
| 13:12:22 | mnaser | and maybe at least the VM will be aware that it's crossing numa paths | |
| 13:12:24 | gibi | stephenfin: no tags, no branches in devstack-plugin-ceph so I don't know how to pin | |
| 13:12:37 | gibi | stephenfin: thanks for fixing | |
| 13:12:38 | stephenfin | I'm not going to pin. I'm going to simply check if Python 3 is enabled | |
| 13:12:47 | gibi | stephenfin: good idea | |
| 13:12:55 | sean-k-mooney | mnaser: you using first gen amd eypc cpus yes | |
| 13:13:02 | mnaser | sean-k-mooney: second gen :) | |
| 13:13:33 | sean-k-mooney | second gen have 1 io die per socket and only 1 numa node per socket | |
| 13:13:55 | sean-k-mooney | well by second gen i ment zen2 backed eypc cpus | |
| 13:14:08 | mnaser | really? i actually have BIOS settings to allow me to change an "NPS" settting which makes me select 1/2/4 | |
| 13:14:09 | sean-k-mooney | im not sure if they had a refresh based on zen 1 | |
| 13:14:30 | sean-k-mooney | that shoudl be there for zen1 | |
| 13:14:49 | mnaser | https://developer.amd.com/wp-content/resources/56745_0.80.pdf see "NUMA Nodes per Socket (NPS)" -- i set that to 4 so i have 8 numa nodes in total | |
| 13:14:57 | sean-k-mooney | the recomendation i have gien internally is to disable multiple numa nodes per socket for zen | |
| 13:15:32 | mnaser | really? i've seen that it has resulted in less than ideal NUMA node latency (but a more consistent one inside a socket) | |
| 13:16:19 | sean-k-mooney | hum ok so you are using gen2 | |
| 13:16:23 | mnaser | yep | |
| 13:16:36 | mnaser | `numactl -N 0 -m 0 sysbench memory --threads=8 --memory-total-size=32G run` with NPS=4 was *WAY* faster than with NPS=1 | |
| 13:16:43 | sean-k-mooney | i might need to reasses | |
| 13:16:48 | sean-k-mooney | yes it would be | |
| 13:17:46 | sean-k-mooney | so basically we were under the impression that gen 2 would only have 1 numa node so we decided not to optimise for the up to 8 numa nodes per socket that can be exposed in gen 1 | |
| 13:18:11 | mnaser | gen 2 can even go up to 16 numa nodes, you can make it expose a numa node per each l3 cache | |
| 13:18:32 | sean-k-mooney | mnaser: so to your orgincal question giving the guest 2 numa nodes will imporve the guest performace in general | |
| 13:18:55 | mnaser | sean-k-mooney: https://softwarefactory-project.io/logs/45/16145/11/check/benchmark-combine/ba8ff3a/phoronix-results-cloud-fedora-30-vexxhost-vexxhost-nodepool-tripleo-0002241618/ in this case, much older hardware vs gen 2 epyc.. look at the difference in memory performance | |
| 13:19:01 | sean-k-mooney | so the trade of with that is it limits the vms signifcantly | |
| 13:19:11 | sean-k-mooney | or rather how you create your flavors | |
| 13:19:43 | sean-k-mooney | if you enable all 16 | |
| 13:20:02 | stephenfin | gibi: https://review.opendev.org/695191 | |
| 13:20:16 | sean-k-mooney | then you vms with a numa topology can only have 3-4 cpus per numa node due to how we do numa in nova | |
| 13:20:51 | stephenfin | gibi: Thank God I included https://review.opendev.org/#/c/692374/6/gate/live_migration/hooks/ceph.sh@11 :D | |
| 13:20:58 | mnaser | i see, i'm ok with 8 if they are living next to each other (and i guess giving the vm the proper awareness means it will know how to address things) | |
| 13:21:06 | sean-k-mooney | e.g. if you expose 1 numa node per ccx it give you the best performance but you are forst to either have multipl numa nodes for larger vms or use no numa features | |
| 13:21:16 | mnaser | i rather 8 because it gives me bigger memory slices and more cores (so i can have an entire VM live in a single NUMA node) | |
| 13:21:49 | sean-k-mooney | yes but that raises an intersting point | |
| 13:21:50 | gibi | stephenfin: deep down you knew that you need to check for python3 enabdled :) | |
| 13:22:13 | sean-k-mooney | we have talked about breaking the 1:1 mapping betwen numa nodes in the guest and the host | |
| 13:22:26 | sean-k-mooney | if we did that we could decuple this | |
| 13:22:39 | gibi | stephenfin: thanks for the fix | |
| 13:23:09 | sean-k-mooney | there are tradoffs but if it was a policy on the guest that would mean you could optimise the hardware config without limiting the slicing | |
| 13:26:10 | stephenfin | artom: Yeah, I started looking at it this morning before lunch. ngl, it's tough going /o\ | |
| 13:26:26 | mnaser | sean-k-mooney: right, which is odd because when i run `numastat -cv qemu-kvm` -- it seems like the VM lives in a single node | |
| 13:26:37 | mnaser | but looking at the performance benchmarks, the numbers are pretty bad | |
| 13:27:15 | mnaser | i mean, its not hosting the *entire* VM in a single NUMA node, but you can tell the memory allocated to the VM does live in that node | |
| 13:27:26 | mnaser | https://www.irccloud.com/pastebin/9h0uziTB/ | |
| 13:27:28 | sean-k-mooney | mnaser: also damb the intel chip got schooled | |
| 13:27:56 | mnaser | sean-k-mooney: it did in everything except memory bound stuff, i think qpi is just insanely fast | |
| 13:28:27 | sean-k-mooney | qpi was retired its no UPI | |
| 13:28:41 | sean-k-mooney | partly for licensing reasons but also upi is fater the qpi | |
| 13:29:30 | artom | stephenfin, I know :( | |
| 13:29:32 | sean-k-mooney | i thought the inifity fabric in zen processor was ment to be faster however but i dont think we have ever really seen benchmarks of the too fabrics | |
| 13:29:36 | artom | You efforts are appreciated | |
| 13:29:48 | artom | It's a big patch, I tried documenting what I thought was worth it | |
| 13:29:49 | mnaser | sean-k-mooney: the numbers i see make it seem pretty slow. | |
| 13:30:11 | artom | sean-k-mooney, so actually I think you have enough good points on the whitebox series to warrant a respin, if you don't mind looking at it again after | |
| 13:30:20 | sean-k-mooney | mnaser: could you provide us with the output form virsh capablitys by the way | |
| 13:30:41 | sean-k-mooney | i would be interseted in seeing what data we are feeding to nova for it to make decieions | |
| 13:31:01 | bauzas | gibi: I updated the placement audit command + added a FUP for testing the PlacementFixture | |
| 13:31:05 | sean-k-mooney | artom: sure ping me when its done | |
| 13:31:29 | sean-k-mooney | artom: there was noting i would block on but they were all minor imporvements | |
| 13:31:31 | gibi | bauzas: It is on my list for a proper code review because so far I only tried the tool but did not properly read the code | |
| 13:31:40 | bauzas | holy shit, pep8 | |
| 13:31:55 | bauzas | gibi: sure, no worries | |
| 13:31:55 | mnaser | https://www.irccloud.com/pastebin/veCa0dq8/ | |
| 13:32:09 | mnaser | sean-k-mooney: ^ | |
| 13:32:10 | bauzas | gibi: I provided a functest for verifying the deletion, it should work | |
| 13:33:22 | bauzas | gibi: given your notes, I think I can try to have better logs | |
| 13:34:10 | gibi | bauzas: the log improvement is not something I will -1 but If you have time then better logs are always welcome | |
| 13:34:22 | bauzas | gibi: I think it's important | |
| 13:34:32 | bauzas | so I would understand a -1 for this | |
| 13:35:45 | sean-k-mooney | i need to jump on a call but im seeing som intersting thing in that output notably the way the cache is reported | |
| 13:37:20 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/train: Don't delete compute node, when deleting service other than nova-compute https://review.opendev.org/695145 | |
| 13:37:21 | sean-k-mooney | mnaser: i dont know if you remember but i have spoke about the need to model cpus per cache node and cache nodes per numa node in the past at the denver ptg | |
| 13:39:24 | sean-k-mooney | mnaser: noone else seams to be on the call so ill expand on that | |
| 13:40:00 | sean-k-mooney | mnaser: cache banks 1 and 2 both part of numa node 1 in that config | |
| 13:40:28 | sean-k-mooney | it looks like the 24 core sku has 3 active cpus per ccx and hence per cache region | |
| 13:40:55 | sean-k-mooney | so your 8 core vm needs to be spread across 3 cache regions and 2 numa nodes | |
| 13:40:56 | mnaser | sean-k-mooney: it does, when i enable numa node per l3 cache, i end up with 6 threads in a numa node | |
| 13:41:43 | sean-k-mooney | oh ya i guess you could fit it on 2 cache regions and 1 numa node if you are using hyperthreads | |
| 13:42:28 | sean-k-mooney | mnaser: so rather then select core fomr the same numa node if we wanted to optimise in nova we would want to select cores in the same cache region first | |
| 13:42:49 | sean-k-mooney | we can actuly model the cache toploy in libvirt too and expose that to the guest | |
| 13:43:23 | sean-k-mooney | mnaser: it would be interesting to see how a 6 core vm faired against the 8 core vm | |
| 13:43:46 | mnaser | yeah, i read a bit about that. i'm wondering if performance wise, exposing numa node per l3 cache and then doing 2 numa nodes in a VM would result in good overall performance | |
| 13:44:05 | mnaser | you'd have 4 cores each sitting in a numa node and hopefully the guest is smart enough to understand the distances involved | |
| 13:44:49 | sean-k-mooney | yes although again nova does not today fully understand the hierachy | |
| 13:45:03 | sean-k-mooney | e.g. nova could select 2 numa nodes form differnet sockets | |
| 13:45:21 | sean-k-mooney | ideally it woudl select 2 form the same socket to miniums the numa distance | |