Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-11-20
11:52:45 gibi lyarwood: ^^ for the yesterday stable/pike issue
12:10:53 artom stephenfin, heya, could I get you to take another look at https://review.opendev.org/#/c/672595/60?
12:11:11 artom sean-k-mooney, same question for https://review.opendev.org/#/c/691062/ :)
12:17:39 sean-k-mooney artom: sure did you fix up the other pathce below if so ill start form the bottom and work up
12:21:13 artom sean-k-mooney, yeah, the series is ready to go - Joe even merged some of the bottom ones
12:22:19 sean-k-mooney i have not written them yet so how can i be angrey :)
12:23:53 artom Ah, the anger comes later, I see
12:24:51 sean-k-mooney ohly after fighting with mocking for about an hour. although what you want me to test should be strait forward so you will likely get a pass
12:25:51 artom Yep, I know how "unit testing" some of those "functions" can be
12:26:00 artom But you made such a nice clean new one... :)
12:30:39 openstackgerrit Kashyap Chamarthy proposed openstack/nova master: Pick NEXT_MIN libvirt/QEMU versions for "V" release https://review.opendev.org/694821
12:30:39 openstackgerrit Kashyap Chamarthy proposed openstack/nova master: libvirt: Bump MIN_{LIBVIRT,QEMU}_VERSION for "Ussuri" https://review.opendev.org/695056
12:57:43 kashyap stephenfin: Heya, one reason why I also separate out the NEXT_MIN is that it lets us settle on something while we chip away at bump + resulting clean-up
13:03:07 stephenfin kashyap: The clean up is all optional though, right?
13:03:14 stephenfin I mean, it's just dead code
13:03:32 kashyap stephenfin: I mean the immediate clean-up to make the bump work; not the "full clean-up of removing dead constants"
13:03:49 stephenfin ah, I figured it would just be test removal
13:04:16 stephenfin gibi: I assume the plugin isn't versioned?
13:04:27 kashyap :-) Hehe, you've also done part of this clean-up before, maybe you forgot briefly
13:04:41 gibi stephenfin: I think it is not branched
13:04:53 stephenfin kashyap: yup, but they could all be done after the fact
13:04:56 gibi stephenfin: I haven't checked if it is versioned somehow
13:05:24 kashyap stephenfin: Oh, certainly; that's how we do anyway - removing the dead constants after the bump.
13:05:33 stephenfin kashyap: so the only thing we'd have to clean up in that patch is anything that assumes the version is == the old minimum
13:05:44 openstackgerrit Mark Goddard proposed openstack/nova master: Add functional regression test for bug 1853009 https://review.opendev.org/695012
13:05:44 openstack bug 1853009 in OpenStack Compute (nova) "Ironic node rebalance race can lead to missing compute nodes in DB" [Undecided,In progress] https://launchpad.net/bugs/1853009 - Assigned to Mark Goddard (mgoddard)
13:05:44 openstackgerrit Mark Goddard proposed openstack/nova master: Prevent deletion of a compute node belonging to another host https://review.opendev.org/694802
13:05:44 openstackgerrit Mark Goddard proposed openstack/nova master: Clear rebalanced compute nodes from resource tracker https://review.opendev.org/695187
13:05:44 openstackgerrit Mark Goddard proposed openstack/nova master: Invalidate provider tree when compute node disappears https://review.opendev.org/695188
13:05:45 openstackgerrit Mark Goddard proposed openstack/nova master: Fix inactive session error in compute node creation https://review.opendev.org/695189
13:06:17 kashyap stephenfin: That's what I did (or I thought I did) here: https://review.opendev.org/#/c/695056/2
13:06:45 kashyap stephenfin: If you skimmed the commit, don't mistake the listed constants as the ones I've removed -- I just noted that they'll be removed in _future_ patches
13:10:24 mnaser sean-k-mooney: i'm trying to optimize a system to minimize memory latency. i have a total of 8 numa nodes, 4 inside every socket (distances numa<=>numa on same socket is 10, but cross numa in same socket is 12, cross socket+cross numa is 32)
13:11:07 mnaser so its really expensive to do cross-numa cross-socket (and even somewhat to do cross-numa in socket too). i have 12 threads inside a NUMA node too
13:12:08 mnaser i'm thinking if i give the VM 2 NUMA nodes (instead of none), will that end up increasing performance because hopefully the two pair of NUMA nodes end up being close by (rather than cross socket)
13:12:22 stephenfin gibi: okay, I can fix that now
13:12:22 mnaser and maybe at least the VM will be aware that it's crossing numa paths
13:12:24 gibi stephenfin: no tags, no branches in devstack-plugin-ceph so I don't know how to pin
13:12:37 gibi stephenfin: thanks for fixing
13:12:38 stephenfin I'm not going to pin. I'm going to simply check if Python 3 is enabled
13:12:47 gibi stephenfin: good idea
13:12:55 sean-k-mooney mnaser: you using first gen amd eypc cpus yes
13:13:02 mnaser sean-k-mooney: second gen :)
13:13:33 sean-k-mooney second gen have 1 io die per socket and only 1 numa node per socket
13:13:55 sean-k-mooney well by second gen i ment zen2 backed eypc cpus
13:14:08 mnaser really? i actually have BIOS settings to allow me to change an "NPS" settting which makes me select 1/2/4
13:14:09 sean-k-mooney im not sure if they had a refresh based on zen 1
13:14:30 sean-k-mooney that shoudl be there for zen1
13:14:49 mnaser https://developer.amd.com/wp-content/resources/56745_0.80.pdf see "NUMA Nodes per Socket (NPS)" -- i set that to 4 so i have 8 numa nodes in total
13:14:57 sean-k-mooney the recomendation i have gien internally is to disable multiple numa nodes per socket for zen
13:15:32 mnaser really? i've seen that it has resulted in less than ideal NUMA node latency (but a more consistent one inside a socket)
13:16:19 sean-k-mooney hum ok so you are using gen2
13:16:23 mnaser yep
13:16:36 mnaser `numactl -N 0 -m 0 sysbench memory --threads=8 --memory-total-size=32G run` with NPS=4 was *WAY* faster than with NPS=1
13:16:43 sean-k-mooney i might need to reasses
13:16:48 sean-k-mooney yes it would be
13:17:46 sean-k-mooney so basically we were under the impression that gen 2 would only have 1 numa node so we decided not to optimise for the up to 8 numa nodes per socket that can be exposed in gen 1
13:18:11 mnaser gen 2 can even go up to 16 numa nodes, you can make it expose a numa node per each l3 cache
13:18:32 sean-k-mooney mnaser: so to your orgincal question giving the guest 2 numa nodes will imporve the guest performace in general
13:18:55 mnaser sean-k-mooney: https://softwarefactory-project.io/logs/45/16145/11/check/benchmark-combine/ba8ff3a/phoronix-results-cloud-fedora-30-vexxhost-vexxhost-nodepool-tripleo-0002241618/ in this case, much older hardware vs gen 2 epyc.. look at the difference in memory performance
13:19:01 sean-k-mooney so the trade of with that is it limits the vms signifcantly
13:19:11 sean-k-mooney or rather how you create your flavors
13:19:43 sean-k-mooney if you enable all 16
13:20:02 stephenfin gibi: https://review.opendev.org/695191
13:20:16 sean-k-mooney then you vms with a numa topology can only have 3-4 cpus per numa node due to how we do numa in nova
13:20:51 stephenfin gibi: Thank God I included https://review.opendev.org/#/c/692374/6/gate/live_migration/hooks/ceph.sh@11 :D
13:20:58 mnaser i see, i'm ok with 8 if they are living next to each other (and i guess giving the vm the proper awareness means it will know how to address things)
13:21:06 sean-k-mooney e.g. if you expose 1 numa node per ccx it give you the best performance but you are forst to either have multipl numa nodes for larger vms or use no numa features
13:21:16 mnaser i rather 8 because it gives me bigger memory slices and more cores (so i can have an entire VM live in a single NUMA node)
13:21:49 sean-k-mooney yes but that raises an intersting point
13:21:50 gibi stephenfin: deep down you knew that you need to check for python3 enabdled :)
13:22:13 sean-k-mooney we have talked about breaking the 1:1 mapping betwen numa nodes in the guest and the host
13:22:26 sean-k-mooney if we did that we could decuple this
13:22:39 gibi stephenfin: thanks for the fix
13:23:09 sean-k-mooney there are tradoffs but if it was a policy on the guest that would mean you could optimise the hardware config without limiting the slicing
13:26:10 stephenfin artom: Yeah, I started looking at it this morning before lunch. ngl, it's tough going /o\
13:26:26 mnaser sean-k-mooney: right, which is odd because when i run `numastat -cv qemu-kvm` -- it seems like the VM lives in a single node
13:26:37 mnaser but looking at the performance benchmarks, the numbers are pretty bad
13:27:15 mnaser i mean, its not hosting the *entire* VM in a single NUMA node, but you can tell the memory allocated to the VM does live in that node
13:27:26 mnaser https://www.irccloud.com/pastebin/9h0uziTB/
13:27:28 sean-k-mooney mnaser: also damb the intel chip got schooled
13:27:56 mnaser sean-k-mooney: it did in everything except memory bound stuff, i think qpi is just insanely fast
13:28:27 sean-k-mooney qpi was retired its no UPI
13:28:41 sean-k-mooney partly for licensing reasons but also upi is fater the qpi
13:29:30 artom stephenfin, I know :(
13:29:32 sean-k-mooney i thought the inifity fabric in zen processor was ment to be faster however but i dont think we have ever really seen benchmarks of the too fabrics
13:29:36 artom You efforts are appreciated
13:29:48 artom It's a big patch, I tried documenting what I thought was worth it
13:29:49 mnaser sean-k-mooney: the numbers i see make it seem pretty slow.
13:30:11 artom sean-k-mooney, so actually I think you have enough good points on the whitebox series to warrant a respin, if you don't mind looking at it again after
13:30:20 sean-k-mooney mnaser: could you provide us with the output form virsh capablitys by the way
13:30:41 sean-k-mooney i would be interseted in seeing what data we are feeding to nova for it to make decieions
13:31:01 bauzas gibi: I updated the placement audit command + added a FUP for testing the PlacementFixture
13:31:05 sean-k-mooney artom: sure ping me when its done
13:31:29 sean-k-mooney artom: there was noting i would block on but they were all minor imporvements
13:31:31 gibi bauzas: It is on my list for a proper code review because so far I only tried the tool but did not properly read the code
13:31:40 bauzas holy shit, pep8
13:31:55 bauzas gibi: sure, no worries
13:31:55 mnaser https://www.irccloud.com/pastebin/veCa0dq8/
13:32:09 mnaser sean-k-mooney: ^

Earlier   Later