| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-11-20 | |||
| 13:26:10 | stephenfin | artom: Yeah, I started looking at it this morning before lunch. ngl, it's tough going /o\ | |
| 13:26:26 | mnaser | sean-k-mooney: right, which is odd because when i run `numastat -cv qemu-kvm` -- it seems like the VM lives in a single node | |
| 13:26:37 | mnaser | but looking at the performance benchmarks, the numbers are pretty bad | |
| 13:27:15 | mnaser | i mean, its not hosting the *entire* VM in a single NUMA node, but you can tell the memory allocated to the VM does live in that node | |
| 13:27:26 | mnaser | https://www.irccloud.com/pastebin/9h0uziTB/ | |
| 13:27:28 | sean-k-mooney | mnaser: also damb the intel chip got schooled | |
| 13:27:56 | mnaser | sean-k-mooney: it did in everything except memory bound stuff, i think qpi is just insanely fast | |
| 13:28:27 | sean-k-mooney | qpi was retired its no UPI | |
| 13:28:41 | sean-k-mooney | partly for licensing reasons but also upi is fater the qpi | |
| 13:29:30 | artom | stephenfin, I know :( | |
| 13:29:32 | sean-k-mooney | i thought the inifity fabric in zen processor was ment to be faster however but i dont think we have ever really seen benchmarks of the too fabrics | |
| 13:29:36 | artom | You efforts are appreciated | |
| 13:29:48 | artom | It's a big patch, I tried documenting what I thought was worth it | |
| 13:29:49 | mnaser | sean-k-mooney: the numbers i see make it seem pretty slow. | |
| 13:30:11 | artom | sean-k-mooney, so actually I think you have enough good points on the whitebox series to warrant a respin, if you don't mind looking at it again after | |
| 13:30:20 | sean-k-mooney | mnaser: could you provide us with the output form virsh capablitys by the way | |
| 13:30:41 | sean-k-mooney | i would be interseted in seeing what data we are feeding to nova for it to make decieions | |
| 13:31:01 | bauzas | gibi: I updated the placement audit command + added a FUP for testing the PlacementFixture | |
| 13:31:05 | sean-k-mooney | artom: sure ping me when its done | |
| 13:31:29 | sean-k-mooney | artom: there was noting i would block on but they were all minor imporvements | |
| 13:31:31 | gibi | bauzas: It is on my list for a proper code review because so far I only tried the tool but did not properly read the code | |
| 13:31:40 | bauzas | holy shit, pep8 | |
| 13:31:55 | bauzas | gibi: sure, no worries | |
| 13:31:55 | mnaser | https://www.irccloud.com/pastebin/veCa0dq8/ | |
| 13:32:09 | mnaser | sean-k-mooney: ^ | |
| 13:32:10 | bauzas | gibi: I provided a functest for verifying the deletion, it should work | |
| 13:33:22 | bauzas | gibi: given your notes, I think I can try to have better logs | |
| 13:34:10 | gibi | bauzas: the log improvement is not something I will -1 but If you have time then better logs are always welcome | |
| 13:34:22 | bauzas | gibi: I think it's important | |
| 13:34:32 | bauzas | so I would understand a -1 for this | |
| 13:35:45 | sean-k-mooney | i need to jump on a call but im seeing som intersting thing in that output notably the way the cache is reported | |
| 13:37:20 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/train: Don't delete compute node, when deleting service other than nova-compute https://review.opendev.org/695145 | |
| 13:37:21 | sean-k-mooney | mnaser: i dont know if you remember but i have spoke about the need to model cpus per cache node and cache nodes per numa node in the past at the denver ptg | |
| 13:39:24 | sean-k-mooney | mnaser: noone else seams to be on the call so ill expand on that | |
| 13:40:00 | sean-k-mooney | mnaser: cache banks 1 and 2 both part of numa node 1 in that config | |
| 13:40:28 | sean-k-mooney | it looks like the 24 core sku has 3 active cpus per ccx and hence per cache region | |
| 13:40:55 | sean-k-mooney | so your 8 core vm needs to be spread across 3 cache regions and 2 numa nodes | |
| 13:40:56 | mnaser | sean-k-mooney: it does, when i enable numa node per l3 cache, i end up with 6 threads in a numa node | |
| 13:41:43 | sean-k-mooney | oh ya i guess you could fit it on 2 cache regions and 1 numa node if you are using hyperthreads | |
| 13:42:28 | sean-k-mooney | mnaser: so rather then select core fomr the same numa node if we wanted to optimise in nova we would want to select cores in the same cache region first | |
| 13:42:49 | sean-k-mooney | we can actuly model the cache toploy in libvirt too and expose that to the guest | |
| 13:43:23 | sean-k-mooney | mnaser: it would be interesting to see how a 6 core vm faired against the 8 core vm | |
| 13:43:46 | mnaser | yeah, i read a bit about that. i'm wondering if performance wise, exposing numa node per l3 cache and then doing 2 numa nodes in a VM would result in good overall performance | |
| 13:44:05 | mnaser | you'd have 4 cores each sitting in a numa node and hopefully the guest is smart enough to understand the distances involved | |
| 13:44:49 | sean-k-mooney | yes although again nova does not today fully understand the hierachy | |
| 13:45:03 | sean-k-mooney | e.g. nova could select 2 numa nodes form differnet sockets | |
| 13:45:21 | sean-k-mooney | ideally it woudl select 2 form the same socket to miniums the numa distance | |
| 13:45:35 | sean-k-mooney | that is actully just an optimization of the current behavior | |
| 13:45:43 | sean-k-mooney | rahter then a change | |
| 13:45:49 | mnaser | sean-k-mooney: right but at least the guest will be aware there is possible latency | |
| 13:45:54 | sean-k-mooney | yep | |
| 13:45:55 | mnaser | yep, indeed, its not optimal, but its an improvement | |
| 13:46:46 | sean-k-mooney | so the reason i suggested disabling the multiple numa nodes in zen 1 was our pms insisting vnf dont understand numa | |
| 13:47:08 | sean-k-mooney | after 5+ years of eveyone explaiming what numa is to vnf vendors i dont really buy that | |
| 13:47:57 | mnaser | im wondering which would be better: NPS=4 + no NUMA per L3 + 1 numa node .. vs .. NPS=4 + NUMA per L3 + 2 numa nodes -- im thinking the latter will likely be much faster (or at least the VM will be 'smarter' at understanding the topology.. somewhat) | |
| 13:48:26 | mnaser | it might be unpredictable because the NUMA node might have a distance of 32, or 12, or 10 .. but at least it might know there is distance | |
| 13:48:40 | sean-k-mooney | the later i think would optimize the memory latency | |
| 13:50:21 | mnaser | im gathering some benchmark info | |
| 13:50:29 | mnaser | and then ill try the two numa node + l3 per numa | |
| 13:50:32 | sean-k-mooney | exposing NUMA per L3 should give the best performance it just requires your vms to have multiple numa nodes if they have more then 3-4 cores or >32-64G of ram | |
| 13:50:53 | mnaser | yep, most of the VMs will be 8 core / 8 gb | |
| 13:51:10 | mnaser | also FWIW those aren't dedicated cores | |
| 13:51:39 | sean-k-mooney | sure you just setting hw:numa_nodes=2 but not enabling pinning | |
| 13:51:48 | aarents | hi dansmith, can you confirm that the last update is fine for you ? https://review.opendev.org/#/c/670000 | |
| 13:51:57 | mnaser | yep, os that way im thinknig nova should do 4 cores per NUMA, 4gb in each numa node | |
| 13:52:03 | sean-k-mooney | if you do that you should also set hw:mem_page_size to either large or small | |
| 13:52:27 | sean-k-mooney | mnaser: yes it would | |
| 13:53:00 | sean-k-mooney | the reason for doing hw:mem_page_size by the way is you want to enable novas numa aware memory tracking | |
| 13:53:24 | sean-k-mooney | if your dont want to use hugepages set it to "small" | |
| 13:53:39 | sean-k-mooney | that will use 4k pages | |
| 13:53:46 | mnaser | sean-k-mooney: i think huge pages is soemthing id want, but i think transparent huge pages are enabled right now | |
| 13:54:26 | sean-k-mooney | right but you should still set hw:mem_page_size=small if you have a numa toplogy and are not using explcit hugepages | |
| 13:54:33 | sean-k-mooney | if you dont you can get OOM events | |
| 13:54:45 | mnaser | i think id probably want large anyways, host has 512G memory and plenty of reserved memoryt | |
| 13:55:12 | sean-k-mooney | ya if you dont allow memory oversubsription then there is no reason not to use hugepages | |
| 13:55:32 | mnaser | ya nope no oversubscription | |
| 13:58:52 | mnaser | epyc rome numbers https://www.irccloud.com/pastebin/3bgON0HX/ | |
| 13:59:18 | mnaser | intel (but much older hardware) https://www.irccloud.com/pastebin/6M7B4QLY/ | |
| 13:59:22 | mnaser | sean-k-mooney: ^ fyi, interesting to see latency numbers | |
| 14:01:05 | coreycb | kashyap: do you know if anything has landed for removing cpu flags, similar to cpu_model_extra_flags? | |
| 14:01:28 | kashyap | coreycb: Hi...afraid, no; but I've filed a Blueprint for it: | |
| 14:01:55 | kashyap | coreycb: https://blueprints.launchpad.net/nova/+spec/allow-disabling-cpu-flags | |
| 14:02:30 | coreycb | kashyap: thanks | |
| 14:03:40 | coreycb | kashyap: fyi this bug is why I'm asking: https://bugs.launchpad.net/bugs/1853200 | |
| 14:03:40 | openstack | Launchpad bug 1853200 in libvirt (Ubuntu) "cpu features hle and rtm disabled for security are present in /usr/share/libvirt/cpu_map.xml" [High,Confirmed] - Assigned to Ubuntu Security Team (ubuntu-security) | |
| 14:03:44 | kashyap | coreycb: I know :-( | |
| 14:03:51 | kashyap | And I guessed as much | |
| 14:04:35 | kashyap | coreycb: One (very valid) 'workaround' is that have QEMU add new "named CPU models" to remove the said flags. | |
| 14:05:00 | coreycb | kashyap: ok I'll mention that, thanks | |
| 14:05:43 | kashyap | coreycb: For that, upstream QEMU folks must add them...please file a QEMU "RFE" bug on launchpad for it | |
| 14:06:03 | sean-k-mooney | mnaser: did you enable cluster on die for the intel system out of interest | |
| 14:06:28 | coreycb | kashyap: thanks I'll pass the details on to cpaelzer, he's our qemu maintainer | |
| 14:06:39 | kashyap | (Nod) | |
| 14:06:50 | sean-k-mooney | mnaser: but yes the amd number seam to be much higher | |
| 14:07:17 | sean-k-mooney | mnaser: that is quite surprising to be honest | |
| 14:39:15 | efried | bauzas, gibi, stephenfin: would one of you please have a look at the nova/cyborg spec update https://review.opendev.org/#/c/684151/ and +A if appropriate? | |
| 14:39:35 | bauzas | efried: yeah, I can do | |
| 14:39:40 | efried | thanks bauzas | |
| 14:39:48 | stephenfin | I didn't review the original so I'll defer to others | |
| 14:39:55 | bauzas | I promised to do spec reviews, in particular following the ones we discussed at the PTG | |