| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-11-20 | |||
| 08:37:55 | gibi | bauzas: debugger helped it is the args definition that is buggy https://review.opendev.org/#/c/670112/10/nova/cmd/manage.py@2887 | |
| 08:38:33 | bauzas | argh | |
| 08:38:38 | gibi | I guess you are missing some test coverage for --delete :) | |
| 08:38:42 | bauzas | but I'm testing it | |
| 08:39:12 | bauzas | gibi: https://review.opendev.org/#/c/670112/10/nova/tests/functional/test_nova_manage.py@1417 | |
| 08:39:14 | bauzas | OH ! | |
| 08:39:14 | gibi | bauzas: you are calling audit() directly from the test so skipping the arg parser | |
| 08:39:19 | bauzas | yeah this | |
| 08:39:28 | bauzas | how could I test it then ? | |
| 08:41:12 | gibi | bauzas: good question. I just quickly checked and the heal_allocation tests do not cover args parsing either | |
| 08:41:18 | bauzas | yeah | |
| 08:44:07 | gibi | I would not worry about it then in the functional env | |
| 08:44:18 | gibi | it seems we don't have the facility there | |
| 08:45:48 | gibi | there are some nova-manage tests in https://github.com/openstack/nova/blob/master/gate/post_test_hook.sh there we can cover the args parsing too | |
| 08:46:24 | openstackgerrit | Sylvain Bauza proposed openstack/nova stable/train: Don't delete compute node, when deleting service other than nova-compute https://review.opendev.org/695145 | |
| 08:46:45 | gibi | I guess this can be replaced with your audit call https://github.com/openstack/nova/blob/master/gate/post_test_hook.sh#L81 | |
| 08:50:22 | openstackgerrit | Balazs Gibizer proposed openstack/nova stable/pike: Only nil az during shelve offload https://review.opendev.org/693839 | |
| 09:17:55 | stephenfin | gibi, bauzas: If either of you have bandwidth available this week, I'd appreciate some reviews on the removal of nova-network series https://review.opendev.org/#/q/topic:bp/remove-nova-network-ussuri+status:open | |
| 09:19:36 | gibi | stephenfin: added to my list, but no hard promises | |
| 09:19:43 | stephenfin | ack | |
| 09:21:52 | bauzas | stephenfin: sure, I want to dig into reviews this week | |
| 09:46:43 | openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Rewrite quotas documentation https://review.opendev.org/667165 | |
| 10:03:54 | openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: libvirt: Bump MIN_{LIBVIRT,QEMU}_VERSION for "Ussuri" https://review.opendev.org/695056 | |
| 10:28:36 | openstackgerrit | Liang Fang proposed openstack/nova-specs master: Support volume local cache https://review.opendev.org/689070 | |
| 10:39:59 | openstackgerrit | Merged openstack/nova master: Don't delete compute node, when deleting service other than nova-compute https://review.opendev.org/694756 | |
| 11:41:57 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: Add a placement audit command https://review.opendev.org/670112 | |
| 11:41:58 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: Avoid PlacementFixture silently swallowing kwargs https://review.opendev.org/695180 | |
| 11:52:06 | openstack | Launchpad bug 1853280 in OpenStack Compute (nova) pike "nova-live-migration job constantly fails on stable/pike" [Undecided,New] | |
| 11:52:06 | gibi | stephenfin: I think your patch https://review.opendev.org/#/c/694330/ in devstack-plugin-ceph the causing nova-live-migration job to fail on stable/pike. Opened a bug here https://bugs.launchpad.net/nova/+bug/1853280 | |
| 11:52:45 | gibi | lyarwood: ^^ for the yesterday stable/pike issue | |
| 12:10:53 | artom | stephenfin, heya, could I get you to take another look at https://review.opendev.org/#/c/672595/60? | |
| 12:11:11 | artom | sean-k-mooney, same question for https://review.opendev.org/#/c/691062/ :) | |
| 12:17:39 | sean-k-mooney | artom: sure did you fix up the other pathce below if so ill start form the bottom and work up | |
| 12:21:13 | artom | sean-k-mooney, yeah, the series is ready to go - Joe even merged some of the bottom ones | |
| 12:22:19 | sean-k-mooney | i have not written them yet so how can i be angrey :) | |
| 12:23:53 | artom | Ah, the anger comes later, I see | |
| 12:24:51 | sean-k-mooney | ohly after fighting with mocking for about an hour. although what you want me to test should be strait forward so you will likely get a pass | |
| 12:25:51 | artom | Yep, I know how "unit testing" some of those "functions" can be | |
| 12:26:00 | artom | But you made such a nice clean new one... :) | |
| 12:30:39 | openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: libvirt: Bump MIN_{LIBVIRT,QEMU}_VERSION for "Ussuri" https://review.opendev.org/695056 | |
| 12:30:39 | openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: Pick NEXT_MIN libvirt/QEMU versions for "V" release https://review.opendev.org/694821 | |
| 12:57:43 | kashyap | stephenfin: Heya, one reason why I also separate out the NEXT_MIN is that it lets us settle on something while we chip away at bump + resulting clean-up | |
| 13:03:07 | stephenfin | kashyap: The clean up is all optional though, right? | |
| 13:03:14 | stephenfin | I mean, it's just dead code | |
| 13:03:32 | kashyap | stephenfin: I mean the immediate clean-up to make the bump work; not the "full clean-up of removing dead constants" | |
| 13:03:49 | stephenfin | ah, I figured it would just be test removal | |
| 13:04:16 | stephenfin | gibi: I assume the plugin isn't versioned? | |
| 13:04:27 | kashyap | :-) Hehe, you've also done part of this clean-up before, maybe you forgot briefly | |
| 13:04:41 | gibi | stephenfin: I think it is not branched | |
| 13:04:53 | stephenfin | kashyap: yup, but they could all be done after the fact | |
| 13:04:56 | gibi | stephenfin: I haven't checked if it is versioned somehow | |
| 13:05:24 | kashyap | stephenfin: Oh, certainly; that's how we do anyway - removing the dead constants after the bump. | |
| 13:05:33 | stephenfin | kashyap: so the only thing we'd have to clean up in that patch is anything that assumes the version is == the old minimum | |
| 13:05:44 | openstackgerrit | Mark Goddard proposed openstack/nova master: Invalidate provider tree when compute node disappears https://review.opendev.org/695188 | |
| 13:05:44 | openstackgerrit | Mark Goddard proposed openstack/nova master: Clear rebalanced compute nodes from resource tracker https://review.opendev.org/695187 | |
| 13:05:44 | openstackgerrit | Mark Goddard proposed openstack/nova master: Prevent deletion of a compute node belonging to another host https://review.opendev.org/694802 | |
| 13:05:44 | openstack | bug 1853009 in OpenStack Compute (nova) "Ironic node rebalance race can lead to missing compute nodes in DB" [Undecided,In progress] https://launchpad.net/bugs/1853009 - Assigned to Mark Goddard (mgoddard) | |
| 13:05:44 | openstackgerrit | Mark Goddard proposed openstack/nova master: Add functional regression test for bug 1853009 https://review.opendev.org/695012 | |
| 13:05:45 | openstackgerrit | Mark Goddard proposed openstack/nova master: Fix inactive session error in compute node creation https://review.opendev.org/695189 | |
| 13:06:17 | kashyap | stephenfin: That's what I did (or I thought I did) here: https://review.opendev.org/#/c/695056/2 | |
| 13:06:45 | kashyap | stephenfin: If you skimmed the commit, don't mistake the listed constants as the ones I've removed -- I just noted that they'll be removed in _future_ patches | |
| 13:10:24 | mnaser | sean-k-mooney: i'm trying to optimize a system to minimize memory latency. i have a total of 8 numa nodes, 4 inside every socket (distances numa<=>numa on same socket is 10, but cross numa in same socket is 12, cross socket+cross numa is 32) | |
| 13:11:07 | mnaser | so its really expensive to do cross-numa cross-socket (and even somewhat to do cross-numa in socket too). i have 12 threads inside a NUMA node too | |
| 13:12:08 | mnaser | i'm thinking if i give the VM 2 NUMA nodes (instead of none), will that end up increasing performance because hopefully the two pair of NUMA nodes end up being close by (rather than cross socket) | |
| 13:12:22 | mnaser | and maybe at least the VM will be aware that it's crossing numa paths | |
| 13:12:22 | stephenfin | gibi: okay, I can fix that now | |
| 13:12:24 | gibi | stephenfin: no tags, no branches in devstack-plugin-ceph so I don't know how to pin | |
| 13:12:37 | gibi | stephenfin: thanks for fixing | |
| 13:12:38 | stephenfin | I'm not going to pin. I'm going to simply check if Python 3 is enabled | |
| 13:12:47 | gibi | stephenfin: good idea | |
| 13:12:55 | sean-k-mooney | mnaser: you using first gen amd eypc cpus yes | |
| 13:13:02 | mnaser | sean-k-mooney: second gen :) | |
| 13:13:33 | sean-k-mooney | second gen have 1 io die per socket and only 1 numa node per socket | |
| 13:13:55 | sean-k-mooney | well by second gen i ment zen2 backed eypc cpus | |
| 13:14:08 | mnaser | really? i actually have BIOS settings to allow me to change an "NPS" settting which makes me select 1/2/4 | |
| 13:14:09 | sean-k-mooney | im not sure if they had a refresh based on zen 1 | |
| 13:14:30 | sean-k-mooney | that shoudl be there for zen1 | |
| 13:14:49 | mnaser | https://developer.amd.com/wp-content/resources/56745_0.80.pdf see "NUMA Nodes per Socket (NPS)" -- i set that to 4 so i have 8 numa nodes in total | |
| 13:14:57 | sean-k-mooney | the recomendation i have gien internally is to disable multiple numa nodes per socket for zen | |
| 13:15:32 | mnaser | really? i've seen that it has resulted in less than ideal NUMA node latency (but a more consistent one inside a socket) | |
| 13:16:19 | sean-k-mooney | hum ok so you are using gen2 | |
| 13:16:23 | mnaser | yep | |
| 13:16:36 | mnaser | `numactl -N 0 -m 0 sysbench memory --threads=8 --memory-total-size=32G run` with NPS=4 was *WAY* faster than with NPS=1 | |
| 13:16:43 | sean-k-mooney | i might need to reasses | |
| 13:16:48 | sean-k-mooney | yes it would be | |
| 13:17:46 | sean-k-mooney | so basically we were under the impression that gen 2 would only have 1 numa node so we decided not to optimise for the up to 8 numa nodes per socket that can be exposed in gen 1 | |
| 13:18:11 | mnaser | gen 2 can even go up to 16 numa nodes, you can make it expose a numa node per each l3 cache | |
| 13:18:32 | sean-k-mooney | mnaser: so to your orgincal question giving the guest 2 numa nodes will imporve the guest performace in general | |
| 13:18:55 | mnaser | sean-k-mooney: https://softwarefactory-project.io/logs/45/16145/11/check/benchmark-combine/ba8ff3a/phoronix-results-cloud-fedora-30-vexxhost-vexxhost-nodepool-tripleo-0002241618/ in this case, much older hardware vs gen 2 epyc.. look at the difference in memory performance | |
| 13:19:01 | sean-k-mooney | so the trade of with that is it limits the vms signifcantly | |
| 13:19:11 | sean-k-mooney | or rather how you create your flavors | |
| 13:19:43 | sean-k-mooney | if you enable all 16 | |
| 13:20:02 | stephenfin | gibi: https://review.opendev.org/695191 | |
| 13:20:16 | sean-k-mooney | then you vms with a numa topology can only have 3-4 cpus per numa node due to how we do numa in nova | |
| 13:20:51 | stephenfin | gibi: Thank God I included https://review.opendev.org/#/c/692374/6/gate/live_migration/hooks/ceph.sh@11 :D | |
| 13:20:58 | mnaser | i see, i'm ok with 8 if they are living next to each other (and i guess giving the vm the proper awareness means it will know how to address things) | |
| 13:21:06 | sean-k-mooney | e.g. if you expose 1 numa node per ccx it give you the best performance but you are forst to either have multipl numa nodes for larger vms or use no numa features | |
| 13:21:16 | mnaser | i rather 8 because it gives me bigger memory slices and more cores (so i can have an entire VM live in a single NUMA node) | |
| 13:21:49 | sean-k-mooney | yes but that raises an intersting point | |
| 13:21:50 | gibi | stephenfin: deep down you knew that you need to check for python3 enabdled :) | |