Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2022-06-14
10:28:34 sean-k-mooney
10:28:36 sean-k-mooney 0:0,1:32,2:512,3:544
10:28:38 sean-k-mooney 0:1,1:2,2:3,3:4
10:28:40 sean-k-mooney 0:5,1:69,2:133,3:197
10:28:42 sean-k-mooney 0:33,1:513,2:34,3:514,4:35,5:515,6:36,7:6
10:28:44 sean-k-mooney 0:2047,1:1983,2:1919,3:1855,4:2015,5:1951,6:1887,7:1823
10:28:46 sean-k-mooney bug repoducer
10:28:48 sean-k-mooney ----------------------------------------
10:28:50 sean-k-mooney 0:0,1:3,2:6,3:9,4:12,5:15,6:18,7:21,8:48,9:51,10:54,11:57,12:60,13:63,14:66,15:69,16:1,17:4,18:7,19:10,20:13,21:16,22:19,23:22,24:49,25:52,26:55,27:58,28:61,29:64,30:67,31:70,32:2,33:5,34:8,35:11,36:14,37:17,38:20,39:23,40:50,41:53,42:56,43:59,44:62,45:65,46:68,47:71
10:28:52 sean-k-mooney real 0m1.004s
10:28:54 sean-k-mooney user 0m0.893s
10:28:56 sean-k-mooney sys 0m0.094s
10:28:58 sean-k-mooney gibi: is ^ better
10:29:18 sean-k-mooney or
10:29:20 sean-k-mooney real 0m0.986s
10:29:22 sean-k-mooney user 0m0.887s
10:29:24 sean-k-mooney sys 0m0.078s
10:29:26 sean-k-mooney if i jus trun the repoducer
10:32:24 gibi the reproducer took 6 mins to run on my laptop
10:32:36 gibi so yours seems to be a loooot faster
10:34:17 sean-k-mooney https://github.com/SeanMooney/cpu-pinning/commit/018a1d0a9abeeec40d3e3ccbfd640363158433c5
10:34:28 sean-k-mooney that is the same case right
10:35:07 sean-k-mooney 1 socket 16 numa nodes and 48 cores with 2 thread pre cpu for 96 theads
10:35:29 sean-k-mooney then booth a 48 core cpu with pinning and the prefer thread polciy with 1G hugepages
10:35:42 gibi there are some usage too in the reproducer
10:35:42 sean-k-mooney and 488GB of ram
10:36:32 gibi the first 7 host numa cell is used
10:36:55 sean-k-mooney ack i can add that and see if it makes any differnce
10:50:50 sean-k-mooney basically the same
10:50:52 sean-k-mooney real 0m1.074s
10:50:54 sean-k-mooney user 0m0.936s
10:50:56 sean-k-mooney sys 0m0.115s
10:51:30 sean-k-mooney https://github.com/SeanMooney/cpu-pinning/blob/master/pinning.py#L378-L415=
10:52:25 sean-k-mooney the issue with my alternitiv implemntation is it only supprots hugepages and cpu pinning
10:52:36 sean-k-mooney it is missing pci supprot and mixed cpu support
10:53:48 sean-k-mooney ill run there repodcuer to have a comparison on the same hardware
10:59:21 sean-k-mooney im going to make coffee whiel this runs...
11:03:42 gibi dansmith, bauzas: what do you think do we want to / can do something about this? https://bugs.launchpad.net/nova/+bug/1978549 It feels like a bug but I'm not sure I want to go back and add a db migration to stein in nova or touch the placement db init script. https://bugs.launchpad.net/nova/+bug/1978549
11:06:20 sean-k-mooney gibi: dont we tend to not drop the columes right away
11:06:29 sean-k-mooney to cater for rolling upgrades
11:06:48 gibi sure
11:06:52 sean-k-mooney we stop the usage and then a few release later we can drop the column
11:07:01 gibi but then we suddenly forget to add it to the placement db init script
11:07:27 gibi so if you have a DB that was created before placemetn was moved out of nova repor then you have can_host
11:07:35 sean-k-mooney well that woudl only be an issue if you were initing on a version that used it right
11:08:23 gibi we removed the can_host column but without a DB migration
11:08:38 gibi the removal happen when the new db init was created for the moved out placement
11:08:46 sean-k-mooney i see
11:10:18 sean-k-mooney gibi: real 12m32.814s
11:10:38 sean-k-mooney maybe my 4 year old laptop is starting to show its age
11:11:55 sean-k-mooney gibi: we could just document this as a workaround for the db issue
11:12:27 sean-k-mooney the fix is only requried if movign form in nova to out of tree placement right
11:12:40 sean-k-mooney if you are doign a db restore
11:12:48 sean-k-mooney and even then only if you are migrating db backends
11:13:16 sean-k-mooney if you were just doing a db export and import in mysql it would be fien because the improt would create the tabels
11:13:46 sean-k-mooney in there case they likely are just migrating the data and using placemnt-mange to init an empty db
11:16:08 gibi I agree to only document the workaround. I'm not sure where to put that documentation though. As adding a reno now in zed in placement for an issue introduced in stein in nova does not feels right
11:16:49 sean-k-mooney we can do a stable only reno
11:16:59 sean-k-mooney on stine
11:17:48 sean-k-mooney or update the existing one for speliting out placment which i assuem exists somewhere
11:18:00 gibi hm there is admin/upgrade-to-stein.rst
11:18:11 gibi that would be OK
11:18:13 gibi thanks
11:32:45 opendevreview Balazs Gibizer proposed openstack/placement master: Add WA about resource_providers.can_host removal https://review.opendev.org/c/openstack/placement/+/845730
11:33:10 sean-k-mooney gibi: im not going to do this now but if i were to update my poc to implemetne all or a subset of the current numa_fit_instance_to_host funciton
11:33:59 sean-k-mooney would we consider that backportable if it was opt in and both implemations could be in the code in parallel even fi only one would be used on a compute host at a time
11:34:26 gibi why we need both? does yours produce a different result than the current?
11:34:41 sean-k-mooney mine does not do everythin the current one does
11:34:59 gibi yepp, then we might need a config flag
11:35:04 sean-k-mooney so im wondering if we coudl make it incremental
11:35:40 sean-k-mooney and also have the option to run both as filters
11:35:55 sean-k-mooney so mine currently only handeles hugepages and cpu pinning
11:36:17 sean-k-mooney its missing pci devices and mixed cpus and maybe pmem
11:36:38 sean-k-mooney pmem enabels a numa toplogy but i dont think we provide affintiy for pmem
11:36:57 sean-k-mooney so its reall just mixed mode and pci devices that woudl initally be missing
11:37:38 sean-k-mooney my tought is you coudl enable both filters and have the fast one run first so you woudl only validate the slow hosts if we knew the cpu and ram request were valid
11:38:07 sean-k-mooney s/slow hosts/hosts with the slow version/
11:38:32 gibi overall I'm OK to make it incremental or even selectable, but I'm not sure stable cores will like it
11:38:51 gibi this will include a new bounch of code to stable branches
11:38:54 sean-k-mooney so basicaly you woudl do pci_passthroughfilter,numa_v2,numa_toplogy_filter
11:39:01 gibi that is liability
11:39:09 sean-k-mooney yep
11:39:12 sean-k-mooney it is
11:39:34 gibi what if we just add what we have to master as selectable, then we improve on it later on master
11:39:50 sean-k-mooney we could yes
11:40:07 sean-k-mooney im just trying to think if there is anything we can do to adress the bug report on older branches too
11:40:52 sean-k-mooney i dont think there is anythign trivial that can be fixed on the older brances to make the perfromance accpetabel.
11:41:39 gibi I don't know. maybe we can look at the reproducer in a profiler
11:42:11 sean-k-mooney i think its to do with our usage so itertools permuations
11:42:33 sean-k-mooney we do a liniar loop over all permuations of numa nodes when trying to fit the instance
11:43:06 sean-k-mooney we early out once the instance fits but that is really not efficent with more then a hand full of numa nodes
11:44:37 sean-k-mooney gibi: i can try an take a look at it breifly
11:45:11 gibi sean-k-mooney: only if you want :) I'm not asking to take this
11:45:29 gibi I'm not promising either that I can fire up a profiler today
11:45:36 sean-k-mooney hehe
11:45:44 sean-k-mooney this is not new
11:46:01 sean-k-mooney so i dont think it supper high priority but i was half way thorugh fixing up my vdpa patches
11:46:07 sean-k-mooney so i want to finish those today
11:46:30 sean-k-mooney so im somehwat interested in is there a minimal fix we can do to make ti better
11:46:39 sean-k-mooney i just dont want to spend all day looking at it

Earlier   Later