| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-06-14 | |||
| 10:04:17 | sean-k-mooney | i did some experiments a few year ago repimlementing it form scratch | |
| 10:04:39 | sean-k-mooney | but the reason i did not continue with that was we were going to implement numa in placement any day now | |
| 10:04:55 | gibi | also we are emitting 18G debug logs during that run which is pretty extreme | |
| 10:05:35 | sean-k-mooney | peopel complaied there was not enough logging in the hardware module :P | |
| 10:06:06 | sean-k-mooney | https://github.com/SeanMooney/cpu-pinning | |
| 10:06:13 | sean-k-mooney | this was my experimal approch | |
| 10:08:31 | gibi | as far as I understand this call does not call out to any extrnal system (no fs, db, rabbit calls) so it runs in pure python. So taking minutes indicate that we are doing something crazy | |
| 10:10:59 | sean-k-mooney | we are try to validate cpu (floating and pinned),ram ,pci, and pmem affinity based on the rules set in teh falvor with regards to thread affintiy, device affintiy and any semetic or asmetirc numa requirements | |
| 10:16:32 | sean-k-mooney | gibi: by the way we have know about this basically from the start that its qudratic or wrose as you scale the number of host/guest numa nodes | |
| 10:16:50 | sean-k-mooney | its why the numa toploty filter shoudl always be last in your filter list if you enable it | |
| 10:19:28 | sean-k-mooney | i am not sure if ic an repodcue that simpel example in my ohter implemntaion but im going to give it a try quickly | |
| 10:28:30 | sean-k-mooney | [11:27:50]❯ ./run.sh | |
| 10:28:32 | sean-k-mooney | Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project. You can set PIPENV_IGNORE_VIRTUALENVS=1 to force pipenv to ignore that environment and create its own instead. You can set PIPENV_VERBOSITY=-1 to suppress this warning. | |
| 10:28:34 | sean-k-mooney | ||
| 10:28:36 | sean-k-mooney | 0:0,1:32,2:512,3:544 | |
| 10:28:38 | sean-k-mooney | 0:1,1:2,2:3,3:4 | |
| 10:28:40 | sean-k-mooney | 0:5,1:69,2:133,3:197 | |
| 10:28:42 | sean-k-mooney | 0:33,1:513,2:34,3:514,4:35,5:515,6:36,7:6 | |
| 10:28:44 | sean-k-mooney | 0:2047,1:1983,2:1919,3:1855,4:2015,5:1951,6:1887,7:1823 | |
| 10:28:46 | sean-k-mooney | bug repoducer | |
| 10:28:48 | sean-k-mooney | ---------------------------------------- | |
| 10:28:50 | sean-k-mooney | 0:0,1:3,2:6,3:9,4:12,5:15,6:18,7:21,8:48,9:51,10:54,11:57,12:60,13:63,14:66,15:69,16:1,17:4,18:7,19:10,20:13,21:16,22:19,23:22,24:49,25:52,26:55,27:58,28:61,29:64,30:67,31:70,32:2,33:5,34:8,35:11,36:14,37:17,38:20,39:23,40:50,41:53,42:56,43:59,44:62,45:65,46:68,47:71 | |
| 10:28:52 | sean-k-mooney | real 0m1.004s | |
| 10:28:54 | sean-k-mooney | user 0m0.893s | |
| 10:28:56 | sean-k-mooney | sys 0m0.094s | |
| 10:28:58 | sean-k-mooney | gibi: is ^ better | |
| 10:29:18 | sean-k-mooney | or | |
| 10:29:20 | sean-k-mooney | real 0m0.986s | |
| 10:29:22 | sean-k-mooney | user 0m0.887s | |
| 10:29:24 | sean-k-mooney | sys 0m0.078s | |
| 10:29:26 | sean-k-mooney | if i jus trun the repoducer | |
| 10:32:24 | gibi | the reproducer took 6 mins to run on my laptop | |
| 10:32:36 | gibi | so yours seems to be a loooot faster | |
| 10:34:17 | sean-k-mooney | https://github.com/SeanMooney/cpu-pinning/commit/018a1d0a9abeeec40d3e3ccbfd640363158433c5 | |
| 10:34:28 | sean-k-mooney | that is the same case right | |
| 10:35:07 | sean-k-mooney | 1 socket 16 numa nodes and 48 cores with 2 thread pre cpu for 96 theads | |
| 10:35:29 | sean-k-mooney | then booth a 48 core cpu with pinning and the prefer thread polciy with 1G hugepages | |
| 10:35:42 | sean-k-mooney | and 488GB of ram | |
| 10:35:42 | gibi | there are some usage too in the reproducer | |
| 10:36:32 | gibi | the first 7 host numa cell is used | |
| 10:36:55 | sean-k-mooney | ack i can add that and see if it makes any differnce | |
| 10:50:50 | sean-k-mooney | basically the same | |
| 10:50:52 | sean-k-mooney | real 0m1.074s | |
| 10:50:54 | sean-k-mooney | user 0m0.936s | |
| 10:50:56 | sean-k-mooney | sys 0m0.115s | |
| 10:51:30 | sean-k-mooney | https://github.com/SeanMooney/cpu-pinning/blob/master/pinning.py#L378-L415= | |
| 10:52:25 | sean-k-mooney | the issue with my alternitiv implemntation is it only supprots hugepages and cpu pinning | |
| 10:52:36 | sean-k-mooney | it is missing pci supprot and mixed cpu support | |
| 10:53:48 | sean-k-mooney | ill run there repodcuer to have a comparison on the same hardware | |
| 10:59:21 | sean-k-mooney | im going to make coffee whiel this runs... | |
| 11:03:42 | gibi | dansmith, bauzas: what do you think do we want to / can do something about this? https://bugs.launchpad.net/nova/+bug/1978549 It feels like a bug but I'm not sure I want to go back and add a db migration to stein in nova or touch the placement db init script. https://bugs.launchpad.net/nova/+bug/1978549 | |
| 11:06:20 | sean-k-mooney | gibi: dont we tend to not drop the columes right away | |
| 11:06:29 | sean-k-mooney | to cater for rolling upgrades | |
| 11:06:48 | gibi | sure | |
| 11:06:52 | sean-k-mooney | we stop the usage and then a few release later we can drop the column | |
| 11:07:01 | gibi | but then we suddenly forget to add it to the placement db init script | |
| 11:07:27 | gibi | so if you have a DB that was created before placemetn was moved out of nova repor then you have can_host | |
| 11:07:35 | sean-k-mooney | well that woudl only be an issue if you were initing on a version that used it right | |
| 11:08:23 | gibi | we removed the can_host column but without a DB migration | |
| 11:08:38 | gibi | the removal happen when the new db init was created for the moved out placement | |
| 11:08:46 | sean-k-mooney | i see | |
| 11:10:18 | sean-k-mooney | gibi: real 12m32.814s | |
| 11:10:38 | sean-k-mooney | maybe my 4 year old laptop is starting to show its age | |
| 11:11:55 | sean-k-mooney | gibi: we could just document this as a workaround for the db issue | |
| 11:12:27 | sean-k-mooney | the fix is only requried if movign form in nova to out of tree placement right | |
| 11:12:40 | sean-k-mooney | if you are doign a db restore | |
| 11:12:48 | sean-k-mooney | and even then only if you are migrating db backends | |
| 11:13:16 | sean-k-mooney | if you were just doing a db export and import in mysql it would be fien because the improt would create the tabels | |
| 11:13:46 | sean-k-mooney | in there case they likely are just migrating the data and using placemnt-mange to init an empty db | |
| 11:16:08 | gibi | I agree to only document the workaround. I'm not sure where to put that documentation though. As adding a reno now in zed in placement for an issue introduced in stein in nova does not feels right | |
| 11:16:49 | sean-k-mooney | we can do a stable only reno | |
| 11:16:59 | sean-k-mooney | on stine | |
| 11:17:48 | sean-k-mooney | or update the existing one for speliting out placment which i assuem exists somewhere | |
| 11:18:00 | gibi | hm there is admin/upgrade-to-stein.rst | |
| 11:18:11 | gibi | that would be OK | |
| 11:18:13 | gibi | thanks | |
| 11:32:45 | opendevreview | Balazs Gibizer proposed openstack/placement master: Add WA about resource_providers.can_host removal https://review.opendev.org/c/openstack/placement/+/845730 | |
| 11:33:10 | sean-k-mooney | gibi: im not going to do this now but if i were to update my poc to implemetne all or a subset of the current numa_fit_instance_to_host funciton | |
| 11:33:59 | sean-k-mooney | would we consider that backportable if it was opt in and both implemations could be in the code in parallel even fi only one would be used on a compute host at a time | |
| 11:34:26 | gibi | why we need both? does yours produce a different result than the current? | |
| 11:34:41 | sean-k-mooney | mine does not do everythin the current one does | |
| 11:34:59 | gibi | yepp, then we might need a config flag | |
| 11:35:04 | sean-k-mooney | so im wondering if we coudl make it incremental | |
| 11:35:40 | sean-k-mooney | and also have the option to run both as filters | |
| 11:35:55 | sean-k-mooney | so mine currently only handeles hugepages and cpu pinning | |
| 11:36:17 | sean-k-mooney | its missing pci devices and mixed cpus and maybe pmem | |
| 11:36:38 | sean-k-mooney | pmem enabels a numa toplogy but i dont think we provide affintiy for pmem | |
| 11:36:57 | sean-k-mooney | so its reall just mixed mode and pci devices that woudl initally be missing | |
| 11:37:38 | sean-k-mooney | my tought is you coudl enable both filters and have the fast one run first so you woudl only validate the slow hosts if we knew the cpu and ram request were valid | |
| 11:38:07 | sean-k-mooney | s/slow hosts/hosts with the slow version/ | |
| 11:38:32 | gibi | overall I'm OK to make it incremental or even selectable, but I'm not sure stable cores will like it | |
| 11:38:51 | gibi | this will include a new bounch of code to stable branches | |
| 11:38:54 | sean-k-mooney | so basicaly you woudl do pci_passthroughfilter,numa_v2,numa_toplogy_filter | |
| 11:39:01 | gibi | that is liability | |
| 11:39:09 | sean-k-mooney | yep | |
| 11:39:12 | sean-k-mooney | it is | |
| 11:39:34 | gibi | what if we just add what we have to master as selectable, then we improve on it later on master | |
| 11:39:50 | sean-k-mooney | we could yes | |
| 11:40:07 | sean-k-mooney | im just trying to think if there is anything we can do to adress the bug report on older branches too | |
| 11:40:52 | sean-k-mooney | i dont think there is anythign trivial that can be fixed on the older brances to make the perfromance accpetabel. | |