Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-01-16
14:40:51 stephenfin not sean-k-mooney but I'm going to guess potentially huge performance penalties if you haven't correctly configured your host
14:42:20 sean-k-mooney so i originally suggested perfered to kashyap as a potentil solution but im more conserned about non hugepage guest that use hw:numa_nodes=*
14:42:34 kashyap belmoreira: Hi
14:42:45 sean-k-mooney for hugepage we pin the pages seperately so they will be allocated form the numa node we expect
14:43:29 sean-k-mooney however setting perfered by default means we can be sure that the memory of a gues that has a numa doploty but is not backed by hugepage will come form the numa node we expect
14:43:58 sean-k-mooney which will cause issue if we ever model numa in placment
14:45:01 sean-k-mooney i suspect that most of the time it would likely be fine but im not sure what all the sideeffect will be
14:45:17 sean-k-mooney belmoreira: have you tested the patch and conrimed that it fixes the issue?
14:45:41 sean-k-mooney belmoreira: we were not able to get our customer to do that and we did not have a plathform where we could repoduce the issue
14:45:59 sean-k-mooney belmoreira: so we dont acatully know that will fix it although we suspect it will
14:46:08 kashyap stephenfin: I don't know when exactly it was introduced; I need to do kernel commit message digging (if it's not there, then worse, do bisection myself or ask a KVM dev)
14:47:10 sean-k-mooney stephenfin: belmoreira i would be more ok with his as a workaround config option and keeping the default to strict with the expection we might remvoe it in the future.
14:48:07 kashyap stephenfin: On forcing distros to backport — normally, that would be my approach such kernel cases. But here, we need to bear in mind all the existing OpenStack deployments in the wild.
14:49:28 sean-k-mooney i think this issue is only going to show up on centos/rhel maybe suse
14:49:56 kashyap sean-k-mooney: I think they'd be the distros that are most likely to _backport_ those
14:50:06 sean-k-mooney if you are using the hwe kernel on ubunutu 18.04 you will be on 5.x
14:50:12 kashyap Right
14:50:55 sean-k-mooney i supose if you stuck with the orginal ubuntu 18.04 kernel that was 4.15
14:52:06 kashyap belmoreira: I take it that rebooting the host kernel is not an option in your deployments?
14:52:27 sean-k-mooney belmoreira: o i see you noted that the patch did solve the problem
14:52:53 kashyap Yeah, he tested it
14:53:33 kashyap sean-k-mooney: And in general, 'preferred' is what we recommend _anyway_, no?
14:53:57 sean-k-mooney no
14:54:07 sean-k-mooney we recommend strict
14:54:09 kashyap I have to remind myself of this everytime.
14:54:43 sean-k-mooney prefered makes the memory allcoation of non hugepage memory a hint not a requirement
14:54:46 kashyap "Where, 'preferred' means: disable NUMA affinity and so the
14:54:46 kashyap request is turned to a "hint", i.e. "if possible, allocate from the
14:54:48 kashyap given NUMA node; otherwise, fallback to other nodes"
14:54:54 kashyap Yes
14:55:10 sean-k-mooney which will break our numa aware memory tracking
14:55:24 sean-k-mooney and will not work if we model memory in plamcent in the future per numa node
14:55:36 sean-k-mooney since we can rely on the memory being allcoated form the numa node we said
14:55:58 sean-k-mooney for hugepage memeory the option will be ignored so hugepage backed vms will be fine
14:56:36 kashyap Yeah, then I don't want to muck with defaults. With all these future caveats (which are not properly documented.)
14:56:40 sean-k-mooney but vm with a numa toplogy but no hugepages could have there memory allcoated from any numa node if the one we select was full
14:58:21 sean-k-mooney stephenfin: kashyap belmoreira i left a comment on the patch
15:14:40 belmoreira sorry, needed to leave the office
15:15:50 belmoreira sean-k-mooney yes, I can reproduce with centos 7.6 kernel: Linux XXX 3.10.0-957.1.3.el7.x86_64 #1 SMP Thu Nov 29 14:49:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
15:15:59 belmoreira libvirtd 4.5.0-23
15:16:05 belmoreira qemu 2.12.0-33
15:16:40 belmoreira changing to "preferred" fix it (can create new instances)
15:16:58 sean-k-mooney ya as far as i know this is not fixed in 7.6
15:17:07 sean-k-mooney i think the fix should be in 7.8
15:17:20 belmoreira kashyap we upgraded to centos 7.7 but didn't reboot the compute nodes. It's a very painful process
15:17:32 sean-k-mooney it might be in 7.7
15:18:02 sean-k-mooney belmoreira: are you ok with it if we make it a workaround config option and default to strict
15:18:10 belmoreira sean-k-mooney for now I can't reproduce it with 7.7
15:18:29 sean-k-mooney ok cool then that likely already has the backport
15:18:41 sean-k-mooney although this is hardware specific
15:18:54 sean-k-mooney so if its a differen model fo server then you migh not see it
15:19:21 belmoreira sean-k-mooney yes, took me some time to isolate this
15:20:08 sean-k-mooney ya it took us a day or two to figure this out downstream too when it was first reported to us
15:20:18 sean-k-mooney its a very rare edgecase
15:20:28 sean-k-mooney normlaly you will have dma regions per socket
15:21:03 sean-k-mooney or numa node
15:21:14 belmoreira I think that the workaround config will be useful for everyone that is running NUMA aware VMs. Then needs to be backported to older releases
15:21:29 sean-k-mooney so you need a server that does not + requst a numa toploy + have a kernel with the kvm bug
15:22:08 sean-k-mooney well no it will be useful for the minority that have ^
15:22:38 sean-k-mooney but i would be ok with proceeding on that basis
15:22:43 belmoreira sean-k-mooney ok, true :)
15:24:27 belmoreira but also shows that the nova community is aware of the issue and has a work around
15:24:32 belmoreira sean-k-mooney thanks
15:25:18 kashyap belmoreira: We're very much aware, FWIW, see this response on 'openstack-discuss' list: http://lists.openstack.org/pipermail/openstack-discuss/2019-September/009770.html
15:26:08 belmoreira sean-k-mooney but to clarify, "preferred" would break the future placement tracking and this should be disable as soon the new fixed kernel is loaded
15:26:37 sean-k-mooney that placement tracking proably wont be a thing until V
15:26:50 sean-k-mooney i could mess up our tracking in the resouce tracker today
15:27:18 sean-k-mooney but if you have the choice between not being able to use an entire socket
15:27:35 sean-k-mooney or slightly reduced performace im guess your ok with the slight perfromace hit
15:27:48 sean-k-mooney and slightly in accurate tracking
15:28:49 sean-k-mooney belmoreira: my conser is only for instace with a numa toplogy that dont use hugepages
15:29:26 belmoreira got it
15:29:47 sean-k-mooney hugepage guess will be tracked properly in either case
15:30:14 belmoreira thanks a lot for this discussion and proposal
15:32:22 Sundar sean-k-mooney: Please make sure to pull in https://review.opendev.org/698846 and https://review.opendev.org/700261
15:32:45 Sundar They are called out as dependencies in the create/bind patch
15:33:14 Sundar i.e. https://review.opendev.org/#/c/631244/
15:33:38 sean-k-mooney i cant have devstack pull in both if they are not in the same patch change
15:33:42 sean-k-mooney *chain
15:33:53 sean-k-mooney so ill have to create a local branch and merge them in
15:34:10 sean-k-mooney i have https://review.opendev.org/#/c/698846/ but not https://review.opendev.org/#/c/700261/
15:34:13 kashyap belmoreira: stephenfin: efried: The kernel issue (https://review.opendev.org/#/c/684375/) has been latent. (Just confirmed w/ a KVM mainatiner.)
15:34:19 Sundar Both are listed as dependencies in the same Nova patch
15:34:42 kashyap sean-k-mooney: (Responded on the change.) Thanks!
15:34:46 sean-k-mooney yes but devstack can only clone one it does not have the zuul logic to create a merge commti from both
15:35:00 sean-k-mooney but i can do it manually and restart the services
15:35:19 stephenfin kashyap: I wonder why there wasn't a sudden spike so? Surely this would have been affecting belmoreira and co. for years now if it's been there forever
15:35:22 sean-k-mooney Sundar: in general depends-on should not be used within the same repo
15:35:46 openstackgerrit Lee Yarwood proposed openstack/nova master: virt: Provide block_device_info during rescue https://review.opendev.org/700811
15:35:46 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Add support for stable device rescue https://review.opendev.org/700812
15:35:47 openstackgerrit Lee Yarwood proposed openstack/nova master: docs: Add stable device rescue docs https://review.opendev.org/700837
15:35:47 kashyap stephenfin: Yeah, I don't have a solid answer to that yet :-(
15:35:50 sean-k-mooney https://review.opendev.org/#/c/698846/ should be rebased on https://review.opendev.org/#/c/700261/
15:36:14 kashyap stephenfin: Let me see the original kernel report I quoted in the commit, maybe there's something in there
15:36:39 sean-k-mooney stephenfin: i have never seen hardware that actully does not have a dma regoin per numa node
15:37:10 openstackgerrit Lee Yarwood proposed openstack/nova master: DNM test stable rescue tempest/devstack changes https://review.opendev.org/700815
15:37:18 sean-k-mooney stephenfin: and i mean ever in the 7+ years i have been working on this stuff
15:37:50 sean-k-mooney so i am not surprised people have not hit this before
15:38:10 stephenfin efried: I think I'm going to have to get you to weigh in on https://review.opendev.org/#/c/696745/, because I _think_ I've addressed the concerns but everyone else is still avoiding it :)
15:38:38 kashyap sean-k-mooney: Ah, interesting

Earlier   Later