Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-01-16
14:50:06 sean-k-mooney if you are using the hwe kernel on ubunutu 18.04 you will be on 5.x
14:50:12 kashyap Right
14:50:55 sean-k-mooney i supose if you stuck with the orginal ubuntu 18.04 kernel that was 4.15
14:52:06 kashyap belmoreira: I take it that rebooting the host kernel is not an option in your deployments?
14:52:27 sean-k-mooney belmoreira: o i see you noted that the patch did solve the problem
14:52:53 kashyap Yeah, he tested it
14:53:33 kashyap sean-k-mooney: And in general, 'preferred' is what we recommend _anyway_, no?
14:53:57 sean-k-mooney no
14:54:07 sean-k-mooney we recommend strict
14:54:09 kashyap I have to remind myself of this everytime.
14:54:43 sean-k-mooney prefered makes the memory allcoation of non hugepage memory a hint not a requirement
14:54:46 kashyap "Where, 'preferred' means: disable NUMA affinity and so the
14:54:46 kashyap request is turned to a "hint", i.e. "if possible, allocate from the
14:54:48 kashyap given NUMA node; otherwise, fallback to other nodes"
14:54:54 kashyap Yes
14:55:10 sean-k-mooney which will break our numa aware memory tracking
14:55:24 sean-k-mooney and will not work if we model memory in plamcent in the future per numa node
14:55:36 sean-k-mooney since we can rely on the memory being allcoated form the numa node we said
14:55:58 sean-k-mooney for hugepage memeory the option will be ignored so hugepage backed vms will be fine
14:56:36 kashyap Yeah, then I don't want to muck with defaults. With all these future caveats (which are not properly documented.)
14:56:40 sean-k-mooney but vm with a numa toplogy but no hugepages could have there memory allcoated from any numa node if the one we select was full
14:58:21 sean-k-mooney stephenfin: kashyap belmoreira i left a comment on the patch
15:14:40 belmoreira sorry, needed to leave the office
15:15:50 belmoreira sean-k-mooney yes, I can reproduce with centos 7.6 kernel: Linux XXX 3.10.0-957.1.3.el7.x86_64 #1 SMP Thu Nov 29 14:49:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
15:15:59 belmoreira libvirtd 4.5.0-23
15:16:05 belmoreira qemu 2.12.0-33
15:16:40 belmoreira changing to "preferred" fix it (can create new instances)
15:16:58 sean-k-mooney ya as far as i know this is not fixed in 7.6
15:17:07 sean-k-mooney i think the fix should be in 7.8
15:17:20 belmoreira kashyap we upgraded to centos 7.7 but didn't reboot the compute nodes. It's a very painful process
15:17:32 sean-k-mooney it might be in 7.7
15:18:02 sean-k-mooney belmoreira: are you ok with it if we make it a workaround config option and default to strict
15:18:10 belmoreira sean-k-mooney for now I can't reproduce it with 7.7
15:18:29 sean-k-mooney ok cool then that likely already has the backport
15:18:41 sean-k-mooney although this is hardware specific
15:18:54 sean-k-mooney so if its a differen model fo server then you migh not see it
15:19:21 belmoreira sean-k-mooney yes, took me some time to isolate this
15:20:08 sean-k-mooney ya it took us a day or two to figure this out downstream too when it was first reported to us
15:20:18 sean-k-mooney its a very rare edgecase
15:20:28 sean-k-mooney normlaly you will have dma regions per socket
15:21:03 sean-k-mooney or numa node
15:21:14 belmoreira I think that the workaround config will be useful for everyone that is running NUMA aware VMs. Then needs to be backported to older releases
15:21:29 sean-k-mooney so you need a server that does not + requst a numa toploy + have a kernel with the kvm bug
15:22:08 sean-k-mooney well no it will be useful for the minority that have ^
15:22:38 sean-k-mooney but i would be ok with proceeding on that basis
15:22:43 belmoreira sean-k-mooney ok, true :)
15:24:27 belmoreira but also shows that the nova community is aware of the issue and has a work around
15:24:32 belmoreira sean-k-mooney thanks
15:25:18 kashyap belmoreira: We're very much aware, FWIW, see this response on 'openstack-discuss' list: http://lists.openstack.org/pipermail/openstack-discuss/2019-September/009770.html
15:26:08 belmoreira sean-k-mooney but to clarify, "preferred" would break the future placement tracking and this should be disable as soon the new fixed kernel is loaded
15:26:37 sean-k-mooney that placement tracking proably wont be a thing until V
15:26:50 sean-k-mooney i could mess up our tracking in the resouce tracker today
15:27:18 sean-k-mooney but if you have the choice between not being able to use an entire socket
15:27:35 sean-k-mooney or slightly reduced performace im guess your ok with the slight perfromace hit
15:27:48 sean-k-mooney and slightly in accurate tracking
15:28:49 sean-k-mooney belmoreira: my conser is only for instace with a numa toplogy that dont use hugepages
15:29:26 belmoreira got it
15:29:47 sean-k-mooney hugepage guess will be tracked properly in either case
15:30:14 belmoreira thanks a lot for this discussion and proposal
15:32:22 Sundar sean-k-mooney: Please make sure to pull in https://review.opendev.org/698846 and https://review.opendev.org/700261
15:32:45 Sundar They are called out as dependencies in the create/bind patch
15:33:14 Sundar i.e. https://review.opendev.org/#/c/631244/
15:33:38 sean-k-mooney i cant have devstack pull in both if they are not in the same patch change
15:33:42 sean-k-mooney *chain
15:33:53 sean-k-mooney so ill have to create a local branch and merge them in
15:34:10 sean-k-mooney i have https://review.opendev.org/#/c/698846/ but not https://review.opendev.org/#/c/700261/
15:34:13 kashyap belmoreira: stephenfin: efried: The kernel issue (https://review.opendev.org/#/c/684375/) has been latent. (Just confirmed w/ a KVM mainatiner.)
15:34:19 Sundar Both are listed as dependencies in the same Nova patch
15:34:42 kashyap sean-k-mooney: (Responded on the change.) Thanks!
15:34:46 sean-k-mooney yes but devstack can only clone one it does not have the zuul logic to create a merge commti from both
15:35:00 sean-k-mooney but i can do it manually and restart the services
15:35:19 stephenfin kashyap: I wonder why there wasn't a sudden spike so? Surely this would have been affecting belmoreira and co. for years now if it's been there forever
15:35:22 sean-k-mooney Sundar: in general depends-on should not be used within the same repo
15:35:46 openstackgerrit Lee Yarwood proposed openstack/nova master: virt: Provide block_device_info during rescue https://review.opendev.org/700811
15:35:46 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Add support for stable device rescue https://review.opendev.org/700812
15:35:47 openstackgerrit Lee Yarwood proposed openstack/nova master: docs: Add stable device rescue docs https://review.opendev.org/700837
15:35:47 kashyap stephenfin: Yeah, I don't have a solid answer to that yet :-(
15:35:50 sean-k-mooney https://review.opendev.org/#/c/698846/ should be rebased on https://review.opendev.org/#/c/700261/
15:36:14 kashyap stephenfin: Let me see the original kernel report I quoted in the commit, maybe there's something in there
15:36:39 sean-k-mooney stephenfin: i have never seen hardware that actully does not have a dma regoin per numa node
15:37:10 openstackgerrit Lee Yarwood proposed openstack/nova master: DNM test stable rescue tempest/devstack changes https://review.opendev.org/700815
15:37:18 sean-k-mooney stephenfin: and i mean ever in the 7+ years i have been working on this stuff
15:37:50 sean-k-mooney so i am not surprised people have not hit this before
15:38:10 stephenfin efried: I think I'm going to have to get you to weigh in on https://review.opendev.org/#/c/696745/, because I _think_ I've addressed the concerns but everyone else is still avoiding it :)
15:38:38 kashyap sean-k-mooney: Ah, interesting
15:38:59 stephenfin It's almost all refactor noise _except_ the rename of 'nova.network.neutronv2.api' to just 'nova.network.neutron', and the subsequent removal of the 'nova.network.api' shim
15:39:35 kashyap sean-k-mooney: I don't have the data about the hardware assmuption here (which hosts might not have the DMA32 region): https://lkml.org/lkml/2018/7/24/152
15:40:26 sean-k-mooney kashyap: i also suspect that htis was not alway the behavior or kvm by the way
15:41:15 sean-k-mooney i proably had been that way for a long time but i susspect it was altered at somepoint
15:41:20 kashyap sean-k-mooney: FWIW, a bit ago I checked w/ Paolo (one of the upstream KVM devs); he confirmed it's "latent"
15:41:23 kashyap Probably
15:42:49 sean-k-mooney maybe i dont knwo why but if feel like if you grabed ubuntu 12.04 or centos 5.4 it likely wont happen
15:42:55 belmoreira stephenfin our batch VMs (running in dedicated compute nodes with NUMA/huge pages aware) are long lived VM
15:43:32 sean-k-mooney kashyap: i just get the feelign like there was a refactor of this at some point but i dont recall why i have that feeling
15:43:36 belmoreira true that we are seeing this issue for awhile. Until now our solution has been to reboot the affected node.
16:00:12 openstackgerrit Lee Yarwood proposed openstack/nova stable/train: Avoid spurious error logging in _get_compute_nodes_in_db https://review.opendev.org/702902
16:06:12 openstackgerrit Balazs Gibizer proposed openstack/nova stable/pike: Error out interrupted builds https://review.opendev.org/687918
16:06:26 openstackgerrit Balazs Gibizer proposed openstack/nova stable/pike: Fix false ERROR message at compute restart https://review.opendev.org/699496
16:10:14 dansmith efried: do you have any opinions on the newer literal set syntax (i.e {1, 2}) ?
16:10:30 dansmith IMHO, it's one of the worst things ever, because it's super confusing and looks like a dict

Earlier   Later