Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-01-16
14:54:09 kashyap I have to remind myself of this everytime.
14:54:43 sean-k-mooney prefered makes the memory allcoation of non hugepage memory a hint not a requirement
14:54:46 kashyap request is turned to a "hint", i.e. "if possible, allocate from the
14:54:46 kashyap "Where, 'preferred' means: disable NUMA affinity and so the
14:54:48 kashyap given NUMA node; otherwise, fallback to other nodes"
14:54:54 kashyap Yes
14:55:10 sean-k-mooney which will break our numa aware memory tracking
14:55:24 sean-k-mooney and will not work if we model memory in plamcent in the future per numa node
14:55:36 sean-k-mooney since we can rely on the memory being allcoated form the numa node we said
14:55:58 sean-k-mooney for hugepage memeory the option will be ignored so hugepage backed vms will be fine
14:56:36 kashyap Yeah, then I don't want to muck with defaults. With all these future caveats (which are not properly documented.)
14:56:40 sean-k-mooney but vm with a numa toplogy but no hugepages could have there memory allcoated from any numa node if the one we select was full
14:58:21 sean-k-mooney stephenfin: kashyap belmoreira i left a comment on the patch
15:14:40 belmoreira sorry, needed to leave the office
15:15:50 belmoreira sean-k-mooney yes, I can reproduce with centos 7.6 kernel: Linux XXX 3.10.0-957.1.3.el7.x86_64 #1 SMP Thu Nov 29 14:49:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
15:15:59 belmoreira libvirtd 4.5.0-23
15:16:05 belmoreira qemu 2.12.0-33
15:16:40 belmoreira changing to "preferred" fix it (can create new instances)
15:16:58 sean-k-mooney ya as far as i know this is not fixed in 7.6
15:17:07 sean-k-mooney i think the fix should be in 7.8
15:17:20 belmoreira kashyap we upgraded to centos 7.7 but didn't reboot the compute nodes. It's a very painful process
15:17:32 sean-k-mooney it might be in 7.7
15:18:02 sean-k-mooney belmoreira: are you ok with it if we make it a workaround config option and default to strict
15:18:10 belmoreira sean-k-mooney for now I can't reproduce it with 7.7
15:18:29 sean-k-mooney ok cool then that likely already has the backport
15:18:41 sean-k-mooney although this is hardware specific
15:18:54 sean-k-mooney so if its a differen model fo server then you migh not see it
15:19:21 belmoreira sean-k-mooney yes, took me some time to isolate this
15:20:08 sean-k-mooney ya it took us a day or two to figure this out downstream too when it was first reported to us
15:20:18 sean-k-mooney its a very rare edgecase
15:20:28 sean-k-mooney normlaly you will have dma regions per socket
15:21:03 sean-k-mooney or numa node
15:21:14 belmoreira I think that the workaround config will be useful for everyone that is running NUMA aware VMs. Then needs to be backported to older releases
15:21:29 sean-k-mooney so you need a server that does not + requst a numa toploy + have a kernel with the kvm bug
15:22:08 sean-k-mooney well no it will be useful for the minority that have ^
15:22:38 sean-k-mooney but i would be ok with proceeding on that basis
15:22:43 belmoreira sean-k-mooney ok, true :)
15:24:27 belmoreira but also shows that the nova community is aware of the issue and has a work around
15:24:32 belmoreira sean-k-mooney thanks
15:25:18 kashyap belmoreira: We're very much aware, FWIW, see this response on 'openstack-discuss' list: http://lists.openstack.org/pipermail/openstack-discuss/2019-September/009770.html
15:26:08 belmoreira sean-k-mooney but to clarify, "preferred" would break the future placement tracking and this should be disable as soon the new fixed kernel is loaded
15:26:37 sean-k-mooney that placement tracking proably wont be a thing until V
15:26:50 sean-k-mooney i could mess up our tracking in the resouce tracker today
15:27:18 sean-k-mooney but if you have the choice between not being able to use an entire socket
15:27:35 sean-k-mooney or slightly reduced performace im guess your ok with the slight perfromace hit
15:27:48 sean-k-mooney and slightly in accurate tracking
15:28:49 sean-k-mooney belmoreira: my conser is only for instace with a numa toplogy that dont use hugepages
15:29:26 belmoreira got it
15:29:47 sean-k-mooney hugepage guess will be tracked properly in either case
15:30:14 belmoreira thanks a lot for this discussion and proposal
15:32:22 Sundar sean-k-mooney: Please make sure to pull in https://review.opendev.org/698846 and https://review.opendev.org/700261
15:32:45 Sundar They are called out as dependencies in the create/bind patch
15:33:14 Sundar i.e. https://review.opendev.org/#/c/631244/
15:33:38 sean-k-mooney i cant have devstack pull in both if they are not in the same patch change
15:33:42 sean-k-mooney *chain
15:33:53 sean-k-mooney so ill have to create a local branch and merge them in
15:34:10 sean-k-mooney i have https://review.opendev.org/#/c/698846/ but not https://review.opendev.org/#/c/700261/
15:34:13 kashyap belmoreira: stephenfin: efried: The kernel issue (https://review.opendev.org/#/c/684375/) has been latent. (Just confirmed w/ a KVM mainatiner.)
15:34:19 Sundar Both are listed as dependencies in the same Nova patch
15:34:42 kashyap sean-k-mooney: (Responded on the change.) Thanks!
15:34:46 sean-k-mooney yes but devstack can only clone one it does not have the zuul logic to create a merge commti from both
15:35:00 sean-k-mooney but i can do it manually and restart the services
15:35:19 stephenfin kashyap: I wonder why there wasn't a sudden spike so? Surely this would have been affecting belmoreira and co. for years now if it's been there forever
15:35:22 sean-k-mooney Sundar: in general depends-on should not be used within the same repo
15:35:46 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Add support for stable device rescue https://review.opendev.org/700812
15:35:46 openstackgerrit Lee Yarwood proposed openstack/nova master: virt: Provide block_device_info during rescue https://review.opendev.org/700811
15:35:47 kashyap stephenfin: Yeah, I don't have a solid answer to that yet :-(
15:35:47 openstackgerrit Lee Yarwood proposed openstack/nova master: docs: Add stable device rescue docs https://review.opendev.org/700837
15:35:50 sean-k-mooney https://review.opendev.org/#/c/698846/ should be rebased on https://review.opendev.org/#/c/700261/
15:36:14 kashyap stephenfin: Let me see the original kernel report I quoted in the commit, maybe there's something in there
15:36:39 sean-k-mooney stephenfin: i have never seen hardware that actully does not have a dma regoin per numa node
15:37:10 openstackgerrit Lee Yarwood proposed openstack/nova master: DNM test stable rescue tempest/devstack changes https://review.opendev.org/700815
15:37:18 sean-k-mooney stephenfin: and i mean ever in the 7+ years i have been working on this stuff
15:37:50 sean-k-mooney so i am not surprised people have not hit this before
15:38:10 stephenfin efried: I think I'm going to have to get you to weigh in on https://review.opendev.org/#/c/696745/, because I _think_ I've addressed the concerns but everyone else is still avoiding it :)
15:38:38 kashyap sean-k-mooney: Ah, interesting
15:38:59 stephenfin It's almost all refactor noise _except_ the rename of 'nova.network.neutronv2.api' to just 'nova.network.neutron', and the subsequent removal of the 'nova.network.api' shim
15:39:35 kashyap sean-k-mooney: I don't have the data about the hardware assmuption here (which hosts might not have the DMA32 region): https://lkml.org/lkml/2018/7/24/152
15:40:26 sean-k-mooney kashyap: i also suspect that htis was not alway the behavior or kvm by the way
15:41:15 sean-k-mooney i proably had been that way for a long time but i susspect it was altered at somepoint
15:41:20 kashyap sean-k-mooney: FWIW, a bit ago I checked w/ Paolo (one of the upstream KVM devs); he confirmed it's "latent"
15:41:23 kashyap Probably
15:42:49 sean-k-mooney maybe i dont knwo why but if feel like if you grabed ubuntu 12.04 or centos 5.4 it likely wont happen
15:42:55 belmoreira stephenfin our batch VMs (running in dedicated compute nodes with NUMA/huge pages aware) are long lived VM
15:43:32 sean-k-mooney kashyap: i just get the feelign like there was a refactor of this at some point but i dont recall why i have that feeling
15:43:36 belmoreira true that we are seeing this issue for awhile. Until now our solution has been to reboot the affected node.
16:00:12 openstackgerrit Lee Yarwood proposed openstack/nova stable/train: Avoid spurious error logging in _get_compute_nodes_in_db https://review.opendev.org/702902
16:06:12 openstackgerrit Balazs Gibizer proposed openstack/nova stable/pike: Error out interrupted builds https://review.opendev.org/687918
16:06:26 openstackgerrit Balazs Gibizer proposed openstack/nova stable/pike: Fix false ERROR message at compute restart https://review.opendev.org/699496
16:10:14 dansmith efried: do you have any opinions on the newer literal set syntax (i.e {1, 2}) ?
16:10:30 dansmith IMHO, it's one of the worst things ever, because it's super confusing and looks like a dict
16:10:59 dansmith just wondering if anyone else would be interested in a developer convention in nova of not adding code with that syntax to our tree
16:11:48 gibi dansmith: I have no problem with the literal set sytnax for me the absense of ':' is enough to notice that it is a set
16:12:06 dansmith it's super easy to not notice that there's no : though
16:12:17 gibi I guess we are different
16:12:20 openstackgerrit Lee Yarwood proposed openstack/nova stable/train: Ensure source service is up before resizing/migrating https://review.opendev.org/701757
16:12:40 dansmith gibi: that's fine, I asked because I was wondering if I'm the only one
16:18:53 artom dansmith, as per your request, I'm not letting you forget https://review.opendev.org/#/c/687404/8
16:20:30 gibi dansmith: I think for me it is mostly about the fact that I used to it. I guess I would be pretty against assignment expression because that would be something I never used before in python
16:21:09 dansmith artom: okay I'm in the middle of one of the cyborg patches, which is making me angry because apparently validating that our new methods are called like we expect them to is far too high a bar

Earlier   Later