| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-01-16 | |||
| 14:33:31 | belmoreira | Hi, we are hitting https://review.opendev.org/#/c/684375/3 can we get an agreement on how to proceed? kashyap stephenfin | |
| 14:33:49 | efried | sean-k-mooney: are other nova api comms working in that env? Could it just be a creds-in-config thing? | |
| 14:34:09 | sean-k-mooney | ill try booting a normal vm | |
| 14:34:17 | sean-k-mooney | that shoudl trigger the neutron events | |
| 14:35:51 | sean-k-mooney | ya that booted fine | |
| 14:35:52 | efried | sean-k-mooney: point would be cyborg.conf's creds to [nova] | |
| 14:35:55 | sean-k-mooney | so it could be | |
| 14:36:22 | sean-k-mooney | nova] | |
| 14:36:24 | sean-k-mooney | project_domain_name = Default | |
| 14:36:26 | sean-k-mooney | project_name = service | |
| 14:36:28 | sean-k-mooney | user_domain_name = Default | |
| 14:36:30 | sean-k-mooney | password = password | |
| 14:36:32 | sean-k-mooney | username = nova | |
| 14:36:34 | sean-k-mooney | auth_url = http://192.168.135.77/identity | |
| 14:36:36 | sean-k-mooney | auth_type = password | |
| 14:36:38 | sean-k-mooney | that looks fine to me | |
| 14:37:30 | sean-k-mooney | well fine for devstack | |
| 14:37:32 | stephenfin | belmoreira: Should have been more exact about that. I'm not happy with changing the default for everyone to work around a temporary blip with a buggy kernel (which is what that appears to be) | |
| 14:38:34 | sean-k-mooney | we should not proceed with that patch unless we really really reallly have too | |
| 14:38:49 | stephenfin | sean-k-mooney: are you referring to the hugepages one or something else? | |
| 14:39:10 | efried | stephenfin: would you be okay if we implemented the [workaround] opt as suggested? | |
| 14:39:13 | sean-k-mooney | the mempages mode=preferred change | |
| 14:40:07 | stephenfin | efried: That depends on when the bug was introduced and when it was fixed. Latter seems to be 4.19 but kashyap's notes don't appear to capture the former | |
| 14:40:19 | belmoreira | sean-k-mooney what's your main concern? | |
| 14:40:51 | stephenfin | not sean-k-mooney but I'm going to guess potentially huge performance penalties if you haven't correctly configured your host | |
| 14:42:20 | sean-k-mooney | so i originally suggested perfered to kashyap as a potentil solution but im more conserned about non hugepage guest that use hw:numa_nodes=* | |
| 14:42:34 | kashyap | belmoreira: Hi | |
| 14:42:45 | sean-k-mooney | for hugepage we pin the pages seperately so they will be allocated form the numa node we expect | |
| 14:43:29 | sean-k-mooney | however setting perfered by default means we can be sure that the memory of a gues that has a numa doploty but is not backed by hugepage will come form the numa node we expect | |
| 14:43:58 | sean-k-mooney | which will cause issue if we ever model numa in placment | |
| 14:45:01 | sean-k-mooney | i suspect that most of the time it would likely be fine but im not sure what all the sideeffect will be | |
| 14:45:17 | sean-k-mooney | belmoreira: have you tested the patch and conrimed that it fixes the issue? | |
| 14:45:41 | sean-k-mooney | belmoreira: we were not able to get our customer to do that and we did not have a plathform where we could repoduce the issue | |
| 14:45:59 | sean-k-mooney | belmoreira: so we dont acatully know that will fix it although we suspect it will | |
| 14:46:08 | kashyap | stephenfin: I don't know when exactly it was introduced; I need to do kernel commit message digging (if it's not there, then worse, do bisection myself or ask a KVM dev) | |
| 14:47:10 | sean-k-mooney | stephenfin: belmoreira i would be more ok with his as a workaround config option and keeping the default to strict with the expection we might remvoe it in the future. | |
| 14:48:07 | kashyap | stephenfin: On forcing distros to backport — normally, that would be my approach such kernel cases. But here, we need to bear in mind all the existing OpenStack deployments in the wild. | |
| 14:49:28 | sean-k-mooney | i think this issue is only going to show up on centos/rhel maybe suse | |
| 14:49:56 | kashyap | sean-k-mooney: I think they'd be the distros that are most likely to _backport_ those | |
| 14:50:06 | sean-k-mooney | if you are using the hwe kernel on ubunutu 18.04 you will be on 5.x | |
| 14:50:12 | kashyap | Right | |
| 14:50:55 | sean-k-mooney | i supose if you stuck with the orginal ubuntu 18.04 kernel that was 4.15 | |
| 14:52:06 | kashyap | belmoreira: I take it that rebooting the host kernel is not an option in your deployments? | |
| 14:52:27 | sean-k-mooney | belmoreira: o i see you noted that the patch did solve the problem | |
| 14:52:53 | kashyap | Yeah, he tested it | |
| 14:53:33 | kashyap | sean-k-mooney: And in general, 'preferred' is what we recommend _anyway_, no? | |
| 14:53:57 | sean-k-mooney | no | |
| 14:54:07 | sean-k-mooney | we recommend strict | |
| 14:54:09 | kashyap | I have to remind myself of this everytime. | |
| 14:54:43 | sean-k-mooney | prefered makes the memory allcoation of non hugepage memory a hint not a requirement | |
| 14:54:46 | kashyap | "Where, 'preferred' means: disable NUMA affinity and so the | |
| 14:54:46 | kashyap | request is turned to a "hint", i.e. "if possible, allocate from the | |
| 14:54:48 | kashyap | given NUMA node; otherwise, fallback to other nodes" | |
| 14:54:54 | kashyap | Yes | |
| 14:55:10 | sean-k-mooney | which will break our numa aware memory tracking | |
| 14:55:24 | sean-k-mooney | and will not work if we model memory in plamcent in the future per numa node | |
| 14:55:36 | sean-k-mooney | since we can rely on the memory being allcoated form the numa node we said | |
| 14:55:58 | sean-k-mooney | for hugepage memeory the option will be ignored so hugepage backed vms will be fine | |
| 14:56:36 | kashyap | Yeah, then I don't want to muck with defaults. With all these future caveats (which are not properly documented.) | |
| 14:56:40 | sean-k-mooney | but vm with a numa toplogy but no hugepages could have there memory allcoated from any numa node if the one we select was full | |
| 14:58:21 | sean-k-mooney | stephenfin: kashyap belmoreira i left a comment on the patch | |
| 15:14:40 | belmoreira | sorry, needed to leave the office | |
| 15:15:50 | belmoreira | sean-k-mooney yes, I can reproduce with centos 7.6 kernel: Linux XXX 3.10.0-957.1.3.el7.x86_64 #1 SMP Thu Nov 29 14:49:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | |
| 15:15:59 | belmoreira | libvirtd 4.5.0-23 | |
| 15:16:05 | belmoreira | qemu 2.12.0-33 | |
| 15:16:40 | belmoreira | changing to "preferred" fix it (can create new instances) | |
| 15:16:58 | sean-k-mooney | ya as far as i know this is not fixed in 7.6 | |
| 15:17:07 | sean-k-mooney | i think the fix should be in 7.8 | |
| 15:17:20 | belmoreira | kashyap we upgraded to centos 7.7 but didn't reboot the compute nodes. It's a very painful process | |
| 15:17:32 | sean-k-mooney | it might be in 7.7 | |
| 15:18:02 | sean-k-mooney | belmoreira: are you ok with it if we make it a workaround config option and default to strict | |
| 15:18:10 | belmoreira | sean-k-mooney for now I can't reproduce it with 7.7 | |
| 15:18:29 | sean-k-mooney | ok cool then that likely already has the backport | |
| 15:18:41 | sean-k-mooney | although this is hardware specific | |
| 15:18:54 | sean-k-mooney | so if its a differen model fo server then you migh not see it | |
| 15:19:21 | belmoreira | sean-k-mooney yes, took me some time to isolate this | |
| 15:20:08 | sean-k-mooney | ya it took us a day or two to figure this out downstream too when it was first reported to us | |
| 15:20:18 | sean-k-mooney | its a very rare edgecase | |
| 15:20:28 | sean-k-mooney | normlaly you will have dma regions per socket | |
| 15:21:03 | sean-k-mooney | or numa node | |
| 15:21:14 | belmoreira | I think that the workaround config will be useful for everyone that is running NUMA aware VMs. Then needs to be backported to older releases | |
| 15:21:29 | sean-k-mooney | so you need a server that does not + requst a numa toploy + have a kernel with the kvm bug | |
| 15:22:08 | sean-k-mooney | well no it will be useful for the minority that have ^ | |
| 15:22:38 | sean-k-mooney | but i would be ok with proceeding on that basis | |
| 15:22:43 | belmoreira | sean-k-mooney ok, true :) | |
| 15:24:27 | belmoreira | but also shows that the nova community is aware of the issue and has a work around | |
| 15:24:32 | belmoreira | sean-k-mooney thanks | |
| 15:25:18 | kashyap | belmoreira: We're very much aware, FWIW, see this response on 'openstack-discuss' list: http://lists.openstack.org/pipermail/openstack-discuss/2019-September/009770.html | |
| 15:26:08 | belmoreira | sean-k-mooney but to clarify, "preferred" would break the future placement tracking and this should be disable as soon the new fixed kernel is loaded | |
| 15:26:37 | sean-k-mooney | that placement tracking proably wont be a thing until V | |
| 15:26:50 | sean-k-mooney | i could mess up our tracking in the resouce tracker today | |
| 15:27:18 | sean-k-mooney | but if you have the choice between not being able to use an entire socket | |
| 15:27:35 | sean-k-mooney | or slightly reduced performace im guess your ok with the slight perfromace hit | |
| 15:27:48 | sean-k-mooney | and slightly in accurate tracking | |
| 15:28:49 | sean-k-mooney | belmoreira: my conser is only for instace with a numa toplogy that dont use hugepages | |
| 15:29:26 | belmoreira | got it | |
| 15:29:47 | sean-k-mooney | hugepage guess will be tracked properly in either case | |
| 15:30:14 | belmoreira | thanks a lot for this discussion and proposal | |
| 15:32:22 | Sundar | sean-k-mooney: Please make sure to pull in https://review.opendev.org/698846 and https://review.opendev.org/700261 | |
| 15:32:45 | Sundar | They are called out as dependencies in the create/bind patch | |