| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-06-18 | |||
| 13:23:06 | rmart04 | qq, you mention the q35 machine type, what type do we use by default? | |
| 13:25:51 | sean-k-mooney | rmart04: if the guest has a numa toploty we do not allow its memory to come form a remote numa node by design | |
| 13:26:01 | sean-k-mooney | rmart04: we use pc | |
| 13:26:23 | sean-k-mooney | or pc-i440fx | |
| 13:26:27 | sean-k-mooney | something like that | |
| 13:26:43 | sean-k-mooney | rmart04: what version of openstack are you using | |
| 13:27:02 | rmart04 | Rocky (Stein upgrade this weekend) | |
| 13:27:26 | sean-k-mooney | do you have gpus on all host numa nodes | |
| 13:27:30 | sean-k-mooney | or just numa 0 | |
| 13:27:42 | rmart04 | Yep, 2 sockets, 8 GPUs | |
| 13:27:50 | rmart04 | 4 each | |
| 13:28:16 | sean-k-mooney | ok what iw was going to say is you might need to use nuam_policy=preferred in the alias | |
| 13:28:21 | sean-k-mooney | if you did not have them split | |
| 13:28:54 | sean-k-mooney | if you do then yes 2 vms with 1 numa each and the default legacy polciy which enforce numa affintiy between cpu/memory and the pci device is what you will want | |
| 13:29:42 | sean-k-mooney | you can create a dual numa guest but the limitation is you will know know what numa node in the guest maps to the actull location of the device on the host | |
| 13:29:59 | sean-k-mooney | rmart04: are your vms using 1 gpu earch or multiple | |
| 13:30:16 | sean-k-mooney | it wont affect the answer just wondering | |
| 13:30:24 | rmart04 | Initial approach was 1VM 8 GPUs, second approach is 2VM's 4 each | |
| 13:31:00 | sean-k-mooney | cool if you can horizontally scale then yes 2 vm with a singel numa node each shoudl give better performance | |
| 13:31:25 | sean-k-mooney | since there will be no corss numa trafic fo the vm cpu memory and gpus | |
| 13:32:24 | rmart04 | Thats the plan, but previously I tried this and got a cannot allocate memory issue, which seemed to be related to no dma32 on node1 in /proc/zoneinfo. Which I believe may be due to the older kernel | |
| 13:33:27 | sean-k-mooney | rmart04: oh you hit that | |
| 13:33:56 | sean-k-mooney | so that is not really a kernel issue so much as a kernl/bios/firmware issue that we worked around with a kvm change | |
| 13:34:17 | sean-k-mooney | rmart04: really there should have been a dma32 region allcoated per numa node | |
| 13:34:34 | sean-k-mooney | the kvm fix was not to require numa affinity for the dma32 region | |
| 13:34:39 | rmart04 | Oh right, interesting. Could you point me at the info for the kvm change? | |
| 13:34:50 | rmart04 | ah Ok, is that strict=false or similar | |
| 13:35:08 | sean-k-mooney | kind of but that would have done it for all the vms memory | |
| 13:35:15 | sean-k-mooney | that was the alternitive workaround | |
| 13:35:34 | rmart04 | Please tell me its fixed in Stein? :D | |
| 13:36:43 | sean-k-mooney | https://lkml.org/lkml/2018/7/24/843 | |
| 13:36:51 | sean-k-mooney | this is not an openstack bug | |
| 13:36:58 | sean-k-mooney | so we did not modify nova | |
| 13:37:09 | sean-k-mooney | what distro are you using | |
| 13:37:32 | rmart04 | ah OK, Yes this is what I was looking at, I thought it was a Kernel patch | |
| 13:37:37 | rmart04 | Centos7 | |
| 13:37:45 | rmart04 | 3.10 kernel | |
| 13:37:55 | sean-k-mooney | it is for the kvm kernel module | |
| 13:38:08 | sean-k-mooney | there might have been another patch too | |
| 13:38:47 | sean-k-mooney | ok i know we backported this in rhel 7 | |
| 13:38:52 | sean-k-mooney | may in 7.6 | |
| 13:39:04 | sean-k-mooney | so hopefully you have that in the lates centos 7 too | |
| 13:39:16 | sean-k-mooney | let me see if i have the bz for it in my history | |
| 13:39:25 | rmart04 | ah that would be amazing | |
| 13:43:30 | sean-k-mooney | so this is the nova patch we decied not to go with https://review.opendev.org/#/c/684375/ partly because we could not test it | |
| 13:43:53 | sean-k-mooney | rmart04: the commit meassage has the links to the relevent bugs and converations | |
| 13:45:39 | openstack | bugzilla.redhat.com bug 1010885 in libvirt "kvm_init_vcpu failed: Cannot allocate memory in NUMA" [Medium,Closed: errata] - Assigned to mkletzan | |
| 13:45:39 | sean-k-mooney | hum it look like https://bugzilla.redhat.com/show_bug.cgi?id=1010885#c2 might also be a workaround but i dont think it is | |
| 13:52:41 | rmart04 | Remove cpuset from cgroup controllers? | |
| 13:53:11 | rmart04 | Is that what also makes the pinning work? | |
| 13:53:13 | sean-k-mooney | rmart04: yes but i dont know if that fully disables pinning | |
| 13:53:58 | sean-k-mooney | so the issue is that the wya libvirt appliees the cgrpus it also confines the allcoations of kernel memory | |
| 13:54:37 | sean-k-mooney | one of the fixes that was only a partial fix was to move that later so that the dma region could be allocate before the cpus are pinned | |
| 13:54:55 | sean-k-mooney | that was done in https://libvirt.org/git/?p=libvirt.git;a=commit;h=7e72ac7878 | |
| 13:55:24 | sean-k-mooney | but that was backin 2014 so it obviouslyu was not a full fix or it was broken angain later | |
| 13:56:01 | rmart04 | OK :/ | |
| 13:59:57 | sean-k-mooney | rmart04: this was the final kernel fix i belive https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=ee6268ba3a68 | |
| 14:00:01 | sean-k-mooney | that was in 4.19 | |
| 14:00:15 | rmart04 | ah ok, my bad I said 4.14 earliar | |
| 14:01:00 | rmart04 | How easy is it to find out whether it was backported in C7? | |
| 14:03:43 | sean-k-mooney | its in the rhel kernel-3.10.0-957.26.1.el7 pacakge and needs libivrt ibvirt-4.5.0-13.el7 or higher | |
| 14:06:26 | rmart04 | OK thats amazing thank you. I think i'll see how far we are away from those package versions and expidite moving to them | |
| 14:46:06 | mordred | sean-k-mooney: you remember every conversation we've had about things from the past, right? | |
| 14:46:29 | sean-k-mooney | mordred: ocourse i was right and you were .... | |
| 14:46:42 | sean-k-mooney | mordred: is there a converstation in partcalar? | |
| 14:46:47 | mordred | sean-k-mooney: what's the story with the api_servers config option for glance - we've asked about deprecating/removing it and getting rid of the idea of being a poor-mans-client-side-load-balancer | |
| 14:47:03 | mordred | but I can't remember where the discussion got to on that | |
| 14:47:33 | sean-k-mooney | right we wanted to not do the crappy round robin thing we do in nova anymore | |
| 14:47:37 | mordred | (I'm trying to help cyborg with their glance support, but since it's copied from nova I wnat to make sure anything I do would eventually be transferrable back to nova) | |
| 14:47:41 | sean-k-mooney | i belive dansmith was onboard with that | |
| 14:47:54 | sean-k-mooney | i dont know if we have done anything to change it in nova | |
| 14:47:58 | mordred | yeah. I guess I should look to see if we marked the option as deprecated yet | |
| 14:48:15 | sean-k-mooney | mordred: efried did bring it up but then he had to move on | |
| 14:48:38 | dansmith | I was not on board with removing it, | |
| 14:48:46 | dansmith | but I think I was the only nova person in that position | |
| 14:49:01 | mordred | Support for image service configuration via standard keystoneauth1 Adapter | |
| 14:49:02 | mordred | options was added in the 17.0.0 Queens release. The api_servers option was | |
| 14:49:04 | mordred | retained temporarily to allow consumers time to cut over to a real load | |
| 14:49:06 | mordred | balancing solution. | |
| 14:49:14 | mordred | there's the help text- so it implies that post-queens its existence is temporary | |
| 14:49:24 | mordred | but with no official deprecation story :) | |
| 14:50:00 | mordred | basically - I would like to either kill this or if we can never kill it support it in ksa so that we can stop it with copying the round-robin code everywhere | |
| 14:50:02 | sean-k-mooney | dansmith: oh i just rememebered you had an opion and i generally rememeber when you dont like something so i assume you were ok with it | |
| 14:51:08 | efried | I would have deprecated it if I had been allowed to. I may be misremembering, but I think we put out a RFC on the ML and someone put up their hand and said they were still using it. Might even have been dansmith :P | |
| 14:51:45 | mordred | efried: :) | |
| 14:51:48 | dansmith | no, wasn't me, | |
| 14:52:14 | dansmith | but there are people in redhat, tripleo and edge-related IIRC, that definitely don't want to lose it | |
| 14:52:33 | sean-k-mooney | because its used for rabbit mq? | |
| 14:52:34 | efried | Also, I think I commented on the cyborg stuff when it went in, saying they really shouldn't be carrying all this warty stuff over from nova -- that is, they *never* should have supported [glance]api_servers. But I think they wound up just merging it for expediency. | |
| 14:52:40 | mordred | I thought all those people thought k8s was super sexy - why is a lb hard? | |
| 14:53:04 | dansmith | the history is a little dim for me without digging that back up, but basically for a very small number of remote edge machines, a "real load balancer" is not an option and having nova be able to try multiple glance servers is a major win | |
| 14:53:15 | mordred | nod | |
| 14:53:26 | efried | If ^ is not an issue for cyborg, I say they kill it, with prejudice. | |
| 14:53:33 | mordred | ok - that's fair enough I suppose | |
| 14:53:39 | sean-k-mooney | dansmith: really? ha proxy is pretty light weight | |
| 14:53:59 | efried | esp if it means they're no longer anchored to ksa and can cut over to sdk | |
| 14:54:04 | sean-k-mooney | im not sure i by that given it will be handleing very little traffic | |
| 14:54:14 | dansmith | sean-k-mooney: it's not just the haproxy of course, it's the config, the need for shared L2 failover VIP, pacemaker to manage it, etc | |
| 14:54:36 | dansmith | it's all the complexity that comes with deploying that architecture | |
| 14:55:11 | mordred | sure. although I'll again point my fingers at all of the people rushing to install k8s in those contexts so they should have an easier time of it | |