| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-06-18 | |||
| 13:15:59 | sean-k-mooney | so it is reported as -1 meaning no numa affinity in the guest | |
| 13:16:55 | sean-k-mooney | to change that we would likely have to use the q35 machine type and create a pcie root complete per numa node then add the passthough deivice to the correct pci root | |
| 13:17:45 | sean-k-mooney | that has other implciation mainliy that on move operation either we have to allow the toploty to change or we have to limit the host we can select to maintain the current toplogy | |
| 13:18:14 | sean-k-mooney | if we allow the toplogy to change the virtual pci address of the devices in the guest would also change | |
| 13:19:16 | sean-k-mooney | rmart04: but yes if you have a multi numa node guest this can result in cross numa traffic worse case twice because you dont know the numa affinity of the device | |
| 13:20:18 | openstackgerrit | Merged openstack/nova master: libvirt: Mark e1000e VIF as supported https://review.opendev.org/734777 | |
| 13:22:32 | rmart04 | OK, appreciate all the info SeanKMooney. I guess another way around this is to split the host into two guests, one on each numa node with their associated pci-passthrough devices (GPUs). Currently I appear to be blocked on this by my older kernel. 3.10. I bump into an issue allocating memory from the second NUMA node for the second machine. I believe this is fixed in 4.14. | |
| 13:23:06 | rmart04 | qq, you mention the q35 machine type, what type do we use by default? | |
| 13:25:51 | sean-k-mooney | rmart04: if the guest has a numa toploty we do not allow its memory to come form a remote numa node by design | |
| 13:26:01 | sean-k-mooney | rmart04: we use pc | |
| 13:26:23 | sean-k-mooney | or pc-i440fx | |
| 13:26:27 | sean-k-mooney | something like that | |
| 13:26:43 | sean-k-mooney | rmart04: what version of openstack are you using | |
| 13:27:02 | rmart04 | Rocky (Stein upgrade this weekend) | |
| 13:27:26 | sean-k-mooney | do you have gpus on all host numa nodes | |
| 13:27:30 | sean-k-mooney | or just numa 0 | |
| 13:27:42 | rmart04 | Yep, 2 sockets, 8 GPUs | |
| 13:27:50 | rmart04 | 4 each | |
| 13:28:16 | sean-k-mooney | ok what iw was going to say is you might need to use nuam_policy=preferred in the alias | |
| 13:28:21 | sean-k-mooney | if you did not have them split | |
| 13:28:54 | sean-k-mooney | if you do then yes 2 vms with 1 numa each and the default legacy polciy which enforce numa affintiy between cpu/memory and the pci device is what you will want | |
| 13:29:42 | sean-k-mooney | you can create a dual numa guest but the limitation is you will know know what numa node in the guest maps to the actull location of the device on the host | |
| 13:29:59 | sean-k-mooney | rmart04: are your vms using 1 gpu earch or multiple | |
| 13:30:16 | sean-k-mooney | it wont affect the answer just wondering | |
| 13:30:24 | rmart04 | Initial approach was 1VM 8 GPUs, second approach is 2VM's 4 each | |
| 13:31:00 | sean-k-mooney | cool if you can horizontally scale then yes 2 vm with a singel numa node each shoudl give better performance | |
| 13:31:25 | sean-k-mooney | since there will be no corss numa trafic fo the vm cpu memory and gpus | |
| 13:32:24 | rmart04 | Thats the plan, but previously I tried this and got a cannot allocate memory issue, which seemed to be related to no dma32 on node1 in /proc/zoneinfo. Which I believe may be due to the older kernel | |
| 13:33:27 | sean-k-mooney | rmart04: oh you hit that | |
| 13:33:56 | sean-k-mooney | so that is not really a kernel issue so much as a kernl/bios/firmware issue that we worked around with a kvm change | |
| 13:34:17 | sean-k-mooney | rmart04: really there should have been a dma32 region allcoated per numa node | |
| 13:34:34 | sean-k-mooney | the kvm fix was not to require numa affinity for the dma32 region | |
| 13:34:39 | rmart04 | Oh right, interesting. Could you point me at the info for the kvm change? | |
| 13:34:50 | rmart04 | ah Ok, is that strict=false or similar | |
| 13:35:08 | sean-k-mooney | kind of but that would have done it for all the vms memory | |
| 13:35:15 | sean-k-mooney | that was the alternitive workaround | |
| 13:35:34 | rmart04 | Please tell me its fixed in Stein? :D | |
| 13:36:43 | sean-k-mooney | https://lkml.org/lkml/2018/7/24/843 | |
| 13:36:51 | sean-k-mooney | this is not an openstack bug | |
| 13:36:58 | sean-k-mooney | so we did not modify nova | |
| 13:37:09 | sean-k-mooney | what distro are you using | |
| 13:37:32 | rmart04 | ah OK, Yes this is what I was looking at, I thought it was a Kernel patch | |
| 13:37:37 | rmart04 | Centos7 | |
| 13:37:45 | rmart04 | 3.10 kernel | |
| 13:37:55 | sean-k-mooney | it is for the kvm kernel module | |
| 13:38:08 | sean-k-mooney | there might have been another patch too | |
| 13:38:47 | sean-k-mooney | ok i know we backported this in rhel 7 | |
| 13:38:52 | sean-k-mooney | may in 7.6 | |
| 13:39:04 | sean-k-mooney | so hopefully you have that in the lates centos 7 too | |
| 13:39:16 | sean-k-mooney | let me see if i have the bz for it in my history | |
| 13:39:25 | rmart04 | ah that would be amazing | |
| 13:43:30 | sean-k-mooney | so this is the nova patch we decied not to go with https://review.opendev.org/#/c/684375/ partly because we could not test it | |
| 13:43:53 | sean-k-mooney | rmart04: the commit meassage has the links to the relevent bugs and converations | |
| 13:45:39 | openstack | bugzilla.redhat.com bug 1010885 in libvirt "kvm_init_vcpu failed: Cannot allocate memory in NUMA" [Medium,Closed: errata] - Assigned to mkletzan | |
| 13:45:39 | sean-k-mooney | hum it look like https://bugzilla.redhat.com/show_bug.cgi?id=1010885#c2 might also be a workaround but i dont think it is | |
| 13:52:41 | rmart04 | Remove cpuset from cgroup controllers? | |
| 13:53:11 | rmart04 | Is that what also makes the pinning work? | |
| 13:53:13 | sean-k-mooney | rmart04: yes but i dont know if that fully disables pinning | |
| 13:53:58 | sean-k-mooney | so the issue is that the wya libvirt appliees the cgrpus it also confines the allcoations of kernel memory | |
| 13:54:37 | sean-k-mooney | one of the fixes that was only a partial fix was to move that later so that the dma region could be allocate before the cpus are pinned | |
| 13:54:55 | sean-k-mooney | that was done in https://libvirt.org/git/?p=libvirt.git;a=commit;h=7e72ac7878 | |
| 13:55:24 | sean-k-mooney | but that was backin 2014 so it obviouslyu was not a full fix or it was broken angain later | |
| 13:56:01 | rmart04 | OK :/ | |
| 13:59:57 | sean-k-mooney | rmart04: this was the final kernel fix i belive https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=ee6268ba3a68 | |
| 14:00:01 | sean-k-mooney | that was in 4.19 | |
| 14:00:15 | rmart04 | ah ok, my bad I said 4.14 earliar | |
| 14:01:00 | rmart04 | How easy is it to find out whether it was backported in C7? | |
| 14:03:43 | sean-k-mooney | its in the rhel kernel-3.10.0-957.26.1.el7 pacakge and needs libivrt ibvirt-4.5.0-13.el7 or higher | |
| 14:06:26 | rmart04 | OK thats amazing thank you. I think i'll see how far we are away from those package versions and expidite moving to them | |
| 14:46:06 | mordred | sean-k-mooney: you remember every conversation we've had about things from the past, right? | |
| 14:46:29 | sean-k-mooney | mordred: ocourse i was right and you were .... | |
| 14:46:42 | sean-k-mooney | mordred: is there a converstation in partcalar? | |
| 14:46:47 | mordred | sean-k-mooney: what's the story with the api_servers config option for glance - we've asked about deprecating/removing it and getting rid of the idea of being a poor-mans-client-side-load-balancer | |
| 14:47:03 | mordred | but I can't remember where the discussion got to on that | |
| 14:47:33 | sean-k-mooney | right we wanted to not do the crappy round robin thing we do in nova anymore | |
| 14:47:37 | mordred | (I'm trying to help cyborg with their glance support, but since it's copied from nova I wnat to make sure anything I do would eventually be transferrable back to nova) | |
| 14:47:41 | sean-k-mooney | i belive dansmith was onboard with that | |
| 14:47:54 | sean-k-mooney | i dont know if we have done anything to change it in nova | |
| 14:47:58 | mordred | yeah. I guess I should look to see if we marked the option as deprecated yet | |
| 14:48:15 | sean-k-mooney | mordred: efried did bring it up but then he had to move on | |
| 14:48:38 | dansmith | I was not on board with removing it, | |
| 14:48:46 | dansmith | but I think I was the only nova person in that position | |
| 14:49:01 | mordred | Support for image service configuration via standard keystoneauth1 Adapter | |
| 14:49:02 | mordred | options was added in the 17.0.0 Queens release. The api_servers option was | |
| 14:49:04 | mordred | retained temporarily to allow consumers time to cut over to a real load | |
| 14:49:06 | mordred | balancing solution. | |
| 14:49:14 | mordred | there's the help text- so it implies that post-queens its existence is temporary | |
| 14:49:24 | mordred | but with no official deprecation story :) | |
| 14:50:00 | mordred | basically - I would like to either kill this or if we can never kill it support it in ksa so that we can stop it with copying the round-robin code everywhere | |
| 14:50:02 | sean-k-mooney | dansmith: oh i just rememebered you had an opion and i generally rememeber when you dont like something so i assume you were ok with it | |
| 14:51:08 | efried | I would have deprecated it if I had been allowed to. I may be misremembering, but I think we put out a RFC on the ML and someone put up their hand and said they were still using it. Might even have been dansmith :P | |
| 14:51:45 | mordred | efried: :) | |
| 14:51:48 | dansmith | no, wasn't me, | |
| 14:52:14 | dansmith | but there are people in redhat, tripleo and edge-related IIRC, that definitely don't want to lose it | |
| 14:52:33 | sean-k-mooney | because its used for rabbit mq? | |
| 14:52:34 | efried | Also, I think I commented on the cyborg stuff when it went in, saying they really shouldn't be carrying all this warty stuff over from nova -- that is, they *never* should have supported [glance]api_servers. But I think they wound up just merging it for expediency. | |
| 14:52:40 | mordred | I thought all those people thought k8s was super sexy - why is a lb hard? | |
| 14:53:04 | dansmith | the history is a little dim for me without digging that back up, but basically for a very small number of remote edge machines, a "real load balancer" is not an option and having nova be able to try multiple glance servers is a major win | |
| 14:53:15 | mordred | nod | |
| 14:53:26 | efried | If ^ is not an issue for cyborg, I say they kill it, with prejudice. | |