| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-03-04 | |||
| 17:31:59 | stephenfin | As above, I say we get a fix and functional test for this and then debate it there | |
| 17:32:00 | sean-k-mooney | stephenfin: we dont but i think claudiub is missing hw:mem_page_size | |
| 17:32:07 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Replace blind retry with libvirt event waiting in detach https://review.opendev.org/c/openstack/nova/+/770246 | |
| 17:32:34 | gibi | I rest my case (at least for today) | |
| 17:32:39 | sean-k-mooney | stephenfin: we said we should start default that to hw:mem_page_size=any for all numa instnaces | |
| 17:32:40 | gibi | see you tomorrow | |
| 17:32:40 | claudiub | hm, I only had the "hw:numa_nodes=1" extra_spec | |
| 17:32:41 | lyarwood | gibi: awesome, I'll queue that up in the morning :) | |
| 17:32:46 | stephenfin | sean-k-mooney: that's a legit request | |
| 17:32:49 | lyarwood | gibi: \o | |
| 17:32:49 | sean-k-mooney | claudiub: yep that incorect | |
| 17:33:13 | gibi | lyarwood: note that it is still not all your comment fixed, but I will add one more separate patch to fix thoes | |
| 17:33:14 | sean-k-mooney | claudiub: you have enbaled numa node but not enable either numa aware cpu tracking or memory tracking | |
| 17:34:10 | sean-k-mooney | if you set hw:numa_nodes=1 and dont set hw:mem_page_size to any valid value its not correct | |
| 17:34:41 | stephenfin | that's a bit strong. It's fine if you don't have hugepages on the host | |
| 17:34:51 | stephenfin | or other things taking memory that you haven't accounted for | |
| 17:34:53 | sean-k-mooney | stephenfin: i did not say use hugepages | |
| 17:35:06 | stephenfin | setting hw:mem_page_size disables memory overcommit | |
| 17:35:08 | sean-k-mooney | stephenfin: i said set hw:mem_page_size to any value | |
| 17:35:26 | sean-k-mooney | stephenfin: yep you cant use over commit with numa properly | |
| 17:36:11 | sean-k-mooney | stephenfin: claudiub look at https://etherpad.opendev.org/p/nova-wallaby-ptg line 712 | |
| 17:36:49 | sean-k-mooney | i explained why this is needed | |
| 17:37:18 | stephenfin | that's a veritable hammer of a solution | |
| 17:37:26 | stephenfin | we could do proper tracking without that | |
| 17:37:32 | claudiub | ouch. didn't know about that | |
| 17:37:44 | sean-k-mooney | stephenfin: we cant we tried many times | |
| 17:38:38 | sean-k-mooney | stephenfin: the only time oversubsript can be used with numa guest is if hw:numa_nodes = the number of numa nodes on the host | |
| 17:38:39 | stephenfin | I appreciate it's complicated because we need to marry the non-NUMA model with the NUMA model | |
| 17:39:01 | stephenfin | that's just because we don't do proper oversubscription modelling on a per-node basis | |
| 17:39:24 | sean-k-mooney | we do proper page tracking on a per numa basis | |
| 17:39:37 | sean-k-mooney | and that woudl be used if we did hw:mem_page_size=small | |
| 17:39:46 | sean-k-mooney | that could support oversubsripion if we wanted | |
| 17:39:55 | claudiub | from what I've seen in the docs, it wasn't specifying that huge pages extra_spec is mandatory, so that could be easily missed, imo. | |
| 17:39:59 | stephenfin | right, and we could use that with overcommit ratio for non-pinned NUMA instances | |
| 17:40:04 | sean-k-mooney | but we need to turn that on which si what hw:mem_page_size does | |
| 17:40:14 | stephenfin | that's the gap | |
| 17:40:27 | stephenfin | that we don't consider that information for instances without explicit page size configuration | |
| 17:40:28 | sean-k-mooney | yep but we were told we can make all guest numa guests | |
| 17:40:42 | sean-k-mooney | this only works if the guest cant float across numa ndoes | |
| 17:40:58 | stephenfin | which they can't if they have a guest NUMA topology | |
| 17:41:18 | sean-k-mooney | stephenfin: yep | |
| 17:41:26 | stephenfin | <stephenfin> I appreciate it's complicated because we need to marry the non-NUMA model with the NUMA model | |
| 17:41:30 | sean-k-mooney | if we can make all guest numa guest in nova we can fix this | |
| 17:41:31 | stephenfin | ^ that's the complicated bit | |
| 17:41:44 | stephenfin | because we can't make all guests NUMA guests, as you say | |
| 17:41:53 | sean-k-mooney | yep | |
| 17:41:59 | stephenfin | so we need to somehow have two views into memory - one with NUMA context and another without it | |
| 17:42:19 | sean-k-mooney | so making all numa guest have hw:mem_page_size=any we trun on the memroy tracking | |
| 17:42:29 | stephenfin | and somehow still be able to get a useful "how much free memory do I have" answer for both NUMA and non-NUMA guests | |
| 17:42:31 | sean-k-mooney | it will use smallpages unless the image asks | |
| 17:42:46 | sean-k-mooney | we can then extend that to do over subsiption if we want | |
| 17:42:49 | stephenfin | that's doable though | |
| 17:43:33 | sean-k-mooney | claudiub: anyway sorry for the info overload | |
| 17:43:50 | claudiub | nono, thanks, this is useful. :) | |
| 17:43:56 | sean-k-mooney | claudiub: just being trying to fix this for 6+ years | |
| 17:43:59 | stephenfin | for example, in the NUMA case, free memory for NUMA N could be seen as (NUMA N total memory * overcommit) - (NUMA N used memory) - (non-NUMA used memory / NUMA count) | |
| 17:44:16 | stephenfin | i.e. just evenly divide the non-NUMA memory usage across all NUMA nodes | |
| 17:44:41 | sean-k-mooney | we cant do that | |
| 17:44:42 | stephenfin | my point being, there are other ways to solve this that don't involve the hw:mem_page_size=any hammer :) | |
| 17:45:03 | sean-k-mooney | that will cause oom issue | |
| 17:45:28 | sean-k-mooney | when a vm has a numa toplogy we tell teh kernel and restict its meory allcoations and the cores it can float over to the numa node | |
| 17:45:54 | stephenfin | by the memory isn't locked | |
| 17:45:58 | stephenfin | so it can be swapped out | |
| 17:46:07 | sean-k-mooney | it can be swapped yes | |
| 17:46:14 | sean-k-mooney | in the case of 4k pages at least | |
| 17:46:20 | stephenfin | and the memory for non-NUMA hosts will move across various NUMA nodes as needed | |
| 17:46:22 | sean-k-mooney | hugepages are not swapabel | |
| 17:46:36 | stephenfin | yeah, I'm only focusing on small pages for now | |
| 17:47:05 | sean-k-mooney | stephenfin: so there are 3 times we shoudl do. 1 numa blanceing, 2 hw:mem_page_size=any 3, make smallpages oversubscibale per numa node | |
| 17:47:15 | sean-k-mooney | well do 2 after 3 | |
| 17:47:39 | stephenfin | I'm still not sure why 2 is needed, but agreed on the other two | |
| 17:47:51 | sean-k-mooney | we do to turn on the memroy tracking | |
| 17:47:55 | stephenfin | I'll try to take a look at this the week after next | |
| 17:48:04 | sean-k-mooney | we dont do the claim otherwise in the hsot numna toplogy object | |
| 17:48:05 | stephenfin | since despite all this talk, I'm not going to start on it today :) | |
| 17:49:14 | sean-k-mooney | ok either way i think we need to disucss this in the ptg again and we can talk about it on irc again before | |
| 17:49:19 | stephenfin | agreed | |
| 17:49:31 | stephenfin | wanna add it to the agenda if you haven't already? | |
| 17:49:35 | stephenfin | if not I can | |
| 17:49:59 | sean-k-mooney | i didnt readded it since i was going to proceed based on what we agreed last ptg | |
| 17:50:13 | sean-k-mooney | i just didnt get time to wrok on this this cycle because of vdpa | |
| 17:50:24 | claudiub | I'll try the hw:numa_pages thing as soon as possible, since it's a bit of a burning issue for us. :) | |
| 17:51:22 | sean-k-mooney | claudiub: setting it to hw:mem_page_size=any or hw:mem_page_size=small should resolve your current issue | |
| 17:51:38 | sean-k-mooney | at the cost of oversubsciption of memory beign blocked | |
| 17:51:44 | lyarwood | stephenfin: do you have a link to you libvirt secure boot bug to hand? | |
| 17:52:01 | lyarwood | your* | |
| 17:52:13 | stephenfin | lyarwood: https://bugzilla.redhat.com/show_bug.cgi?id=1929357 | |
| 17:52:14 | openstack | bugzilla.redhat.com bug 1929357 in libvirt "UEFI: Provide a way how to configure different combinations of secure boot enabled/disabled and keys enrolled/not enrolled" [Medium,New] - Assigned to phrdina | |
| 17:52:25 | lyarwood | thanks | |
| 17:52:44 | sean-k-mooney | claudiub: it that is not an option for you and you can carry a downstream only patch you would add a random shuffle here https://github.com/openstack/nova/blob/20459e3e88cb8382d450c7fdb042e2016d5560c5/nova/virt/hardware.py#L2276 of host_cells | |
| 17:54:41 | claudiub | will try both. :) | |
| 17:56:42 | sean-k-mooney | the balancing without over subctiption or randomisation with it likely could be backproted as a workaround bugfix. with proper feature in the future | |
| 18:24:45 | sean-k-mooney | stephenfin:related to the previous topic it look like libvirt is locking the guest memroy any time a guest has vfio(pci passthough/sriov), mdev or nvme devices assigned nova was previously not aware of that and them means many guests we previously assumed were swapable are not. i still have to file a bug for this but its emrging forlowing the memory locking disucssion we had for vdpa | |
| 18:26:29 | sean-k-mooney | im not sure how to fix that yet without requiring all guest with any kind of passthoug device to be numa guests or otherewise use hw:mem_page_size in some form | |
| 18:27:13 | sean-k-mooney | we can mitagate the problem for q35 guests by enabling the viommu | |
| 18:27:36 | sean-k-mooney | but i suspect this is the cause of many OOM bug that have been filed in the past | |
| 18:29:54 | sean-k-mooney | oh and more fun file backed memroy does not seam to work the way we tought either | |
| 18:30:36 | sean-k-mooney | i should do some more testing but usign it i was not actully able to allocate more vms then i had memory for without OOM issues killing the running vms | |
| 18:31:08 | sean-k-mooney | so it looks like instead of mmaping the guest memory form the files as the qemu/libvirt docs impleis | |
| 18:31:39 | sean-k-mooney | qemu just malloc the memroy normally and then also create a mapping of the memory to a file | |