Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-03-04
17:34:10 sean-k-mooney if you set hw:numa_nodes=1 and dont set hw:mem_page_size to any valid value its not correct
17:34:41 stephenfin that's a bit strong. It's fine if you don't have hugepages on the host
17:34:51 stephenfin or other things taking memory that you haven't accounted for
17:34:53 sean-k-mooney stephenfin: i did not say use hugepages
17:35:06 stephenfin setting hw:mem_page_size disables memory overcommit
17:35:08 sean-k-mooney stephenfin: i said set hw:mem_page_size to any value
17:35:26 sean-k-mooney stephenfin: yep you cant use over commit with numa properly
17:36:11 sean-k-mooney stephenfin: claudiub look at https://etherpad.opendev.org/p/nova-wallaby-ptg line 712
17:36:49 sean-k-mooney i explained why this is needed
17:37:18 stephenfin that's a veritable hammer of a solution
17:37:26 stephenfin we could do proper tracking without that
17:37:32 claudiub ouch. didn't know about that
17:37:44 sean-k-mooney stephenfin: we cant we tried many times
17:38:38 sean-k-mooney stephenfin: the only time oversubsript can be used with numa guest is if hw:numa_nodes = the number of numa nodes on the host
17:38:39 stephenfin I appreciate it's complicated because we need to marry the non-NUMA model with the NUMA model
17:39:01 stephenfin that's just because we don't do proper oversubscription modelling on a per-node basis
17:39:24 sean-k-mooney we do proper page tracking on a per numa basis
17:39:37 sean-k-mooney and that woudl be used if we did hw:mem_page_size=small
17:39:46 sean-k-mooney that could support oversubsripion if we wanted
17:39:55 claudiub from what I've seen in the docs, it wasn't specifying that huge pages extra_spec is mandatory, so that could be easily missed, imo.
17:39:59 stephenfin right, and we could use that with overcommit ratio for non-pinned NUMA instances
17:40:04 sean-k-mooney but we need to turn that on which si what hw:mem_page_size does
17:40:14 stephenfin that's the gap
17:40:27 stephenfin that we don't consider that information for instances without explicit page size configuration
17:40:28 sean-k-mooney yep but we were told we can make all guest numa guests
17:40:42 sean-k-mooney this only works if the guest cant float across numa ndoes
17:40:58 stephenfin which they can't if they have a guest NUMA topology
17:41:18 sean-k-mooney stephenfin: yep
17:41:26 stephenfin <stephenfin> I appreciate it's complicated because we need to marry the non-NUMA model with the NUMA model
17:41:30 sean-k-mooney if we can make all guest numa guest in nova we can fix this
17:41:31 stephenfin ^ that's the complicated bit
17:41:44 stephenfin because we can't make all guests NUMA guests, as you say
17:41:53 sean-k-mooney yep
17:41:59 stephenfin so we need to somehow have two views into memory - one with NUMA context and another without it
17:42:19 sean-k-mooney so making all numa guest have hw:mem_page_size=any we trun on the memroy tracking
17:42:29 stephenfin and somehow still be able to get a useful "how much free memory do I have" answer for both NUMA and non-NUMA guests
17:42:31 sean-k-mooney it will use smallpages unless the image asks
17:42:46 sean-k-mooney we can then extend that to do over subsiption if we want
17:42:49 stephenfin that's doable though
17:43:33 sean-k-mooney claudiub: anyway sorry for the info overload
17:43:50 claudiub nono, thanks, this is useful. :)
17:43:56 sean-k-mooney claudiub: just being trying to fix this for 6+ years
17:43:59 stephenfin for example, in the NUMA case, free memory for NUMA N could be seen as (NUMA N total memory * overcommit) - (NUMA N used memory) - (non-NUMA used memory / NUMA count)
17:44:16 stephenfin i.e. just evenly divide the non-NUMA memory usage across all NUMA nodes
17:44:41 sean-k-mooney we cant do that
17:44:42 stephenfin my point being, there are other ways to solve this that don't involve the hw:mem_page_size=any hammer :)
17:45:03 sean-k-mooney that will cause oom issue
17:45:28 sean-k-mooney when a vm has a numa toplogy we tell teh kernel and restict its meory allcoations and the cores it can float over to the numa node
17:45:54 stephenfin by the memory isn't locked
17:45:58 stephenfin so it can be swapped out
17:46:07 sean-k-mooney it can be swapped yes
17:46:14 sean-k-mooney in the case of 4k pages at least
17:46:20 stephenfin and the memory for non-NUMA hosts will move across various NUMA nodes as needed
17:46:22 sean-k-mooney hugepages are not swapabel
17:46:36 stephenfin yeah, I'm only focusing on small pages for now
17:47:05 sean-k-mooney stephenfin: so there are 3 times we shoudl do. 1 numa blanceing, 2 hw:mem_page_size=any 3, make smallpages oversubscibale per numa node
17:47:15 sean-k-mooney well do 2 after 3
17:47:39 stephenfin I'm still not sure why 2 is needed, but agreed on the other two
17:47:51 sean-k-mooney we do to turn on the memroy tracking
17:47:55 stephenfin I'll try to take a look at this the week after next
17:48:04 sean-k-mooney we dont do the claim otherwise in the hsot numna toplogy object
17:48:05 stephenfin since despite all this talk, I'm not going to start on it today :)
17:49:14 sean-k-mooney ok either way i think we need to disucss this in the ptg again and we can talk about it on irc again before
17:49:19 stephenfin agreed
17:49:31 stephenfin wanna add it to the agenda if you haven't already?
17:49:35 stephenfin if not I can
17:49:59 sean-k-mooney i didnt readded it since i was going to proceed based on what we agreed last ptg
17:50:13 sean-k-mooney i just didnt get time to wrok on this this cycle because of vdpa
17:50:24 claudiub I'll try the hw:numa_pages thing as soon as possible, since it's a bit of a burning issue for us. :)
17:51:22 sean-k-mooney claudiub: setting it to hw:mem_page_size=any or hw:mem_page_size=small should resolve your current issue
17:51:38 sean-k-mooney at the cost of oversubsciption of memory beign blocked
17:51:44 lyarwood stephenfin: do you have a link to you libvirt secure boot bug to hand?
17:52:01 lyarwood your*
17:52:13 stephenfin lyarwood: https://bugzilla.redhat.com/show_bug.cgi?id=1929357
17:52:14 openstack bugzilla.redhat.com bug 1929357 in libvirt "UEFI: Provide a way how to configure different combinations of secure boot enabled/disabled and keys enrolled/not enrolled" [Medium,New] - Assigned to phrdina
17:52:25 lyarwood thanks
17:52:44 sean-k-mooney claudiub: it that is not an option for you and you can carry a downstream only patch you would add a random shuffle here https://github.com/openstack/nova/blob/20459e3e88cb8382d450c7fdb042e2016d5560c5/nova/virt/hardware.py#L2276 of host_cells
17:54:41 claudiub will try both. :)
17:56:42 sean-k-mooney the balancing without over subctiption or randomisation with it likely could be backproted as a workaround bugfix. with proper feature in the future
18:24:45 sean-k-mooney stephenfin:related to the previous topic it look like libvirt is locking the guest memroy any time a guest has vfio(pci passthough/sriov), mdev or nvme devices assigned nova was previously not aware of that and them means many guests we previously assumed were swapable are not. i still have to file a bug for this but its emrging forlowing the memory locking disucssion we had for vdpa
18:26:29 sean-k-mooney im not sure how to fix that yet without requiring all guest with any kind of passthoug device to be numa guests or otherewise use hw:mem_page_size in some form
18:27:13 sean-k-mooney we can mitagate the problem for q35 guests by enabling the viommu
18:27:36 sean-k-mooney but i suspect this is the cause of many OOM bug that have been filed in the past
18:29:54 sean-k-mooney oh and more fun file backed memroy does not seam to work the way we tought either
18:30:36 sean-k-mooney i should do some more testing but usign it i was not actully able to allocate more vms then i had memory for without OOM issues killing the running vms
18:31:08 sean-k-mooney so it looks like instead of mmaping the guest memory form the files as the qemu/libvirt docs impleis
18:31:39 sean-k-mooney qemu just malloc the memroy normally and then also create a mapping of the memory to a file
18:32:22 sean-k-mooney that might be because we are usign the legacy api for this or it might be for a different reason but either way it makes me sad :(
18:49:25 sean-k-mooney actully i think i know why its broken if we want vms to only have file backed memory which is what we inteded we need to explitly set the normal memory to 0 i belive and then add the file using the memory hotplug feature so ya it likely a qemu bug in the old api.
19:43:05 openstackgerrit Merged openstack/nova master: libvirt: Parse the 'os' element from domainCapabilities https://review.opendev.org/c/openstack/nova/+/673790
20:17:26 openstackgerrit Merged openstack/nova master: nova-manage: Add libvirt get_machine_type command https://review.opendev.org/c/openstack/nova/+/769548
23:03:14 melwitt lyarwood: could you pls take a look at these stable/victoria backports? they've disabled tests in tripleo ci to workaround intermittent failures due to the bug https://review.opendev.org/c/openstack/nova/+/777121 and https://review.opendev.org/c/openstack/nova/+/777209
23:22:20 openstackgerrit Takashi Kajinami proposed openstack/nova master: WIP: Clean up allocations left by evacuation https://review.opendev.org/c/openstack/nova/+/778696
23:51:50 openstackgerrit melanie witt proposed openstack/nova master: Add functional test for bug 1837995 https://review.opendev.org/c/openstack/nova/+/775449
23:51:52 openstackgerrit melanie witt proposed openstack/nova master: Dynamically archive FK related records in archive_deleted_rows https://review.opendev.org/c/openstack/nova/+/773834
23:51:52 openstack bug 1837995 in OpenStack Compute (nova) ""Unexpected API Error" when use "openstack usage show" command" [Undecided,In progress] https://launchpad.net/bugs/1837995 - Assigned to melanie witt (melwitt)
#openstack-nova - 2021-03-05
00:06:21 openstackgerrit melanie witt proposed openstack/nova master: Add functional regression test for bug 1853009 https://review.opendev.org/c/openstack/nova/+/695012
00:06:23 openstackgerrit melanie witt proposed openstack/nova master: Invalidate provider tree when compute node disappears https://review.opendev.org/c/openstack/nova/+/695188
00:06:23 openstackgerrit melanie witt proposed openstack/nova master: Clear rebalanced compute nodes from resource tracker https://review.opendev.org/c/openstack/nova/+/695187
00:06:23 openstack bug 1853009 in OpenStack Compute (nova) ussuri "Ironic node rebalance race can lead to missing compute nodes in DB" [High,In progress] https://launchpad.net/bugs/1853009 - Assigned to Mark Goddard (mgoddard)

Earlier   Later