Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-03-04
16:38:21 lyarwood stephenfin: https://review.opendev.org/c/openstack/nova/+/769548 - just a reminder if you didn't have this on your list, that would then move the series into the gate.
16:38:38 claudiub Hello! I've noticed that in the NUMA-related docs (https://docs.openstack.org/nova/latest/admin/cpu-topologies.html#customizing-instance-numa-placement-policies) it says that "The NUMA node(s) used are normally chosen at random",
16:38:40 lyarwood oh and https://review.opendev.org/c/openstack/nova/+/778462/2 that gibi++ added in
16:38:42 claudiub but numa_fit_instance_to_host (https://github.com/openstack/nova/blob/master/nova/virt/hardware.py#L2235) says that it will return a new InstanceNUMATopology with its cell ids set to host cell ids of the first successful permutation, or None.
16:39:26 claudiub so, there's a mismatch there. From what I've seen, all the instances end up in the 1st NUMA node.
16:39:52 stephenfin lyarwood: was in the middle of reviewing. +2 on the whole series now
16:39:57 claudiub that could be problematic. Basically, I can have nodes with 50% consumed resources, but 1 NUMA node completely empty.
16:40:10 stephenfin claudiub: Yup, that's a known bug :(
16:40:21 claudiub so, I'm wondering which should be corrected: The docs, or the code.
16:40:29 stephenfin I think sean-k-mooney filed a bug for same
16:40:34 stephenfin claudiub: the code
16:40:40 claudiub ah, gotcha. :)
16:41:05 claudiub Wondering what the backport potential for this would be, if a fix would be added. :)
16:41:07 stephenfin we should shuffle the nodes or sort by least-allocated node
16:41:23 stephenfin depends on the implementation of course but definitely backportable IMO
16:41:30 stephenfin *should definitely be
16:42:08 claudiub is someone working on this? I could look into it if no one is
16:42:52 stephenfin Not right now. If you could look, I'd be happy to review
16:42:56 openstackgerrit Artom Lifshitz proposed openstack/nova master: pci: track host NUMA topology in stats https://review.opendev.org/c/openstack/nova/+/774149
16:42:58 openstackgerrit Artom Lifshitz proposed openstack/nova master: pci: implement the 'socket' NUMA affinity policy https://review.opendev.org/c/openstack/nova/+/772779
16:43:00 openstackgerrit Artom Lifshitz proposed openstack/nova master: pci: always pass node_id to manager https://review.opendev.org/c/openstack/nova/+/778747
16:43:08 lyarwood stephenfin: awesome thanks for that
16:43:25 openstackgerrit Kashyap Chamarthy proposed openstack/nova master: libvirt: Allow disabling CPU flags via `cpu_model_extra_flags` https://review.opendev.org/c/openstack/nova/+/774240
16:43:34 claudiub great, will let you know how it goes. :)
16:44:05 kashyap stephenfin: gibi (lost your +2, if you'd like to add the stamp again) --^ Addressed the doc nits from Stephen
16:44:13 gibi sure
16:45:25 bauzas sean-k-mooney: could you please confirm that the os-vif SHA1 for the next release looks good to you ?
16:45:30 bauzas sean-k-mooney: https://review.opendev.org/c/openstack/releases/+/777955/1/deliverables/wallaby/os-vif.yaml
16:45:56 bauzas from what I see, yup
16:46:38 sean-k-mooney looking
16:49:19 sean-k-mooney yep its the current head of master and the main delta is just fixing the lower-constratists-job and a deprecation wraning
16:49:46 sean-k-mooney normlaly i would say that it could be a bugfix release but we always do feature version bump for the end of a cycle
16:49:49 sean-k-mooney so this looks correct
16:50:37 claudiub Also, speaking of NUMA, I was wondering if you know if an instance placed in a single NUMA node can be live-migrated to another node in a different NUMA node, or it has to be in the same NUMA node? Trying to gauge the severity of that bug that places all the instances in the same NUMA node.
16:51:00 sean-k-mooney claudiub: that can be done but only from train
16:51:23 sean-k-mooney claudiub: artom added numa live migration in the train release whre we can regenerate teh xml as part of the migration
16:51:31 claudiub live-migrating instances with numa topologies, right? I've seen that bit in code
16:52:04 sean-k-mooney what bug are you triaging?
16:52:04 artom claudiub, except the user can't specify which NUMA node - which I think is what you're getting at
16:52:10 artom Nova just finds a "free" one
16:52:52 claudiub artom: that's perfect then. :)
16:53:09 sean-k-mooney claudiub: please do not file a new bug for numa blanacing
16:53:41 claudiub sean-k-mooney: I was hitting an issue where all the created instances were created on the same numa node, so I was asking about that. :)
16:53:58 sean-k-mooney claudiub: yep that is by design
16:54:19 sean-k-mooney its somethign we could chagne but when i brough tit up i was told it was a featur not a bug
16:54:57 sean-k-mooney we talked about in the last ptg and when numa was first being added many years ago
16:55:06 claudiub well, one thing's for certain: the docs and the code don't match. :) The docs say the chosen NUMA node is random.
16:55:28 sean-k-mooney from a user perspecitve it is
16:55:41 claudiub Wondering if we could at least have a config option to randomize the selected numa node (ofc, if it can be randomized. not talking about PCI devices)
16:55:43 sean-k-mooney its not actully random form an api point of view it underfied
16:56:13 sean-k-mooney i have a much better solution for this just trying to fine the bug i already had filed
16:56:34 sean-k-mooney https://bugs.launchpad.net/nova/+bug/1893121
16:56:35 openstack Launchpad bug 1893121 in OpenStack Compute (nova) "nova does not balance vm across numa node or prefer numa node with pci device when one is requested" [Undecided,Confirmed] - Assigned to sean mooney (sean-k-mooney)
16:57:23 sean-k-mooney claudiub: this is what determins the order
16:57:25 sean-k-mooney https://github.com/openstack/nova/blob/20459e3e88cb8382d450c7fdb042e2016d5560c5/nova/virt/hardware.py#L2268-L2277
16:57:42 sean-k-mooney as an end user you are not allowed to rely on the detail of that implemenation
16:58:05 sean-k-mooney what i am proposing and had plannd on woking on is doing muliple sorts
16:58:21 sean-k-mooney so that we will blance vm plamcne based on avaiable resouces on a host
16:59:20 sean-k-mooney claudiub: line 693 is the ptg dicussion form the last ptg on the topic https://etherpad.opendev.org/p/nova-wallaby-ptg
17:00:28 sean-k-mooney this is one of the topic i need to bring up internally but i would like to adress this next cycle if i can make time but if you have time to work on it then that woudl be good too.
17:01:25 sean-k-mooney numa blancing is the non invaisive way to optimise better then we do today.
17:02:01 sean-k-mooney long term we should be doing this more abstractly. e.g. computeing a cost metic for any give plamcnet based on a number of factors and then minimsiing that.
17:02:22 sean-k-mooney kind of like how the wehers work but placment complictates that.
17:03:52 sean-k-mooney claudiub: the workaround for now is to follow the advice we always gave. try to create flavors that aproximate the host toplogy. e.g. if you hosts all have 2 numa nodes then default to createign flavors with hw:numa_nodes=2
17:04:40 openstackgerrit Balazs Gibizer proposed openstack/nova master: Replace blind retry with libvirt event waiting in detach https://review.opendev.org/c/openstack/nova/+/770246
17:08:57 claudiub sean-k-mooney: Hmm, I see. but correct me if I'm wrong, but in that code snippet, the host cells are not randomized or sorted in any other way if there are no pci_requests and no pci_stats. It's the same host_cells athat are set in host_topology.cells (not sure if the order ever changes here). I could spend some time on it, will read the PTG notes as well.
17:09:55 sean-k-mooney if pci devices are not requested we sort the numa nodes to prefer the onces without numa nodes
17:09:56 claudiub But in any case, setting hw_numa_nodes=2 doesn't help in our scenario, especially since we also have nodes with just 1 NUMA node. :) Additionally, setting the instances on 2 numa nodes could affect the guest performance as well.
17:10:26 claudiub sean-k-mooney: indeed, agreed there.
17:10:27 sean-k-mooney otherwise itertools.permutations iterates over them in a determisict maner
17:10:51 sean-k-mooney which for singel numa nodes instace is acending order form numa node 0
17:11:04 sean-k-mooney then we bail out if it fits and dont try any other nodes
17:11:10 sean-k-mooney so that packs numa0
17:11:14 claudiub yep
17:11:39 sean-k-mooney this is has alwasy been the case since numa was firsts added
17:12:01 sean-k-mooney the specific behavior is implemation defiend and is not garenteed by the api
17:12:02 artom stephenfin, D: how did you not jump on https://review.opendev.org/c/openstack/nova/+/774240/12/nova/virt/libvirt/driver.py#697 with a -2?
17:12:04 artom ;)
17:12:35 sean-k-mooney so it can be modified but the current beahivor optimises for being able to spawn large vms
17:12:56 sean-k-mooney if we balance the vms between numa ndoes it will improve performance in genreal but pessimise spawning large vms
17:13:11 sean-k-mooney so there is a tradeoff between packeing and spreading/blancing
17:13:17 claudiub i agree there, and I am aware of that. :)
17:13:36 sean-k-mooney yep so this is why we said this cant be change in a bug and need a spec
17:13:44 sean-k-mooney which means upstream at least not backportable
17:14:03 sean-k-mooney downstream we likely would backport it but not chagne the behavior by default
17:14:13 stephenfin artom: My yoga guy said I needed to be more chill about these things
17:14:17 stephenfin I'll fire him in the morning
17:14:20 sean-k-mooney upstream if accpetd i would like to change the default but in the cycle after its added
17:14:43 stephenfin <sean-k-mooney> so it can be modified but the current beahivor optimises for being able to spawn large vms
17:14:51 stephenfin are you talking about unpinned instances?
17:15:01 sean-k-mooney stephenfin: no all numa instances
17:15:16 sean-k-mooney stephenfin: by packing numa nodes before moving on
17:15:22 sean-k-mooney we keep the rest free
17:15:33 sean-k-mooney so vms that need all teh ram or cpus on a numa node can boot
17:15:44 stephenfin ah, yeah I don't think that's a good argument for the unpinned case
17:15:45 sean-k-mooney if we spread then that makes large vms less likely to fit
17:15:56 stephenfin spreading makes sense there IMO
17:15:59 sean-k-mooney unpinned float so ya not an issue
17:16:03 stephenfin yup
17:16:16 claudiub in our scenario, large vms is not a concern, since we have pretty large hosts, so in our case, it would work better for a spread-out approach. But indeed, not everyone is the same. Could this be a config option then?

Earlier   Later