Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-03-04
16:06:30 kashyap stephenfin: Ah, okay; yes, just for the newly-added ones might as well address that
16:10:11 openstackgerrit Claudiu Belu proposed openstack/nova master: POC: tests: Adds test checking unbalanced NUMA node association https://review.opendev.org/c/openstack/nova/+/778740
16:13:20 lyarwood dansmith: sorry joined the meeting late, re the cinder failures, anything like https://bugs.launchpad.net/cinder/+bug/1917750 ?
16:13:23 openstack Launchpad bug 1917750 in Cinder "Running parallel iSCSI/LVM c-vol backends is causing random failures in CI" [Undecided,New]
16:14:12 dansmith lyarwood: depends on how that manifests, but can't say I've seen that specifically
16:14:38 dansmith the two major symptoms I see are a complaint about state conflict, and "unable to delete volume"
16:14:45 lyarwood right sorry, that basically leads to two instances looking at the same volume even when it isn't multiattached
16:15:04 dansmith ack
16:15:23 dansmith (and also.. ouch)
16:15:33 lyarwood that could be related if we are trying to detach the volume in Nova but it's still attached to another instance
16:16:15 dansmith well, I haven't seen detach fails, so much as failure to delete, but I guess it's possible it's just how we report it
16:16:45 lyarwood oh if it's the actual delete on the cinder side then it's likely something else
16:17:03 dansmith yeah I think it's like "delete volume, poll until it's gone...timeout"
16:17:03 lyarwood we've already nuked the connections to the computes at that point
16:17:34 lyarwood kk, melwitt had a bug for lvcreate being slow, assuming it's waiting on lvdelete it could be related
16:17:52 lyarwood lvchange*
16:17:56 lyarwood there's no lvdelete
16:18:06 dansmith ack
16:30:48 kashyap stephenfin: Isn't your "while" spurious, here, on line-8? (The "it. If niether..." bit makes sense, though.) -- https://review.opendev.org/c/openstack/nova/+/774240/11/releasenotes/notes/allow-disabling-cpu-flags-cc861a3bdfffadf8.yaml#8
16:31:52 kashyap It looks like so. I'll disregard it.
16:33:51 stephenfin kashyap: I think it's relevant
16:33:54 stephenfin This is possible via a '+' / '-' notation, where if you specify a CPU flag prefixed with a '+' sign (without quotes), it will be enabled for the guest, a prefix of '-' will disable it
16:34:11 stephenfin This is possible via a '+' / '-' notation, where if you specify a CPU flag prefixed with a '+' sign (without quotes) then it will be enabled for the guest while a prefix of '-' will disable it.
16:34:32 stephenfin The latter reads better to me
16:35:18 kashyap stephenfin: Ah, there; yes. But you're missing a comma after "guest" :)
16:35:44 kashyap Either that, or I've gone comma-wild (I was accused of this once, in a friendly way, on qemu-devel list before :D)
16:36:09 stephenfin Correct, missing comma
16:36:13 stephenfin since it's a comparison
16:36:27 stephenfin *Correct. Missing comma ;)
16:37:04 kashyap stephenfin: I.e. the missing comma in the second version is correct? Yeah?
16:37:18 stephenfin yeah, add the comma
16:37:35 kashyap Ah, nod.
16:38:21 lyarwood stephenfin: https://review.opendev.org/c/openstack/nova/+/769548 - just a reminder if you didn't have this on your list, that would then move the series into the gate.
16:38:38 claudiub Hello! I've noticed that in the NUMA-related docs (https://docs.openstack.org/nova/latest/admin/cpu-topologies.html#customizing-instance-numa-placement-policies) it says that "The NUMA node(s) used are normally chosen at random",
16:38:40 lyarwood oh and https://review.opendev.org/c/openstack/nova/+/778462/2 that gibi++ added in
16:38:42 claudiub but numa_fit_instance_to_host (https://github.com/openstack/nova/blob/master/nova/virt/hardware.py#L2235) says that it will return a new InstanceNUMATopology with its cell ids set to host cell ids of the first successful permutation, or None.
16:39:26 claudiub so, there's a mismatch there. From what I've seen, all the instances end up in the 1st NUMA node.
16:39:52 stephenfin lyarwood: was in the middle of reviewing. +2 on the whole series now
16:39:57 claudiub that could be problematic. Basically, I can have nodes with 50% consumed resources, but 1 NUMA node completely empty.
16:40:10 stephenfin claudiub: Yup, that's a known bug :(
16:40:21 claudiub so, I'm wondering which should be corrected: The docs, or the code.
16:40:29 stephenfin I think sean-k-mooney filed a bug for same
16:40:34 stephenfin claudiub: the code
16:40:40 claudiub ah, gotcha. :)
16:41:05 claudiub Wondering what the backport potential for this would be, if a fix would be added. :)
16:41:07 stephenfin we should shuffle the nodes or sort by least-allocated node
16:41:23 stephenfin depends on the implementation of course but definitely backportable IMO
16:41:30 stephenfin *should definitely be
16:42:08 claudiub is someone working on this? I could look into it if no one is
16:42:52 stephenfin Not right now. If you could look, I'd be happy to review
16:42:56 openstackgerrit Artom Lifshitz proposed openstack/nova master: pci: track host NUMA topology in stats https://review.opendev.org/c/openstack/nova/+/774149
16:42:58 openstackgerrit Artom Lifshitz proposed openstack/nova master: pci: implement the 'socket' NUMA affinity policy https://review.opendev.org/c/openstack/nova/+/772779
16:43:00 openstackgerrit Artom Lifshitz proposed openstack/nova master: pci: always pass node_id to manager https://review.opendev.org/c/openstack/nova/+/778747
16:43:08 lyarwood stephenfin: awesome thanks for that
16:43:25 openstackgerrit Kashyap Chamarthy proposed openstack/nova master: libvirt: Allow disabling CPU flags via `cpu_model_extra_flags` https://review.opendev.org/c/openstack/nova/+/774240
16:43:34 claudiub great, will let you know how it goes. :)
16:44:05 kashyap stephenfin: gibi (lost your +2, if you'd like to add the stamp again) --^ Addressed the doc nits from Stephen
16:44:13 gibi sure
16:45:25 bauzas sean-k-mooney: could you please confirm that the os-vif SHA1 for the next release looks good to you ?
16:45:30 bauzas sean-k-mooney: https://review.opendev.org/c/openstack/releases/+/777955/1/deliverables/wallaby/os-vif.yaml
16:45:56 bauzas from what I see, yup
16:46:38 sean-k-mooney looking
16:49:19 sean-k-mooney yep its the current head of master and the main delta is just fixing the lower-constratists-job and a deprecation wraning
16:49:46 sean-k-mooney normlaly i would say that it could be a bugfix release but we always do feature version bump for the end of a cycle
16:49:49 sean-k-mooney so this looks correct
16:50:37 claudiub Also, speaking of NUMA, I was wondering if you know if an instance placed in a single NUMA node can be live-migrated to another node in a different NUMA node, or it has to be in the same NUMA node? Trying to gauge the severity of that bug that places all the instances in the same NUMA node.
16:51:00 sean-k-mooney claudiub: that can be done but only from train
16:51:23 sean-k-mooney claudiub: artom added numa live migration in the train release whre we can regenerate teh xml as part of the migration
16:51:31 claudiub live-migrating instances with numa topologies, right? I've seen that bit in code
16:52:04 sean-k-mooney what bug are you triaging?
16:52:04 artom claudiub, except the user can't specify which NUMA node - which I think is what you're getting at
16:52:10 artom Nova just finds a "free" one
16:52:52 claudiub artom: that's perfect then. :)
16:53:09 sean-k-mooney claudiub: please do not file a new bug for numa blanacing
16:53:41 claudiub sean-k-mooney: I was hitting an issue where all the created instances were created on the same numa node, so I was asking about that. :)
16:53:58 sean-k-mooney claudiub: yep that is by design
16:54:19 sean-k-mooney its somethign we could chagne but when i brough tit up i was told it was a featur not a bug
16:54:57 sean-k-mooney we talked about in the last ptg and when numa was first being added many years ago
16:55:06 claudiub well, one thing's for certain: the docs and the code don't match. :) The docs say the chosen NUMA node is random.
16:55:28 sean-k-mooney from a user perspecitve it is
16:55:41 claudiub Wondering if we could at least have a config option to randomize the selected numa node (ofc, if it can be randomized. not talking about PCI devices)
16:55:43 sean-k-mooney its not actully random form an api point of view it underfied
16:56:13 sean-k-mooney i have a much better solution for this just trying to fine the bug i already had filed
16:56:34 sean-k-mooney https://bugs.launchpad.net/nova/+bug/1893121
16:56:35 openstack Launchpad bug 1893121 in OpenStack Compute (nova) "nova does not balance vm across numa node or prefer numa node with pci device when one is requested" [Undecided,Confirmed] - Assigned to sean mooney (sean-k-mooney)
16:57:23 sean-k-mooney claudiub: this is what determins the order
16:57:25 sean-k-mooney https://github.com/openstack/nova/blob/20459e3e88cb8382d450c7fdb042e2016d5560c5/nova/virt/hardware.py#L2268-L2277
16:57:42 sean-k-mooney as an end user you are not allowed to rely on the detail of that implemenation
16:58:05 sean-k-mooney what i am proposing and had plannd on woking on is doing muliple sorts
16:58:21 sean-k-mooney so that we will blance vm plamcne based on avaiable resouces on a host
16:59:20 sean-k-mooney claudiub: line 693 is the ptg dicussion form the last ptg on the topic https://etherpad.opendev.org/p/nova-wallaby-ptg
17:00:28 sean-k-mooney this is one of the topic i need to bring up internally but i would like to adress this next cycle if i can make time but if you have time to work on it then that woudl be good too.
17:01:25 sean-k-mooney numa blancing is the non invaisive way to optimise better then we do today.
17:02:01 sean-k-mooney long term we should be doing this more abstractly. e.g. computeing a cost metic for any give plamcnet based on a number of factors and then minimsiing that.
17:02:22 sean-k-mooney kind of like how the wehers work but placment complictates that.
17:03:52 sean-k-mooney claudiub: the workaround for now is to follow the advice we always gave. try to create flavors that aproximate the host toplogy. e.g. if you hosts all have 2 numa nodes then default to createign flavors with hw:numa_nodes=2
17:04:40 openstackgerrit Balazs Gibizer proposed openstack/nova master: Replace blind retry with libvirt event waiting in detach https://review.opendev.org/c/openstack/nova/+/770246
17:08:57 claudiub sean-k-mooney: Hmm, I see. but correct me if I'm wrong, but in that code snippet, the host cells are not randomized or sorted in any other way if there are no pci_requests and no pci_stats. It's the same host_cells athat are set in host_topology.cells (not sure if the order ever changes here). I could spend some time on it, will read the PTG notes as well.
17:09:55 sean-k-mooney if pci devices are not requested we sort the numa nodes to prefer the onces without numa nodes
17:09:56 claudiub But in any case, setting hw_numa_nodes=2 doesn't help in our scenario, especially since we also have nodes with just 1 NUMA node. :) Additionally, setting the instances on 2 numa nodes could affect the guest performance as well.

Earlier   Later