| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-02-06 | |||
| 22:47:02 | efried | cfriesen: What does that mean? Multi-numa-node guests? You mean guests that don't care about NUMA affinity? Or do they explicitly care about *anti* affinity? | |
| 22:47:05 | sean-k-mooney | cfriesen: multi numa guest do not need to be on multiple host numa nodes | |
| 22:47:14 | cfriesen | efried: no, I mean guests with multi numa nodes | |
| 22:47:22 | sean-k-mooney | cfriesen: i consider that limitation in libvirt to be a bug not a feature | |
| 22:47:30 | sean-k-mooney | libvirt virt driver that is | |
| 22:47:45 | cfriesen | sean-k-mooney: one reason why you might "require" multi-numa-node guests is to get access to more memory bandwidth | |
| 22:48:48 | sean-k-mooney | cfriesen: yes but there is a flavour extra spec defined for that that only the hyperv dirver support | |
| 22:49:00 | sean-k-mooney | cfriesen: i think the libvirt one should too | |
| 22:49:17 | cfriesen | sean-k-mooney: sorry, don't follow. | |
| 22:50:09 | sean-k-mooney | cfriesen: hw:numa_mempolicy | |
| 22:50:13 | cfriesen | sean-k-mooney: I can currently specify a multi-numa-node guest with libvirt, and I get access to that many host numa nodes, with accordingly more host memory bandwidth. (presumably combined with dedicated cpus) | |
| 22:50:59 | cfriesen | numa_mempolicy isn't mentioned anywhere in nova | |
| 22:51:41 | sean-k-mooney | cfriesen: i used to be in the hyperv driver | |
| 22:52:06 | sean-k-mooney | anyway that is off topic | |
| 22:52:25 | sean-k-mooney | efried: a multi numa gues is a guest with multiple virtual numa nodes | |
| 22:52:55 | cfriesen | efried: and currently they map to different physical numa nodes of the same host. | |
| 22:53:03 | sean-k-mooney | efried: you can choose what virtual cores are associated to each virtual numa node and you can device the memoy between each virtual numa node | |
| 22:54:09 | efried | Well, if you wanted to get that specific, you could probably swang something with granular request groups. | |
| 22:54:29 | efried | But we still wouldn't be able to guarantee you anti-affinity. | |
| 22:54:49 | cfriesen | if you can't guarantee numa-antiaffinity it'll change the current behaviour | |
| 22:55:02 | cfriesen | (ie if both virtual numa nodes could end up on the same host numa node) | |
| 22:55:06 | sean-k-mooney | placement does not need to give that guarentee | |
| 22:55:14 | sean-k-mooney | the virt driver can via the resouce tracker | |
| 22:55:40 | efried | sean-k-mooney: Yeahhhh, but we eventually *should* be able to implement that filter in placement. | |
| 22:55:44 | sean-k-mooney | or the nova numa topology filter | |
| 22:56:09 | sean-k-mooney | efried: talk to jay about that. in principal i agree | |
| 22:57:23 | sean-k-mooney | i would like placement to fully handel numa. but if it can handel 80% of numa in rocky with the rest in the filetr i would be happy with that too | |
| 22:57:46 | efried | Okay. I'll try to put up that spec later this week. | |
| 22:59:43 | sean-k-mooney | cfriesen: im sure you know this also but just requesting guest numa nodes was never ment to give any form of cpu affinity either. from an api perspcetive it does not but for libvir it pince the cores of that numa node to float over a host numa node | |
| 23:00:44 | sean-k-mooney | cfriesen: similay requsting hugepage was not ment to create a numa topology for the guest and also restice the floating cores to a host numa node | |
| 23:01:17 | sean-k-mooney | cfriesen: all of those implit behaviors are internal implentation behaviors of the libvirt driver | |
| 23:01:50 | cfriesen | sean-k-mooney: the hugepage one is a qemu limitation though since you can only point at a single backing file for memory | |
| 23:01:57 | cfriesen | (on a given guest numa node) | |
| 23:03:04 | sean-k-mooney | cfriesen: yes and no you numa node number is optionaly and numactl will be used to determin the optimal numa node to use of you dont specify one | |
| 23:04:02 | cfriesen | sean-k-mooney: how do you tell qemu to use hugepages without specifying a backing file? | |
| 23:05:05 | sean-k-mooney | cfriesen: you do specify a backing file but the backing file itesf does not have numa affintiy | |
| 23:06:02 | sean-k-mooney | cfriesen: when you create a hugepage backing file under /dev/hugepages/libvrt/qemu the numa node is not part of the path | |
| 23:06:45 | sean-k-mooney | cfriesen: checkout https://libvirt.org/formatdomain.html#elementsMemoryBacking the note on the nodeset being optional | |
| 23:10:03 | sean-k-mooney | cfriesen: tacking hugepages per numa node made the resouce track code simpeler which is why it was doen per numa node originally | |
| 23:17:30 | cfriesen | sean-k-mooney: ah, right. I was thinking that mapping hugetlbfs was numa-affined, but it's not necessarily. | |
| 23:18:04 | sean-k-mooney | cfriesen: you might be able to do somthing via the mount options but by defualt no | |
| 23:19:19 | sean-k-mooney | efried: actully speaking of memory and numa we should really be tracking 4k pages e.g. non hugepage memory per numa node too | |
| 23:19:46 | sean-k-mooney | the kernel out of memory killer is triggered per numa node not gloably | |
| 23:20:06 | sean-k-mooney | so the reserved memory config option does not work properly on a numa system | |
| 23:20:53 | sean-k-mooney | vms can still be killed if a numa node has no memoy left and the kernel need to allocate some on that node but the host othersise has free ram | |
| 23:21:33 | sean-k-mooney | our just make all your vms use hugepages that works too | |
| 23:25:06 | cfriesen | that's what we do by default (2M pages) | |
| 23:26:36 | sean-k-mooney | cfriesen: ya we spent 6 months trying to track down what was causeing OOM to kill our vms in the the intel nfv ci and it turned out to be this. hugepages fixed it | |
| 23:27:31 | sean-k-mooney | the anoying thing was it took about 2-3 weeks to start showing up as it need the memory to become fragmented enough to force it | |
| 23:42:38 | openstackgerrit | Patricia Domingues proposed openstack/nova master: load up the volume drivers by checking architecture https://review.openstack.org/541393 | |
| 23:44:22 | openstackgerrit | Patricia Domingues proposed openstack/nova master: load up the volume drivers by checking architecture https://review.openstack.org/541393 | |
| 23:58:46 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add a prelude release note for the 17.0.0 Queens GA https://review.openstack.org/541495 | |
| #openstack-nova - 2018-02-07 | |||
| 00:22:28 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Fix pike GA prelude release note https://review.openstack.org/541498 | |
| 00:44:23 | openstackgerrit | Merged openstack/nova master: docs: Add booting from an encrypted volume https://review.openstack.org/540506 | |
| 01:06:42 | openstackgerrit | Arvind proposed openstack/nova-specs master: BP Support traits in Glance https://review.openstack.org/541502 | |
| 01:51:06 | openstackgerrit | Arvind proposed openstack/nova-specs master: Support traits in Glance https://review.openstack.org/541507 | |
| 02:19:16 | openstackgerrit | Merged openstack/nova master: libvirt: fix native luks encryption failure to find volume_id https://review.openstack.org/539739 | |
| 02:19:28 | mriedem | woo finally | |
| 02:19:32 | openstackgerrit | Merged openstack/nova master: Remove instance.keypairs migration code https://review.openstack.org/517158 | |
| 02:19:44 | openstackgerrit | Merged openstack/nova master: Fix the format file name https://review.openstack.org/523010 | |
| 02:19:54 | openstackgerrit | Merged openstack/nova master: Add regression test for bug 1735407 https://review.openstack.org/526095 | |
| 02:19:55 | openstack | bug 1735407 in OpenStack Compute (nova) "[Nova] Evacuation doesn't respect anti-affinity rules" [Medium,In progress] https://launchpad.net/bugs/1735407 - Assigned to Balazs Gibizer (balazs-gibizer) | |
| 02:20:57 | openstackgerrit | Nakanishi Tomotaka proposed openstack/nova master: Test Compute API in multiple cells https://review.openstack.org/534116 | |
| 02:39:36 | openstackgerrit | chenpengzi proposed openstack/nova master: update docstring param discreption https://review.openstack.org/541519 | |
| 02:56:38 | openstackgerrit | Tetsuro Nakamura proposed openstack/nova master: doc: merge numa.rst to cpu-topologies.rst https://review.openstack.org/540585 | |
| 03:01:34 | openstackgerrit | chenpengzi proposed openstack/nova master: update docstring param description https://review.openstack.org/541519 | |
| 03:45:37 | openstackgerrit | Arvind proposed openstack/nova-specs master: Support traits in Glance https://review.openstack.org/541535 | |
| 03:46:36 | openstackgerrit | Merged openstack/nova master: Add late server group policy check to rebuild https://review.openstack.org/525242 | |
| 03:46:47 | openstackgerrit | Merged openstack/nova master: doc: mark the max microversions for queens https://review.openstack.org/539978 | |
| 03:48:00 | openstackgerrit | Tetsuro Nakamura proposed openstack/nova master: Not use thread alloc policy for emulator thread https://review.openstack.org/538700 | |
| 03:51:09 | openstackgerrit | Arvind proposed openstack/nova-specs master: Support traits in Glance https://review.openstack.org/541507 | |
| 05:07:00 | openstackgerrit | Merged openstack/nova master: Make sure that we have usable input for graphical console https://review.openstack.org/538003 | |
| 05:07:36 | openstackgerrit | Merged openstack/nova master: Fix wrong link for "Manage Flavors" in CPU topologies doc https://review.openstack.org/541116 | |
| 05:17:24 | openstackgerrit | Arvind Nadendla proposed openstack/nova-specs master: Support traits in Glance https://review.openstack.org/541507 | |
| 05:29:01 | openstackgerrit | Merged openstack/nova master: Reset the _RC_CACHE between tests https://review.openstack.org/540404 | |
| 05:29:20 | openstackgerrit | Merged openstack/nova master: Clarify CONF.scheduler.max_attempts https://review.openstack.org/541031 | |
| 05:29:30 | openstackgerrit | Merged openstack/nova master: [placement] Add sending global request ID in put (2) https://review.openstack.org/531266 | |
| 06:25:36 | openstackgerrit | Zhenyu Zheng proposed openstack/nova master: nova-manage db archive_deleted_rows is not multi-cell aware https://review.openstack.org/507486 | |
| 06:30:13 | openstackgerrit | OpenStack Proposal Bot proposed openstack/nova master: Imported Translations from Zanata https://review.openstack.org/541561 | |
| 07:38:43 | openstackgerrit | Andreas Jaeger proposed openstack/os-vif master: zuul: Enable functional tests in gate https://review.openstack.org/530961 | |
| 07:56:50 | bauzas | good morning folks | |
| 08:01:46 | hrw | Graphical console stuff got merged! yay and hurray! | |
| 08:10:28 | openstack | Launchpad bug 1741319 in OpenStack nova-compute charm "arm64: Migration pre-check error: CPU doesn't have compatibility." [Medium,Incomplete] | |
| 08:10:28 | bauzas | hrw: you might be interested in https://bugs.launchpad.net/nova/+bug/1741319 | |
| 08:10:42 | bauzas | hrw: just let me know if that's a valid bug so I can triage it | |
| 08:11:35 | hrw | bauzas: there is no cpu_model = host-model on aarch64 | |
| 08:12:07 | hrw | bauzas: so bug is valid and not solvable | |
| 08:12:22 | bauzas | hrw: that's what I understood | |
| 08:12:32 | bauzas | since we compare models for live migrating | |
| 08:14:28 | hrw | added note | |
| 08:15:34 | bauzas | hrw: so, for the moment, live-migration isn't supported for ARM64, I guess then | |
| 08:15:49 | hrw | bauzas: not quite. | |
| 08:15:49 | bauzas | if so, we should amend the doc | |
| 08:16:06 | hrw | bauzas: you may migrate but between same cpus | |
| 08:16:36 | hrw | will ask our testers to check it (as my setup is all-in-one) | |
| 08:17:11 | bauzas | hrw: the reporter says they have exact CPUs | |
| 08:17:15 | bauzas | exact same* | |
| 08:17:33 | bauzas | hrw: but the point is, nova is checking something that ARM64 libvirt doesn't pass, that's ity | |
| 08:17:45 | hrw | ok | |
| 08:18:07 | bauzas | so that probably needs some kind of discussion about the possible solutions | |