Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-02-06
22:25:51 efried sean-k-mooney: Yes.
22:26:02 efried One aggregate per NUMA node, in the global sense :)
22:26:24 sean-k-mooney oh ame that aggreate will be am large
22:26:35 efried Is that "a lot" of aggregates? Meh. Aggregates are cheap.
22:27:16 efried I feel like we're not talking in the same terms. Here: If you have 10 hosts, and each host has 2 NUMA nodes, you have 20 aggregates total.
22:27:18 sean-k-mooney i think i would prefer lots of small aggregate then a few giant cluster wide ones
22:27:58 sean-k-mooney efried: ah ok that what i was thinking we should have if we used aggregates so im cool with that
22:29:04 sean-k-mooney we dont have traits on aggregates but at least for numa i dont think we need them on teh aggreages so it could work
22:29:21 cfriesen so you'd have these per-numa-node aggregates for system use, then per-host aggregates to match against flavor/image properties and act as availability zones?
22:29:44 efried sean-k-mooney: We can always stamp the involved providers with traits. But not sure we need 'em. At least to mark NUMA nodes.
22:30:02 efried cfriesen: I know nothing about host aggregates or availability zones.
22:30:16 sean-k-mooney cfriesen: well an aggreaget is just a bag of resouce providers nothing more or less. you can use them to group anything you want
22:30:18 efried cfriesen: Except that placement aggregates != host aggregates
22:30:39 efried cfriesen: The other main use case for aggregates is to associate sharing resource providers, like a shared storage pool.
22:31:25 efried ^ placement aggregates, that is.
22:32:07 cfriesen efried: okay...I'm not up on the placement terminology. :)
22:32:35 efried cfriesen: I'm not up on... pretty much everything else. Between the two of us, we're gonna need a pidgin.
22:33:22 sean-k-mooney cfriesen: a sharing resouce provider is jsut a normal resouce provider with a trait and then its inventores are consumable by anything in the same aggreates as it
22:34:36 cfriesen so how do we handle the idea of an instance consuming cpu/ram from one placement aggregate but a PCI device from another placement aggregate? (for the non-strict PCI affinity case)
22:34:44 openstackgerrit Matt Riedemann proposed openstack/nova master: Test websocketproxy with TLS in the nova-next job https://review.openstack.org/513160
22:34:49 mriedem melwitt: cleaned that up ^
22:35:28 sean-k-mooney cfriesen: instance consume resocues form invetories provided by resouce providers
22:35:40 melwitt mriedem: thanks. sorry I lost track of that one
22:35:47 sean-k-mooney cfriesen: and resouce provires can be member of 0-n aggregates
22:36:30 sean-k-mooney so you are not consuming form the aggregates
22:37:22 efried cfriesen: In the design I'm strawpersoning, if you care about CPU/mem affinity and you don't care about VF affinity, you would say affine_resources=VCPU,MEMORY_MB but not SRIOV_NET_VF
22:37:46 efried cfriesen: s/VF/PCI device/ if you wish.
22:39:08 sean-k-mooney efried: that may break down a bit with anti afinity
22:39:33 efried sean-k-mooney: I'm saying we're not handling anti-affinity at all with this model.
22:39:34 sean-k-mooney e.g. i want 2 VF whoes parent PF are anti affined
22:39:48 sean-k-mooney efried: ah ok
22:39:52 efried sean-k-mooney: Yeah, I get that. But that's not a NUMA use case, right?
22:40:01 efried It's more about HA
22:40:05 efried redundancy
22:40:15 efried powerfail grouping
22:40:18 efried that kind of thing.
22:40:21 sean-k-mooney ya its for ha bonding so my bond dont come from the same card
22:40:51 sean-k-mooney e.g. pulling out one cable wont bring donw the bond
22:40:51 efried So yeah, this doesn't cover that. And if there's a generic solution that encompasses both, great. But I'm gonna have to pick Jay's brain for that.
22:42:24 sean-k-mooney efried: if you can show your propasl support affinity in a clean way and does not block future addtion of anti afinity by design then its better then nothing
22:43:02 sean-k-mooney e.g. we dont have to support everything in one go in rocky
22:43:33 sean-k-mooney it would be nice however
22:44:25 efried That's kinda what I was thinking. Propose it this way, show that it wouldn't prevent a more generic solution in the future, and then Jay can propose (or not) that more generic solution.
22:46:14 cfriesen efried: what about multi-numa-node guests? they would need to essentially do numa anti-affinity (but host affinity) between the two pools of memory
22:47:02 efried cfriesen: What does that mean? Multi-numa-node guests? You mean guests that don't care about NUMA affinity? Or do they explicitly care about *anti* affinity?
22:47:05 sean-k-mooney cfriesen: multi numa guest do not need to be on multiple host numa nodes
22:47:14 cfriesen efried: no, I mean guests with multi numa nodes
22:47:22 sean-k-mooney cfriesen: i consider that limitation in libvirt to be a bug not a feature
22:47:30 sean-k-mooney libvirt virt driver that is
22:47:45 cfriesen sean-k-mooney: one reason why you might "require" multi-numa-node guests is to get access to more memory bandwidth
22:48:48 sean-k-mooney cfriesen: yes but there is a flavour extra spec defined for that that only the hyperv dirver support
22:49:00 sean-k-mooney cfriesen: i think the libvirt one should too
22:49:17 cfriesen sean-k-mooney: sorry, don't follow.
22:50:09 sean-k-mooney cfriesen: hw:numa_mempolicy
22:50:13 cfriesen sean-k-mooney: I can currently specify a multi-numa-node guest with libvirt, and I get access to that many host numa nodes, with accordingly more host memory bandwidth. (presumably combined with dedicated cpus)
22:50:59 cfriesen numa_mempolicy isn't mentioned anywhere in nova
22:51:41 sean-k-mooney cfriesen: i used to be in the hyperv driver
22:52:06 sean-k-mooney anyway that is off topic
22:52:25 sean-k-mooney efried: a multi numa gues is a guest with multiple virtual numa nodes
22:52:55 cfriesen efried: and currently they map to different physical numa nodes of the same host.
22:53:03 sean-k-mooney efried: you can choose what virtual cores are associated to each virtual numa node and you can device the memoy between each virtual numa node
22:54:09 efried Well, if you wanted to get that specific, you could probably swang something with granular request groups.
22:54:29 efried But we still wouldn't be able to guarantee you anti-affinity.
22:54:49 cfriesen if you can't guarantee numa-antiaffinity it'll change the current behaviour
22:55:02 cfriesen (ie if both virtual numa nodes could end up on the same host numa node)
22:55:06 sean-k-mooney placement does not need to give that guarentee
22:55:14 sean-k-mooney the virt driver can via the resouce tracker
22:55:40 efried sean-k-mooney: Yeahhhh, but we eventually *should* be able to implement that filter in placement.
22:55:44 sean-k-mooney or the nova numa topology filter
22:56:09 sean-k-mooney efried: talk to jay about that. in principal i agree
22:57:23 sean-k-mooney i would like placement to fully handel numa. but if it can handel 80% of numa in rocky with the rest in the filetr i would be happy with that too
22:57:46 efried Okay. I'll try to put up that spec later this week.
22:59:43 sean-k-mooney cfriesen: im sure you know this also but just requesting guest numa nodes was never ment to give any form of cpu affinity either. from an api perspcetive it does not but for libvir it pince the cores of that numa node to float over a host numa node
23:00:44 sean-k-mooney cfriesen: similay requsting hugepage was not ment to create a numa topology for the guest and also restice the floating cores to a host numa node
23:01:17 sean-k-mooney cfriesen: all of those implit behaviors are internal implentation behaviors of the libvirt driver
23:01:50 cfriesen sean-k-mooney: the hugepage one is a qemu limitation though since you can only point at a single backing file for memory
23:01:57 cfriesen (on a given guest numa node)
23:03:04 sean-k-mooney cfriesen: yes and no you numa node number is optionaly and numactl will be used to determin the optimal numa node to use of you dont specify one
23:04:02 cfriesen sean-k-mooney: how do you tell qemu to use hugepages without specifying a backing file?
23:05:05 sean-k-mooney cfriesen: you do specify a backing file but the backing file itesf does not have numa affintiy
23:06:02 sean-k-mooney cfriesen: when you create a hugepage backing file under /dev/hugepages/libvrt/qemu the numa node is not part of the path
23:06:45 sean-k-mooney cfriesen: checkout https://libvirt.org/formatdomain.html#elementsMemoryBacking the note on the nodeset being optional
23:10:03 sean-k-mooney cfriesen: tacking hugepages per numa node made the resouce track code simpeler which is why it was doen per numa node originally
23:17:30 cfriesen sean-k-mooney: ah, right. I was thinking that mapping hugetlbfs was numa-affined, but it's not necessarily.
23:18:04 sean-k-mooney cfriesen: you might be able to do somthing via the mount options but by defualt no
23:19:19 sean-k-mooney efried: actully speaking of memory and numa we should really be tracking 4k pages e.g. non hugepage memory per numa node too
23:19:46 sean-k-mooney the kernel out of memory killer is triggered per numa node not gloably
23:20:06 sean-k-mooney so the reserved memory config option does not work properly on a numa system
23:20:53 sean-k-mooney vms can still be killed if a numa node has no memoy left and the kernel need to allocate some on that node but the host othersise has free ram
23:21:33 sean-k-mooney our just make all your vms use hugepages that works too
23:25:06 cfriesen that's what we do by default (2M pages)
23:26:36 sean-k-mooney cfriesen: ya we spent 6 months trying to track down what was causeing OOM to kill our vms in the the intel nfv ci and it turned out to be this. hugepages fixed it
23:27:31 sean-k-mooney the anoying thing was it took about 2-3 weeks to start showing up as it need the memory to become fragmented enough to force it
23:42:38 openstackgerrit Patricia Domingues proposed openstack/nova master: load up the volume drivers by checking architecture https://review.openstack.org/541393
23:44:22 openstackgerrit Patricia Domingues proposed openstack/nova master: load up the volume drivers by checking architecture https://review.openstack.org/541393
23:58:46 openstackgerrit Matt Riedemann proposed openstack/nova master: Add a prelude release note for the 17.0.0 Queens GA https://review.openstack.org/541495
#openstack-nova - 2018-02-07
00:22:28 openstackgerrit Matt Riedemann proposed openstack/nova stable/pike: Fix pike GA prelude release note https://review.openstack.org/541498
00:44:23 openstackgerrit Merged openstack/nova master: docs: Add booting from an encrypted volume https://review.openstack.org/540506
01:06:42 openstackgerrit Arvind proposed openstack/nova-specs master: BP Support traits in Glance https://review.openstack.org/541502
01:51:06 openstackgerrit Arvind proposed openstack/nova-specs master: Support traits in Glance https://review.openstack.org/541507

Earlier   Later