| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-05-30 | |||
| 15:06:15 | bauzas | jaypipes: ^ | |
| 15:06:43 | bauzas | because nested RPs is still in early bits, and because I shit have no time to further work on implementing it for Rocky anyway | |
| 15:06:57 | bauzas | so I'll work on providing a new rev' of course, but not prioritary | |
| 15:07:12 | stephenfin | bauzas: Wait, how does that compare with jaypipes' spec? | |
| 15:07:13 | ildikov | stephenfin: I can bug you again if that helps :) | |
| 15:07:14 | jaypipes | bauzas: I have a feeling tetsuro will have made significant progress on much of that in Rocky. | |
| 15:07:28 | jaypipes | stephenfin: the cpu-resources spec is about dedicated cores, not NUMA. | |
| 15:07:40 | jaypipes | dedicated and shared cores. | |
| 15:07:41 | mriedem | cfriesen: i would like to see it merged, so i don't have thoughts on my own patch :) | |
| 15:07:45 | mriedem | cfriesen: you should ask other cores | |
| 15:07:48 | bauzas | jaypipes: I looked at tetsuro's patches and those are still under review, right? | |
| 15:07:57 | jaypipes | bauzas: of course, yes | |
| 15:08:05 | bauzas | so, it's in my pipe | |
| 15:08:09 | mriedem | cfriesen: just throw it in the starlingx mega patch-o-doom and it will get reviews from at&t for sure! | |
| 15:08:11 | stephenfin | jaypipes: But you're using NUMA, no? | |
| 15:08:23 | jaypipes | bauzas: not just tetsuro, though. efried and Citrix guys are pushing ahead. | |
| 15:08:33 | bauzas | but my point is, nested resource providers is just incepted, and possibly vGPUs are the first usecases that'll use those | |
| 15:08:47 | mriedem | cfriesen: the todo is for future refactor | |
| 15:08:48 | stephenfin | jaypipes: e.g. https://review.openstack.org/#/c/555081/10/specs/rocky/approved/cpu-resources.rst@217 | |
| 15:08:51 | bauzas | jaypipes: I'm working on that too, remember | |
| 15:08:52 | jaypipes | stephenfin: that latest revision I tried to remove as much ass possible any mentions of NUMA in so much as extra specs or anything about the NUMATopologyFilter. | |
| 15:08:59 | mriedem | cfriesen: if i meant to hold up the patch on the todo, i would have wip'ed it | |
| 15:09:07 | cfriesen | mriedem: the plan is to submit individual small patches... Actually, we're already using your patch so it's already in the list. | |
| 15:09:17 | bauzas | jaypipes: http://lists.openstack.org/pipermail/openstack-dev/2018-May/130783.html | |
| 15:09:18 | mriedem | cfriesen: heh i know | |
| 15:09:19 | cfriesen | mriedem: yeah, I caught that on re-read | |
| 15:09:29 | bauzas | jaypipes: you possibly missed it | |
| 15:09:30 | jaypipes | bauzas: I'm not holding up progress on n-r-p's various efforts ongoing. | |
| 15:09:33 | mriedem | cfriesen: i just need to stab the starlingx thing at every opportunity | |
| 15:09:35 | jangutter | sahid: Apologies for side-tracking, but I've got a question regarding the thorny issue of multiqueue and os-vif, if you have a few seconds. | |
| 15:09:43 | cfriesen | mriedem: bring it on, I can take it | |
| 15:09:45 | jaypipes | bauzas: I didn't miss it. just haven't had a chance to go through it. | |
| 15:09:51 | jaypipes | bauzas: will do today. | |
| 15:09:55 | bauzas | ack, thanks | |
| 15:09:58 | jaypipes | np | |
| 15:10:06 | mriedem | cfriesen: also, poutine, wtf?! | |
| 15:10:07 | bauzas | because I'm just litterally working on the implementation | |
| 15:10:20 | cfriesen | mriedem: good poutine is awesome. crappy poutine sucks | |
| 15:10:26 | stephenfin | jaypipes: I'm guessing you can't ignore NUMA completely though, if you want to support use case 4 and the likes? https://review.openstack.org/#/c/555081/10/specs/rocky/approved/cpu-resources.rst@425 | |
| 15:10:27 | bauzas | jaypipes: and I'm hold until we get consensus on the right approach | |
| 15:10:42 | bauzas | jaypipes: and FWIW, naichuan's implementation for Xen is impacted too | |
| 15:11:40 | jaypipes | stephenfin: no, I specifically *don't* want to include the realtime extra specs or any of the numa_nodes/numa_cells goo. | |
| 15:12:02 | jaypipes | stephenfin: because the purpose of that spec is to handle requests for dedicated and shared CPU resources. nothing to do with NUMA really. | |
| 15:12:17 | sahid | jangutter: i imagine what you are going to do it's add in nova a new condition for vif detail vrouter, which will enable the multiqueue=True | |
| 15:12:27 | mriedem | stephenfin: wasn't trying to bug you for reviews on my spec, just saying i'm in the same boat | |
| 15:12:30 | sahid | that under the section vhostuser | |
| 15:12:32 | jaypipes | stephenfin: and I'm deliberately trying to not bring in the awful coupling of NUMATopology and InstanceNUMATopology and the horrible extra specs that serve to construct that mess. | |
| 15:12:34 | mriedem | stephenfin: we can happily drown together | |
| 15:12:37 | mriedem | you can be leo | |
| 15:12:39 | mriedem | i'll be kate | |
| 15:12:49 | mriedem | / who didn't drown / | |
| 15:12:51 | jaypipes | mriedem: wow, how chivalrous of you :P | |
| 15:13:12 | jangutter | sahid: actually, the idea is to migrate the 'vrouter' vif which is not vhostuser... | |
| 15:13:14 | openstackgerrit | Merged openstack/nova-specs master: Libvirt file backed memory https://review.openstack.org/563704 | |
| 15:13:37 | stephenfin | jaypipes: I've missed something so. I thought requesting 'resources1:PCPU=8 resources2:PCPU=8 group_policy=isolate' means host NUMA topology would need to be accounted for | |
| 15:13:43 | jangutter | sahid: I did some checking, and it looks like multiqueue is an instance-wide setting, not a vif-specific one. | |
| 15:14:30 | jaypipes | stephenfin: no. it has nothing explicitly to do with NUMA. that just means "I need 2 groups of 8 dedicated CPU resources. they need to be provided by two distinct resource providers" | |
| 15:14:36 | jangutter | sahid: ref: https://github.com/openstack/nova/blob/master/nova/virt/libvirt/vif.py#L708 | |
| 15:15:21 | jaypipes | stephenfin: the fact that the compute node resource provider might be a parent to two other child providers that *represent* different sockets or NUMA cells is really not relevant to the request that two different providers of dedicated CPU supply the requested resource groups. | |
| 15:16:07 | sahid | jangutter: i'm not sure i understand what you want do? | |
| 15:16:20 | jaypipes | stephenfin: in other words, if some non-NUMA devices *happened* to expose PCPU inventory under a provider tree, they would meet the requirements... | |
| 15:16:25 | bauzas | jaypipes: stephenfin: just a point, we could have the same upgrade impact with VCPU resources being in child RPs than the one I described with http://lists.openstack.org/pipermail/openstack-dev/2018-May/130783.html | |
| 15:16:32 | stephenfin | jaypipes: Practically speaking so, requesting 'resources1:PCPU=8 resources2:PCPU=8 group_policy=isolate' will always fail until NUMA is modelled in placement? | |
| 15:16:40 | stephenfin | Ah, gotcha | |
| 15:16:41 | jaypipes | stephenfin: again, we are trying desperately to free ourselves of the NUMA coupling that has plagued all of this for years. | |
| 15:17:13 | jangutter | sahid: vrouter kernel mode plugging needs to create a tap with multiqueue enabled, if libvirt starts the VM up looking for it. | |
| 15:17:30 | stephenfin | jaypipes: But we're still coupled, no? I mean, we're going to need NUMA for the above request to work | |
| 15:17:42 | bhagyashris | efried: any suggestion about second approach | |
| 15:17:43 | jaypipes | stephenfin: I'm trying my best to treat NUMA as just the model of distances to various memory (and memory page) caches to a processor/bus, instead of coupling all that mess with CPU pinning. | |
| 15:18:07 | jangutter | sahid: the VIFGeneric portion can set the libvirt config correctly for multiqueue, but, it looks like it's going to be difficult to pass "multiqueue" on a per-vif basis without modding large portions of the os-vif plugging code. | |
| 15:18:09 | sahid | jangutter: ok so you plugin needs to know whether multiqueue is enable, which is via the os-vif object nova is filling then passing the the os-vif plugin | |
| 15:18:21 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add osprofiler config options to generated reference https://review.openstack.org/571225 | |
| 15:18:51 | jangutter | sahid: yep, so, either we can pass "multiqueue=True" via the vif os-vif object, or via the instance os-vif object. | |
| 15:19:59 | bauzas | jaypipes: unfortunately from what I understood from operators, pinning is there because NUMA affinity misses for PCI devices | |
| 15:20:09 | bauzas | jaypipes: so both are somehow related | |
| 15:20:16 | stephenfin | jaypipes: To be clear though, use case 4 in that spec cannot be achieved with modelling NUMA in placement, right? | |
| 15:20:19 | jangutter | sahid: the trick is, passing it via the VIF looks very tricky. A lot of code has to be added there, since, paradoxically, the choice whether to enable multiqueue depends on the instance and flavor settings, not VIF settings. | |
| 15:20:37 | stephenfin | jaypipes: Use case 4 = https://review.openstack.org/#/c/555081/10/specs/rocky/approved/cpu-resources.rst@425 | |
| 15:20:43 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add osprofiler config options to generated reference https://review.openstack.org/571225 | |
| 15:21:03 | bauzas | jaypipes: in terms of priority, I'd consider NUMA placement and NUMA affinity be prioritary to CPU pinning, because folks care more about proximity and throughput than real placement decisions | |
| 15:21:13 | openstackgerrit | Merged openstack/nova master: doc: Start using openstackdoctheme's extlink extension https://review.openstack.org/540888 | |
| 15:21:14 | sahid | jangutter: ok i see, I could see it live in instance actually | |
| 15:21:19 | bauzas | CPU pinning is just a hack | |
| 15:21:33 | stephenfin | Hurrah! The extlink patch merged at last | |
| 15:21:44 | jangutter | sahid: for example, you can spawn an instance asking for multiqueue, but only 1 cpu, and multiqueue will magically disable :-p | |
| 15:21:50 | jaypipes | bauzas: I'm referring to the poor software design of coupling the topology objects together with memory pages, usage of instance virtual NUMA topology on a host NUMA topology, and all that. | |
| 15:22:00 | bauzas | jaypipes: sure, I understood that point | |
| 15:22:03 | stephenfin | bauzas: I think jaypipes is suggesting we can do CPU pinning without _needing_ to think about NUMA affinity. They don't have to be tied | |
| 15:22:29 | bauzas | stephenfin: my own take is that CPU pinning isn't required if we go with NUMA affinity | |
| 15:22:30 | sahid | jangutter: yes i that is not what you want? | |
| 15:22:38 | stephenfin | But anyone that is using this will want to couple them, given the performance implications of not doing so | |
| 15:22:54 | stephenfin | bauzas: It isn't currently, no | |
| 15:22:57 | jangutter | sahid: That's perfectly acceptable, as long as both sides agree. | |
| 15:23:06 | bauzas | stephenfin: I don't disagree with your point :) | |
| 15:23:17 | jaypipes | stephenfin: use case 4 can be implemented, sure, but until sockets are modeled as child providers or a compute node parent provider, the scheduler would always return NoValidHosts for a request like use case 4. that clear? | |
| 15:23:26 | bauzas | stephenfin: I mean, CPU pinning is tied to NUMA resources because people care of throughput | |
| 15:23:45 | stephenfin | jaypipes: Sure, if s/sockets/NUMA nodes/ | |
| 15:23:48 | jangutter | sahid: another reason for keeping it as a parameter of the instance... | |
| 15:23:52 | stephenfin | because of Cluster-on-Die etc. | |
| 15:23:58 | jaypipes | stephenfin: sigh... | |