Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-03-22
16:17:51 sean-k-mooney jaypipes: yes that is what i ment
16:18:23 sean-k-mooney i just wanted to make sure we dont claim 4VCPUs and 2 PCPUS
16:18:50 cfriesen sean-k-mooney: jaypipes: hw:overhead_dedicated_set doesn't make sense since we may want to run emulator threads separate from all vCPU threads (this is what "isolate" does today)
16:19:05 sean-k-mooney and in this case the guest recived 5 logical cpus and the emulator tread is pinned to the PCPU that was allocated to the vm
16:19:11 cdent efried: it probably doesn't matter _that_ much as long as documentation etc
16:19:17 cdent (whichever choice we make)
16:19:31 jaypipes cfriesen: ack. I'm reworking this in the spec after your feedback.
16:19:56 sean-k-mooney cfriesen when you say isolate which isolate do you mean
16:20:00 efried cdent, dansmith, edleafe: Okay, let's make a call one way or another. And send an email. And propose a spec delta. And fix the code. And close the books.
16:20:00 jaypipes cfriesen, sean-k-mooney: can there ever be >1 emulator thread, though?
16:20:08 cfriesen jaypipes: yes
16:20:12 edleafe cdent: efried: I didn't realize that we didn't accept multiple resources= params. But I do agree that changing that now is not a good use of time
16:20:15 jaypipes sean-k-mooney: he is referring to hw:emulator_threads_policy=isolate
16:20:22 cfriesen jaypipes: and there can be multiple I/O threads too
16:20:50 jaypipes cfriesen: and how does "hw:emulator_threads_policy=isolate" account for >1 emulator thread? that was the whole point of the overhead_dedicated_set being a pinset -- allowing >1 emulator thread to be specified.
16:20:56 sean-k-mooney jaypipes: ah ok hw:cpu_threads_policy=isolate is different just makeing sure
16:21:11 cfriesen jaypipes: all the emulator threads get affined to a single physical cpu
16:21:44 jaypipes cfriesen: that sounds like a libvirt-specific assumption that will likely change at any given time (as most of these things tend to do)
16:22:03 sahid jaypipes: oh so you can't change VCPU? in that case i'm ok with PCPU
16:22:18 cfriesen jaypipes: the emulator threads don't usually have a lot of work to do (the exception is during live migration I think). the goal of "hw:emulator_threads_policy=isolate" is to ensure that the minimal work they do doesn't interrupt the guest cpu threads
16:22:25 jaypipes sahid: I mean, we *could*, but that would be a seriously annoying code change ;)
16:23:00 mriedem melwitt: if you approve a blueprint like https://blueprints.launchpad.net/nova/+spec/hurrah-for-privsep-again you should set the 'series goal' to rocky so it shows up on the actual rocky blueprints page: https://blueprints.launchpad.net/nova/rocky
16:23:36 melwitt mriedem: sorry, I missed that it wasn't set. setting now
16:24:05 jaypipes cfriesen: so it never makes sense to have emulator threads on dedicated CPUs and guest vCPU threads on shared CPUs, right?
16:24:12 cfriesen jaypipes: yes, I suppose that's qemu-specific
16:24:34 cfriesen jaypipes: no, that wouldn't make sense.
16:24:37 sahid jaypipes: yes i'm agreed with cfriesen, that never make sense
16:25:04 jaypipes cfriesen: furthermore, from what I gather from your and sean-k-mooney's feedback, it also would never make sense to have emulator threads policy set to *anything* if the guest vCPU threads were *not* pinned to dedicated CPUs.
16:25:28 cfriesen jaypipes: yes
16:25:35 jaypipes k
16:26:16 cfriesen jaypipes: but it might make sense to have vcpu threads on dedicated cpu, and emulator threads on shared cpu (lumped together with other emulator threads and shared vcpu threads)
16:27:01 stephenfin yeah, that's one of the ideas that came up in reviews of sahid's spec ^
16:27:08 cfriesen jaypipes: sahid: I'm just not sure if we need to keep the "allocate a whole host cpu for my emulator threads because I'm super important and don't want other guests to impact my emulator threads"
16:27:42 sahid cfriesen: yes, you have asked me, but did not have responded. I think that is not necessary
16:27:44 stephenfin assuming we gain 'pcpu_pin_set' option (or whatever tetsuro's spec called for)
16:28:22 sahid we just want in some cases the emulthreads to run somewhere
16:28:26 jaypipes stephenfin: CONF.cpu_dedicated_set
16:28:32 cfriesen If we don't need that, then the simplest thing would be to enforce that we always have at least one shared host CPU, and if you ask for hw:cpu_threads_policy=isolate it runs there.
16:28:33 stephenfin that's the one
16:28:44 jaypipes stephenfin: is my proposal, along with CONF.cpu_shared_set being the other disjoint set.
16:28:59 stephenfin or maybe not, but the idea's the same
16:29:13 jaypipes cfriesen: you mean emulator_threads_policy, not cpu_threads_policy.
16:29:18 cfriesen bah, yes
16:29:20 sahid cfriesen: i see you point, we can't have cpu_shared_set empty
16:29:29 jaypipes yet another reason all this stuff is so confusing...
16:30:09 stephenfin So we'd completely remove the ability to have one core per guest for emulator threads (e.g. what we currently have)
16:30:19 stephenfin not that I'm against that. Just making sure I understand
16:30:29 cfriesen stephenfin: that's really an implementation detail currently, but yes
16:30:29 sahid stephenfin: yes, no need of that anymore
16:30:37 stephenfin (y)
16:31:38 jaypipes stephenfin: I don't understand that last statement...
16:31:47 jaypipes stephenfin: why would we be removing that ability?
16:32:03 cfriesen jaypipes: if we assume that the emulator thread work scales with the number of vcpus, then we don't need to account for it separately, it's already in cpu_allocation_ratio. This would simplify things since you wouldn't need to ask for an extra VCPU from placement.
16:32:21 stephenfin jaypipes: Because you can't account for the extra per-guest CPU
16:32:24 cfriesen jaypipes: we're saying we'd remove the code that allocates a whole extra host CPU to run the emulator threads
16:32:36 jaypipes cfriesen: that's exactly *not* what I want.
16:32:36 stephenfin what cfriesen says
16:33:03 cfriesen jaypipes: sorry, which conversation thread is that for?
16:33:04 cfriesen :)
16:33:08 jaypipes cfriesen: *something* needs to account for CPU resources consumed by emulator threads. right now, nothing is accounted for in placement.
16:33:43 cfriesen jaypipes: there are two possibilities. 1) emulator thread work scales with the number of guests. 2) emulator thread work scales with the number of vcpus
16:34:06 stephenfin jaypipes: Does it though? I mean, if those threads are running on cores meant for 'shared' workloads, do we care?
16:34:40 cfriesen jaypipes: if 1 is true, then yes, we need to account for the work done by the emulator threads
16:34:41 stephenfin After all, we don't account for the overhead of emulator threads _without_ the policy applied. We could just ignore it
16:34:44 openstackgerrit Matt Riedemann proposed openstack/osc-placement master: Resolve nits from I552688b9ee32b719a576a7a9ed5e4d5aa31d7b3f https://review.openstack.org/537971
16:34:57 jaypipes stephenfin: what's the difference between an emulator thread running on a shared CPU and a guest vCPU thread running on a shared CPU, then?
16:35:15 stephenfin You charge for the latter
16:35:17 jaypipes stephenfin: we account for the latter, not the formter.
16:35:25 cfriesen jaypipes: if 2 is true, then the work done by the emulator thread is "pooled" with the work done by the vCPU threads and it's all accounted for in cpu_allocation_ratio
16:35:31 jaypipes stephenfin: but that's my point... we're not accounting for the former when we should be.
16:36:04 stephenfin Also, the potential cycles consumed by emulator thread <<< guest vCPU thread
16:36:19 jaypipes stephenfin: except during live migration?
16:36:28 stephenfin good point
16:36:47 jaypipes my point being, we're currently not accounting for any of this stuff.
16:37:04 cfriesen jaypipes: stephenfin: I think the interesting case would be if you have a bunch of "dedicated" guest cpus, and the emulator thread running on the "shared" host cpu pool
16:37:22 cfriesen because in that case we really don't account for the emulator thread work currently
16:37:41 mriedem johnthetubaguy: if you're still around, https://review.openstack.org/#/c/520248/
16:37:52 stephenfin cfriesen: I thought we did. sahid extended some 'overhead' function to do that
16:37:59 stephenfin *currently did
16:38:11 johnthetubaguy mriedem: been meaning to catch you about that
16:38:19 cfriesen stephenfin: right now the emulator threads for an instance run on a dedicated host cpu
16:38:23 cfriesen and we do account for that
16:38:33 mriedem stephenfin: he did https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L804
16:38:35 cfriesen (if "isolate" is enable)
16:38:49 stephenfin cfriesen: Ah, ok. I missed that nuance
16:39:00 cfriesen stephenfin: but we're talking about running the emulator threads on the "shared" host cpus now
16:39:03 mriedem note that we don't account for overhead in placement at all
16:39:08 mriedem since it's per-compute, and done on the compute
16:39:24 jaypipes cfriesen: well, *all* emulator threads run on the same dedicated CPU? or *each* emulator thread runs on a dedicated host CPU?
16:39:35 cfriesen jaypipes: all on the same
16:39:39 openstackgerrit sahid proposed openstack/nova-specs master: virt: allow instances to be booted with trusted VFs https://review.openstack.org/485522
16:39:49 jaypipes cfriesen: k
16:39:55 stephenfin cfriesen: At the moment?
16:40:03 jaypipes mriedem: this is a different overhead...
16:40:07 johnthetubaguy mriedem: ah https://specs.openstack.org/openstack/nova-specs/specs/queens/implemented/remove-configurable-hide-server-address-feature.html would fix that
16:40:13 cfriesen jaypipes: but each instance with "isolate" enabled gets a separate host CPU
16:40:25 jaypipes cfriesen: that's what I just asked you...
16:40:43 stephenfin jaypipes: A guest can have multiple emulator threads
16:40:51 cfriesen jaypipes: all the emulator threads for a given instance run on a single dedicated host CPU
16:40:52 jaypipes cfriesen: ok, so it's each instance gets a separate dedicated host CPU for its emulator thread if emulator_threads_policy=isolate.

Earlier   Later