| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-22 | |||
| 16:22:25 | jaypipes | sahid: I mean, we *could*, but that would be a seriously annoying code change ;) | |
| 16:23:00 | mriedem | melwitt: if you approve a blueprint like https://blueprints.launchpad.net/nova/+spec/hurrah-for-privsep-again you should set the 'series goal' to rocky so it shows up on the actual rocky blueprints page: https://blueprints.launchpad.net/nova/rocky | |
| 16:23:36 | melwitt | mriedem: sorry, I missed that it wasn't set. setting now | |
| 16:24:05 | jaypipes | cfriesen: so it never makes sense to have emulator threads on dedicated CPUs and guest vCPU threads on shared CPUs, right? | |
| 16:24:12 | cfriesen | jaypipes: yes, I suppose that's qemu-specific | |
| 16:24:34 | cfriesen | jaypipes: no, that wouldn't make sense. | |
| 16:24:37 | sahid | jaypipes: yes i'm agreed with cfriesen, that never make sense | |
| 16:25:04 | jaypipes | cfriesen: furthermore, from what I gather from your and sean-k-mooney's feedback, it also would never make sense to have emulator threads policy set to *anything* if the guest vCPU threads were *not* pinned to dedicated CPUs. | |
| 16:25:28 | cfriesen | jaypipes: yes | |
| 16:25:35 | jaypipes | k | |
| 16:26:16 | cfriesen | jaypipes: but it might make sense to have vcpu threads on dedicated cpu, and emulator threads on shared cpu (lumped together with other emulator threads and shared vcpu threads) | |
| 16:27:01 | stephenfin | yeah, that's one of the ideas that came up in reviews of sahid's spec ^ | |
| 16:27:08 | cfriesen | jaypipes: sahid: I'm just not sure if we need to keep the "allocate a whole host cpu for my emulator threads because I'm super important and don't want other guests to impact my emulator threads" | |
| 16:27:42 | sahid | cfriesen: yes, you have asked me, but did not have responded. I think that is not necessary | |
| 16:27:44 | stephenfin | assuming we gain 'pcpu_pin_set' option (or whatever tetsuro's spec called for) | |
| 16:28:22 | sahid | we just want in some cases the emulthreads to run somewhere | |
| 16:28:26 | jaypipes | stephenfin: CONF.cpu_dedicated_set | |
| 16:28:32 | cfriesen | If we don't need that, then the simplest thing would be to enforce that we always have at least one shared host CPU, and if you ask for hw:cpu_threads_policy=isolate it runs there. | |
| 16:28:33 | stephenfin | that's the one | |
| 16:28:44 | jaypipes | stephenfin: is my proposal, along with CONF.cpu_shared_set being the other disjoint set. | |
| 16:28:59 | stephenfin | or maybe not, but the idea's the same | |
| 16:29:13 | jaypipes | cfriesen: you mean emulator_threads_policy, not cpu_threads_policy. | |
| 16:29:18 | cfriesen | bah, yes | |
| 16:29:20 | sahid | cfriesen: i see you point, we can't have cpu_shared_set empty | |
| 16:29:29 | jaypipes | yet another reason all this stuff is so confusing... | |
| 16:30:09 | stephenfin | So we'd completely remove the ability to have one core per guest for emulator threads (e.g. what we currently have) | |
| 16:30:19 | stephenfin | not that I'm against that. Just making sure I understand | |
| 16:30:29 | cfriesen | stephenfin: that's really an implementation detail currently, but yes | |
| 16:30:29 | sahid | stephenfin: yes, no need of that anymore | |
| 16:30:37 | stephenfin | (y) | |
| 16:31:38 | jaypipes | stephenfin: I don't understand that last statement... | |
| 16:31:47 | jaypipes | stephenfin: why would we be removing that ability? | |
| 16:32:03 | cfriesen | jaypipes: if we assume that the emulator thread work scales with the number of vcpus, then we don't need to account for it separately, it's already in cpu_allocation_ratio. This would simplify things since you wouldn't need to ask for an extra VCPU from placement. | |
| 16:32:21 | stephenfin | jaypipes: Because you can't account for the extra per-guest CPU | |
| 16:32:24 | cfriesen | jaypipes: we're saying we'd remove the code that allocates a whole extra host CPU to run the emulator threads | |
| 16:32:36 | jaypipes | cfriesen: that's exactly *not* what I want. | |
| 16:32:36 | stephenfin | what cfriesen says | |
| 16:33:03 | cfriesen | jaypipes: sorry, which conversation thread is that for? | |
| 16:33:04 | cfriesen | :) | |
| 16:33:08 | jaypipes | cfriesen: *something* needs to account for CPU resources consumed by emulator threads. right now, nothing is accounted for in placement. | |
| 16:33:43 | cfriesen | jaypipes: there are two possibilities. 1) emulator thread work scales with the number of guests. 2) emulator thread work scales with the number of vcpus | |
| 16:34:06 | stephenfin | jaypipes: Does it though? I mean, if those threads are running on cores meant for 'shared' workloads, do we care? | |
| 16:34:40 | cfriesen | jaypipes: if 1 is true, then yes, we need to account for the work done by the emulator threads | |
| 16:34:41 | stephenfin | After all, we don't account for the overhead of emulator threads _without_ the policy applied. We could just ignore it | |
| 16:34:44 | openstackgerrit | Matt Riedemann proposed openstack/osc-placement master: Resolve nits from I552688b9ee32b719a576a7a9ed5e4d5aa31d7b3f https://review.openstack.org/537971 | |
| 16:34:57 | jaypipes | stephenfin: what's the difference between an emulator thread running on a shared CPU and a guest vCPU thread running on a shared CPU, then? | |
| 16:35:15 | stephenfin | You charge for the latter | |
| 16:35:17 | jaypipes | stephenfin: we account for the latter, not the formter. | |
| 16:35:25 | cfriesen | jaypipes: if 2 is true, then the work done by the emulator thread is "pooled" with the work done by the vCPU threads and it's all accounted for in cpu_allocation_ratio | |
| 16:35:31 | jaypipes | stephenfin: but that's my point... we're not accounting for the former when we should be. | |
| 16:36:04 | stephenfin | Also, the potential cycles consumed by emulator thread <<< guest vCPU thread | |
| 16:36:19 | jaypipes | stephenfin: except during live migration? | |
| 16:36:28 | stephenfin | good point | |
| 16:36:47 | jaypipes | my point being, we're currently not accounting for any of this stuff. | |
| 16:37:04 | cfriesen | jaypipes: stephenfin: I think the interesting case would be if you have a bunch of "dedicated" guest cpus, and the emulator thread running on the "shared" host cpu pool | |
| 16:37:22 | cfriesen | because in that case we really don't account for the emulator thread work currently | |
| 16:37:41 | mriedem | johnthetubaguy: if you're still around, https://review.openstack.org/#/c/520248/ | |
| 16:37:52 | stephenfin | cfriesen: I thought we did. sahid extended some 'overhead' function to do that | |
| 16:37:59 | stephenfin | *currently did | |
| 16:38:11 | johnthetubaguy | mriedem: been meaning to catch you about that | |
| 16:38:19 | cfriesen | stephenfin: right now the emulator threads for an instance run on a dedicated host cpu | |
| 16:38:23 | cfriesen | and we do account for that | |
| 16:38:33 | mriedem | stephenfin: he did https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L804 | |
| 16:38:35 | cfriesen | (if "isolate" is enable) | |
| 16:38:49 | stephenfin | cfriesen: Ah, ok. I missed that nuance | |
| 16:39:00 | cfriesen | stephenfin: but we're talking about running the emulator threads on the "shared" host cpus now | |
| 16:39:03 | mriedem | note that we don't account for overhead in placement at all | |
| 16:39:08 | mriedem | since it's per-compute, and done on the compute | |
| 16:39:24 | jaypipes | cfriesen: well, *all* emulator threads run on the same dedicated CPU? or *each* emulator thread runs on a dedicated host CPU? | |
| 16:39:35 | cfriesen | jaypipes: all on the same | |
| 16:39:39 | openstackgerrit | sahid proposed openstack/nova-specs master: virt: allow instances to be booted with trusted VFs https://review.openstack.org/485522 | |
| 16:39:49 | jaypipes | cfriesen: k | |
| 16:39:55 | stephenfin | cfriesen: At the moment? | |
| 16:40:03 | jaypipes | mriedem: this is a different overhead... | |
| 16:40:07 | johnthetubaguy | mriedem: ah https://specs.openstack.org/openstack/nova-specs/specs/queens/implemented/remove-configurable-hide-server-address-feature.html would fix that | |
| 16:40:13 | cfriesen | jaypipes: but each instance with "isolate" enabled gets a separate host CPU | |
| 16:40:25 | jaypipes | cfriesen: that's what I just asked you... | |
| 16:40:43 | stephenfin | jaypipes: A guest can have multiple emulator threads | |
| 16:40:51 | cfriesen | jaypipes: all the emulator threads for a given instance run on a single dedicated host CPU | |
| 16:40:52 | jaypipes | cfriesen: ok, so it's each instance gets a separate dedicated host CPU for its emulator thread if emulator_threads_policy=isolate. | |
| 16:40:53 | stephenfin | so your question was confusing :) | |
| 16:41:07 | stephenfin | jaypipes: *for its emulator threads | |
| 16:41:13 | stephenfin | (plural) | |
| 16:41:16 | jaypipes | stephenfin: we don't currently support >1 emulator thread. | |
| 16:41:29 | cfriesen | jaypipes: qemu has multiple emulator threads, so we do | |
| 16:41:34 | mriedem | johnthetubaguy: yeah mentioned that in reply to the patch | |
| 16:41:37 | stephenfin | indeed | |
| 16:41:38 | cfriesen | jaypipes: but they're all pinned the same | |
| 16:41:48 | jaypipes | cfriesen: but *we* (i.e. nova libvirt virt driver) never sets >1 right? | |
| 16:41:53 | johnthetubaguy | mriedem: yeah, that is where I got that from | |
| 16:42:03 | cfriesen | jaypipes: it's not up to libvirt, I don't think. it's internal to qemu. | |
| 16:42:09 | stephenfin | this is where sahid should butt in | |
| 16:42:47 | cfriesen | jaypipes: the existance of multiple threads is an implementation detail that is unimportant to the present discussion. | |
| 16:43:00 | sahid | jaypipes: by default when using cpu_policy:dedicated we pin the emulatore threads to the set of pCPUs dedicated for guest | |
| 16:43:02 | johnthetubaguy | mriedem: I think I am about to +W this given we hide it during building, which was my main concern, that seem reasonable? | |
| 16:43:03 | sahid | if that is your question | |
| 16:43:15 | mriedem | johnthetubaguy: i think you +Wing my patch is certainly reasonable yes | |
| 16:43:19 | cfriesen | jaypipes: libvirt will only let you set a single emulator thread affinity for a given domain | |
| 16:43:20 | mriedem | thank you for being british | |
| 16:43:25 | johnthetubaguy | mriedem: heh | |