| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-22 | |||
| 16:30:29 | sahid | stephenfin: yes, no need of that anymore | |
| 16:30:37 | stephenfin | (y) | |
| 16:31:38 | jaypipes | stephenfin: I don't understand that last statement... | |
| 16:31:47 | jaypipes | stephenfin: why would we be removing that ability? | |
| 16:32:03 | cfriesen | jaypipes: if we assume that the emulator thread work scales with the number of vcpus, then we don't need to account for it separately, it's already in cpu_allocation_ratio. This would simplify things since you wouldn't need to ask for an extra VCPU from placement. | |
| 16:32:21 | stephenfin | jaypipes: Because you can't account for the extra per-guest CPU | |
| 16:32:24 | cfriesen | jaypipes: we're saying we'd remove the code that allocates a whole extra host CPU to run the emulator threads | |
| 16:32:36 | jaypipes | cfriesen: that's exactly *not* what I want. | |
| 16:32:36 | stephenfin | what cfriesen says | |
| 16:33:03 | cfriesen | jaypipes: sorry, which conversation thread is that for? | |
| 16:33:04 | cfriesen | :) | |
| 16:33:08 | jaypipes | cfriesen: *something* needs to account for CPU resources consumed by emulator threads. right now, nothing is accounted for in placement. | |
| 16:33:43 | cfriesen | jaypipes: there are two possibilities. 1) emulator thread work scales with the number of guests. 2) emulator thread work scales with the number of vcpus | |
| 16:34:06 | stephenfin | jaypipes: Does it though? I mean, if those threads are running on cores meant for 'shared' workloads, do we care? | |
| 16:34:40 | cfriesen | jaypipes: if 1 is true, then yes, we need to account for the work done by the emulator threads | |
| 16:34:41 | stephenfin | After all, we don't account for the overhead of emulator threads _without_ the policy applied. We could just ignore it | |
| 16:34:44 | openstackgerrit | Matt Riedemann proposed openstack/osc-placement master: Resolve nits from I552688b9ee32b719a576a7a9ed5e4d5aa31d7b3f https://review.openstack.org/537971 | |
| 16:34:57 | jaypipes | stephenfin: what's the difference between an emulator thread running on a shared CPU and a guest vCPU thread running on a shared CPU, then? | |
| 16:35:15 | stephenfin | You charge for the latter | |
| 16:35:17 | jaypipes | stephenfin: we account for the latter, not the formter. | |
| 16:35:25 | cfriesen | jaypipes: if 2 is true, then the work done by the emulator thread is "pooled" with the work done by the vCPU threads and it's all accounted for in cpu_allocation_ratio | |
| 16:35:31 | jaypipes | stephenfin: but that's my point... we're not accounting for the former when we should be. | |
| 16:36:04 | stephenfin | Also, the potential cycles consumed by emulator thread <<< guest vCPU thread | |
| 16:36:19 | jaypipes | stephenfin: except during live migration? | |
| 16:36:28 | stephenfin | good point | |
| 16:36:47 | jaypipes | my point being, we're currently not accounting for any of this stuff. | |
| 16:37:04 | cfriesen | jaypipes: stephenfin: I think the interesting case would be if you have a bunch of "dedicated" guest cpus, and the emulator thread running on the "shared" host cpu pool | |
| 16:37:22 | cfriesen | because in that case we really don't account for the emulator thread work currently | |
| 16:37:41 | mriedem | johnthetubaguy: if you're still around, https://review.openstack.org/#/c/520248/ | |
| 16:37:52 | stephenfin | cfriesen: I thought we did. sahid extended some 'overhead' function to do that | |
| 16:37:59 | stephenfin | *currently did | |
| 16:38:11 | johnthetubaguy | mriedem: been meaning to catch you about that | |
| 16:38:19 | cfriesen | stephenfin: right now the emulator threads for an instance run on a dedicated host cpu | |
| 16:38:23 | cfriesen | and we do account for that | |
| 16:38:33 | mriedem | stephenfin: he did https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L804 | |
| 16:38:35 | cfriesen | (if "isolate" is enable) | |
| 16:38:49 | stephenfin | cfriesen: Ah, ok. I missed that nuance | |
| 16:39:00 | cfriesen | stephenfin: but we're talking about running the emulator threads on the "shared" host cpus now | |
| 16:39:03 | mriedem | note that we don't account for overhead in placement at all | |
| 16:39:08 | mriedem | since it's per-compute, and done on the compute | |
| 16:39:24 | jaypipes | cfriesen: well, *all* emulator threads run on the same dedicated CPU? or *each* emulator thread runs on a dedicated host CPU? | |
| 16:39:35 | cfriesen | jaypipes: all on the same | |
| 16:39:39 | openstackgerrit | sahid proposed openstack/nova-specs master: virt: allow instances to be booted with trusted VFs https://review.openstack.org/485522 | |
| 16:39:49 | jaypipes | cfriesen: k | |
| 16:39:55 | stephenfin | cfriesen: At the moment? | |
| 16:40:03 | jaypipes | mriedem: this is a different overhead... | |
| 16:40:07 | johnthetubaguy | mriedem: ah https://specs.openstack.org/openstack/nova-specs/specs/queens/implemented/remove-configurable-hide-server-address-feature.html would fix that | |
| 16:40:13 | cfriesen | jaypipes: but each instance with "isolate" enabled gets a separate host CPU | |
| 16:40:25 | jaypipes | cfriesen: that's what I just asked you... | |
| 16:40:43 | stephenfin | jaypipes: A guest can have multiple emulator threads | |
| 16:40:51 | cfriesen | jaypipes: all the emulator threads for a given instance run on a single dedicated host CPU | |
| 16:40:52 | jaypipes | cfriesen: ok, so it's each instance gets a separate dedicated host CPU for its emulator thread if emulator_threads_policy=isolate. | |
| 16:40:53 | stephenfin | so your question was confusing :) | |
| 16:41:07 | stephenfin | jaypipes: *for its emulator threads | |
| 16:41:13 | stephenfin | (plural) | |
| 16:41:16 | jaypipes | stephenfin: we don't currently support >1 emulator thread. | |
| 16:41:29 | cfriesen | jaypipes: qemu has multiple emulator threads, so we do | |
| 16:41:34 | mriedem | johnthetubaguy: yeah mentioned that in reply to the patch | |
| 16:41:37 | stephenfin | indeed | |
| 16:41:38 | cfriesen | jaypipes: but they're all pinned the same | |
| 16:41:48 | jaypipes | cfriesen: but *we* (i.e. nova libvirt virt driver) never sets >1 right? | |
| 16:41:53 | johnthetubaguy | mriedem: yeah, that is where I got that from | |
| 16:42:03 | cfriesen | jaypipes: it's not up to libvirt, I don't think. it's internal to qemu. | |
| 16:42:09 | stephenfin | this is where sahid should butt in | |
| 16:42:47 | cfriesen | jaypipes: the existance of multiple threads is an implementation detail that is unimportant to the present discussion. | |
| 16:43:00 | sahid | jaypipes: by default when using cpu_policy:dedicated we pin the emulatore threads to the set of pCPUs dedicated for guest | |
| 16:43:02 | johnthetubaguy | mriedem: I think I am about to +W this given we hide it during building, which was my main concern, that seem reasonable? | |
| 16:43:03 | sahid | if that is your question | |
| 16:43:15 | mriedem | johnthetubaguy: i think you +Wing my patch is certainly reasonable yes | |
| 16:43:19 | cfriesen | jaypipes: libvirt will only let you set a single emulator thread affinity for a given domain | |
| 16:43:20 | mriedem | thank you for being british | |
| 16:43:25 | johnthetubaguy | mriedem: heh | |
| 16:43:46 | cfriesen | jaypipes: you *can* set individual IO thread affinities, but we don't muck with those at all at the moment | |
| 16:44:04 | stephenfin | cfriesen: Agreed - it's not important. Sorry for bogging us down there | |
| 16:44:23 | stephenfin | jaypipes: So yeah, one guest = one extra dedicated CPU | |
| 16:44:27 | jaypipes | HOW MANY FRIGGIN CPU RESOURCES IS THIS GUEST CONSUMING? <-- why is it so hard to answer this damn question. | |
| 16:45:08 | cfriesen | jaypipes: with qemu you could easily have a dozen or more host threads for an instance with a single vCPU | |
| 16:45:08 | stephenfin | without emulator and CPU thread policies, N | |
| 16:45:28 | stephenfin | with emulator thread policy but no CPU thread policy, N + 1 | |
| 16:45:29 | jaypipes | cfriesen: all I care about is the resource accounting. | |
| 16:45:50 | stephenfin | with emulator thread policy and CPU thread policy == isolate, (N * M) + 1 | |
| 16:45:57 | stephenfin | where N = requested CPUs for guest | |
| 16:46:05 | stephenfin | and M is size of the sibling sets of the host | |
| 16:46:26 | stephenfin | typically 2 for x86 platforms (HyperThreading) | |
| 16:46:48 | jaypipes | cfriesen: I don't care how the instance knows which of its virtual processors are pinned to dedicated pCPUs. I don't care about which OS threads are used for emulator processing. All I care about is **how many CPU resources** is the guest consuming. | |
| 16:47:50 | cfriesen | jaypipes: stephenfin's comments are basically valid, I think. The reason why it's slippery is that the amount of work done by the emulator thread is usually quite small, except for exceptions like live migration. | |
| 16:47:58 | jaypipes | cfriesen: if there isn't a way to calculate that simple resource accounting question, then something is completely f**ked about all of this. | |
| 16:48:45 | sean-k-mooney | jaypipes: well as stephenfin said its for emulator thread policy and CPU thread policy == isolate, (N * M) + 1 but N and M depend on the compute host that is selected | |
| 16:49:05 | jaypipes | cfriesen: furthermore, if there isn't a simple way to do resource accounting, all of this smells like over-engineering to me. | |
| 16:49:20 | sean-k-mooney | well actully N is the numer of guest cpus but M is host dependent | |
| 16:49:25 | jaypipes | ugh | |
| 16:50:18 | stephenfin | jaypipes: It's totally hardware specific and we don't want to leak that level of detail to the user | |
| 16:50:19 | sean-k-mooney | jaypipes: to your privous point however if we have a PCPU resouce and VCPU resouce then we can get rid fo the host depency | |
| 16:50:56 | cfriesen | jaypipes: currently with dedicated cpus it's deterministic because the emulator threads get a whole separate host CPU | |
| 16:50:56 | sean-k-mooney | actully no we cant. we would need to get teh host info to the scheduer filter still | |
| 16:51:03 | jaypipes | stephenfin: I *also* do not want to leak *any* of this crap to the end user. My whole spec is intended to convey a simple to understand concept about consumable CPU resources. | |
| 16:51:49 | jaypipes | stephenfin: i.e. the user wants a dedicated CPU, they ask for that. if they want 4 shared CPUs, they ask for that. | |
| 16:52:04 | cfriesen | jaypipes: If you set the CPU threading policy to "ISOLATE", then the number of host CPUs actually consumed will depend on whether the host has hyperthreading enabled or not. | |
| 16:52:22 | jaypipes | stephenfin: the problem is that for some reason, everyone wants to complicate the situation endlessly with hardware-specific doodads that nobody outside of Intel and a couple folks at Red Hat even understand. | |
| 16:53:05 | jaypipes | stephenfin: sorry if I'm frustrated, but by God, why is this stuff so complicated? | |