Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-03-22
16:36:28 stephenfin good point
16:36:47 jaypipes my point being, we're currently not accounting for any of this stuff.
16:37:04 cfriesen jaypipes: stephenfin: I think the interesting case would be if you have a bunch of "dedicated" guest cpus, and the emulator thread running on the "shared" host cpu pool
16:37:22 cfriesen because in that case we really don't account for the emulator thread work currently
16:37:41 mriedem johnthetubaguy: if you're still around, https://review.openstack.org/#/c/520248/
16:37:52 stephenfin cfriesen: I thought we did. sahid extended some 'overhead' function to do that
16:37:59 stephenfin *currently did
16:38:11 johnthetubaguy mriedem: been meaning to catch you about that
16:38:19 cfriesen stephenfin: right now the emulator threads for an instance run on a dedicated host cpu
16:38:23 cfriesen and we do account for that
16:38:33 mriedem stephenfin: he did https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L804
16:38:35 cfriesen (if "isolate" is enable)
16:38:49 stephenfin cfriesen: Ah, ok. I missed that nuance
16:39:00 cfriesen stephenfin: but we're talking about running the emulator threads on the "shared" host cpus now
16:39:03 mriedem note that we don't account for overhead in placement at all
16:39:08 mriedem since it's per-compute, and done on the compute
16:39:24 jaypipes cfriesen: well, *all* emulator threads run on the same dedicated CPU? or *each* emulator thread runs on a dedicated host CPU?
16:39:35 cfriesen jaypipes: all on the same
16:39:39 openstackgerrit sahid proposed openstack/nova-specs master: virt: allow instances to be booted with trusted VFs https://review.openstack.org/485522
16:39:49 jaypipes cfriesen: k
16:39:55 stephenfin cfriesen: At the moment?
16:40:03 jaypipes mriedem: this is a different overhead...
16:40:07 johnthetubaguy mriedem: ah https://specs.openstack.org/openstack/nova-specs/specs/queens/implemented/remove-configurable-hide-server-address-feature.html would fix that
16:40:13 cfriesen jaypipes: but each instance with "isolate" enabled gets a separate host CPU
16:40:25 jaypipes cfriesen: that's what I just asked you...
16:40:43 stephenfin jaypipes: A guest can have multiple emulator threads
16:40:51 cfriesen jaypipes: all the emulator threads for a given instance run on a single dedicated host CPU
16:40:52 jaypipes cfriesen: ok, so it's each instance gets a separate dedicated host CPU for its emulator thread if emulator_threads_policy=isolate.
16:40:53 stephenfin so your question was confusing :)
16:41:07 stephenfin jaypipes: *for its emulator threads
16:41:13 stephenfin (plural)
16:41:16 jaypipes stephenfin: we don't currently support >1 emulator thread.
16:41:29 cfriesen jaypipes: qemu has multiple emulator threads, so we do
16:41:34 mriedem johnthetubaguy: yeah mentioned that in reply to the patch
16:41:37 stephenfin indeed
16:41:38 cfriesen jaypipes: but they're all pinned the same
16:41:48 jaypipes cfriesen: but *we* (i.e. nova libvirt virt driver) never sets >1 right?
16:41:53 johnthetubaguy mriedem: yeah, that is where I got that from
16:42:03 cfriesen jaypipes: it's not up to libvirt, I don't think. it's internal to qemu.
16:42:09 stephenfin this is where sahid should butt in
16:42:47 cfriesen jaypipes: the existance of multiple threads is an implementation detail that is unimportant to the present discussion.
16:43:00 sahid jaypipes: by default when using cpu_policy:dedicated we pin the emulatore threads to the set of pCPUs dedicated for guest
16:43:02 johnthetubaguy mriedem: I think I am about to +W this given we hide it during building, which was my main concern, that seem reasonable?
16:43:03 sahid if that is your question
16:43:15 mriedem johnthetubaguy: i think you +Wing my patch is certainly reasonable yes
16:43:19 cfriesen jaypipes: libvirt will only let you set a single emulator thread affinity for a given domain
16:43:20 mriedem thank you for being british
16:43:25 johnthetubaguy mriedem: heh
16:43:46 cfriesen jaypipes: you *can* set individual IO thread affinities, but we don't muck with those at all at the moment
16:44:04 stephenfin cfriesen: Agreed - it's not important. Sorry for bogging us down there
16:44:23 stephenfin jaypipes: So yeah, one guest = one extra dedicated CPU
16:44:27 jaypipes HOW MANY FRIGGIN CPU RESOURCES IS THIS GUEST CONSUMING? <-- why is it so hard to answer this damn question.
16:45:08 cfriesen jaypipes: with qemu you could easily have a dozen or more host threads for an instance with a single vCPU
16:45:08 stephenfin without emulator and CPU thread policies, N
16:45:28 stephenfin with emulator thread policy but no CPU thread policy, N + 1
16:45:29 jaypipes cfriesen: all I care about is the resource accounting.
16:45:50 stephenfin with emulator thread policy and CPU thread policy == isolate, (N * M) + 1
16:45:57 stephenfin where N = requested CPUs for guest
16:46:05 stephenfin and M is size of the sibling sets of the host
16:46:26 stephenfin typically 2 for x86 platforms (HyperThreading)
16:46:48 jaypipes cfriesen: I don't care how the instance knows which of its virtual processors are pinned to dedicated pCPUs. I don't care about which OS threads are used for emulator processing. All I care about is **how many CPU resources** is the guest consuming.
16:47:50 cfriesen jaypipes: stephenfin's comments are basically valid, I think. The reason why it's slippery is that the amount of work done by the emulator thread is usually quite small, except for exceptions like live migration.
16:47:58 jaypipes cfriesen: if there isn't a way to calculate that simple resource accounting question, then something is completely f**ked about all of this.
16:48:45 sean-k-mooney jaypipes: well as stephenfin said its for emulator thread policy and CPU thread policy == isolate, (N * M) + 1 but N and M depend on the compute host that is selected
16:49:05 jaypipes cfriesen: furthermore, if there isn't a simple way to do resource accounting, all of this smells like over-engineering to me.
16:49:20 sean-k-mooney well actully N is the numer of guest cpus but M is host dependent
16:49:25 jaypipes ugh
16:50:18 stephenfin jaypipes: It's totally hardware specific and we don't want to leak that level of detail to the user
16:50:19 sean-k-mooney jaypipes: to your privous point however if we have a PCPU resouce and VCPU resouce then we can get rid fo the host depency
16:50:56 cfriesen jaypipes: currently with dedicated cpus it's deterministic because the emulator threads get a whole separate host CPU
16:50:56 sean-k-mooney actully no we cant. we would need to get teh host info to the scheduer filter still
16:51:03 jaypipes stephenfin: I *also* do not want to leak *any* of this crap to the end user. My whole spec is intended to convey a simple to understand concept about consumable CPU resources.
16:51:49 jaypipes stephenfin: i.e. the user wants a dedicated CPU, they ask for that. if they want 4 shared CPUs, they ask for that.
16:52:04 cfriesen jaypipes: If you set the CPU threading policy to "ISOLATE", then the number of host CPUs actually consumed will depend on whether the host has hyperthreading enabled or not.
16:52:22 jaypipes stephenfin: the problem is that for some reason, everyone wants to complicate the situation endlessly with hardware-specific doodads that nobody outside of Intel and a couple folks at Red Hat even understand.
16:53:05 jaypipes stephenfin: sorry if I'm frustrated, but by God, why is this stuff so complicated?
16:53:06 sahid jaypipes: i think you are making it more complicated by mixing VCPU/PCPU also that, two option overhed_dedicated/shared_set, we should keep all of that easy, you ask for PCPU or VCPU and you ask to isolate or not the overhead
16:53:29 stephenfin Yup ^ ISOLATE gives us an additional case, where a user ask for entire CPU including thread siblings
16:53:39 sahid one case that run the same as the guest pCPUs assigned and the other torun on CODN.cpu_shared_set
16:53:45 jaypipes sahid: how is that simpler than just having the user ask for a quantity of dedicated CPU and a quantity of shared CPU resources?
16:53:45 cfriesen jaypipes: realistically, it's complicated because people can use the complexity to get better performance.
16:53:46 stephenfin cores, in 'lscpu' terminology
16:53:49 sean-k-mooney stephenfin: ya isolate is teh real issue here
16:54:00 sahid because how are you doing the pinning
16:54:37 stephenfin We could kill the 'ISOLATE' feature, but then folks, me included, are going to wonder what we're gaining for the people that use this
16:54:45 jaypipes sahid: again, I don't care about the pinning. all I care about here is the resource accounting. how *many* of dedicated CPU and shared CPU resources are being consumed on the host.
16:54:47 sean-k-mooney jaypipes: and jay ya i know this is hardware defined infrastruct at its best. sorry to bring up the edgecases
16:55:35 sean-k-mooney stephenfin: well we could replace isolate with avoid. whic would meen we just make sure you land on a host that has no hyperthreads so you get teh same performance
16:55:40 cfriesen jaypipes: for the simple case I think it should be possible to make it simple.
16:55:47 sean-k-mooney that could be modled as a trait
16:55:55 stephenfin jaypipes: and it's understandably frustrating but it's telcos that are the main driver of this stuff. We're just enabling it
16:56:07 jaypipes sahid: and by "I", I mean "this spec doesn't care about the pinning". Not that the virt driver doesn't care. Just that the placement service doesn't care or know at all which guest CPU is pinned to which host CPU.
16:56:42 sahid jaypipes: i think we care about the pinnig, because the end user will not see the CPU that will be dedicated for running emultreads
16:56:52 cfriesen sean-k-mooney: then you're back to compute-node level granularity and may as well use aggregates
16:57:26 stephenfin aye, what sahid said is the main thing bothering me. The user isn't getting a CPU. They're getting their emulator threads offloaded
16:57:26 jaypipes stephenfin: telcos are *not* demanding an utterly incomprehensible way of configuring workloads. they are demanding that there high performance workloads be set up in a way that takes advantage of the hardware as much as possible.
16:57:27 dansmith mriedem: sorry I keep missing those comments.. unintentional
16:57:28 sean-k-mooney cfriesen: ya i know...
16:58:16 jaypipes stephenfin: trust me, nobody in Verizon HQ Planning likes trying to understand these CONF and extra spec options. In fact, they pretty much have to rely on RH's triple-o people to configure stuff for them because nobody can understand any of it.
16:58:25 cfriesen stephenphin: agreed. but *internally* we could model that by bumping the VCPU resource count (assuming we ran the emulator thread on the "shared" host cpus).

Earlier   Later