| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-22 | |||
| 16:43:02 | johnthetubaguy | mriedem: I think I am about to +W this given we hide it during building, which was my main concern, that seem reasonable? | |
| 16:43:03 | sahid | if that is your question | |
| 16:43:15 | mriedem | johnthetubaguy: i think you +Wing my patch is certainly reasonable yes | |
| 16:43:19 | cfriesen | jaypipes: libvirt will only let you set a single emulator thread affinity for a given domain | |
| 16:43:20 | mriedem | thank you for being british | |
| 16:43:25 | johnthetubaguy | mriedem: heh | |
| 16:43:46 | cfriesen | jaypipes: you *can* set individual IO thread affinities, but we don't muck with those at all at the moment | |
| 16:44:04 | stephenfin | cfriesen: Agreed - it's not important. Sorry for bogging us down there | |
| 16:44:23 | stephenfin | jaypipes: So yeah, one guest = one extra dedicated CPU | |
| 16:44:27 | jaypipes | HOW MANY FRIGGIN CPU RESOURCES IS THIS GUEST CONSUMING? <-- why is it so hard to answer this damn question. | |
| 16:45:08 | cfriesen | jaypipes: with qemu you could easily have a dozen or more host threads for an instance with a single vCPU | |
| 16:45:08 | stephenfin | without emulator and CPU thread policies, N | |
| 16:45:28 | stephenfin | with emulator thread policy but no CPU thread policy, N + 1 | |
| 16:45:29 | jaypipes | cfriesen: all I care about is the resource accounting. | |
| 16:45:50 | stephenfin | with emulator thread policy and CPU thread policy == isolate, (N * M) + 1 | |
| 16:45:57 | stephenfin | where N = requested CPUs for guest | |
| 16:46:05 | stephenfin | and M is size of the sibling sets of the host | |
| 16:46:26 | stephenfin | typically 2 for x86 platforms (HyperThreading) | |
| 16:46:48 | jaypipes | cfriesen: I don't care how the instance knows which of its virtual processors are pinned to dedicated pCPUs. I don't care about which OS threads are used for emulator processing. All I care about is **how many CPU resources** is the guest consuming. | |
| 16:47:50 | cfriesen | jaypipes: stephenfin's comments are basically valid, I think. The reason why it's slippery is that the amount of work done by the emulator thread is usually quite small, except for exceptions like live migration. | |
| 16:47:58 | jaypipes | cfriesen: if there isn't a way to calculate that simple resource accounting question, then something is completely f**ked about all of this. | |
| 16:48:45 | sean-k-mooney | jaypipes: well as stephenfin said its for emulator thread policy and CPU thread policy == isolate, (N * M) + 1 but N and M depend on the compute host that is selected | |
| 16:49:05 | jaypipes | cfriesen: furthermore, if there isn't a simple way to do resource accounting, all of this smells like over-engineering to me. | |
| 16:49:20 | sean-k-mooney | well actully N is the numer of guest cpus but M is host dependent | |
| 16:49:25 | jaypipes | ugh | |
| 16:50:18 | stephenfin | jaypipes: It's totally hardware specific and we don't want to leak that level of detail to the user | |
| 16:50:19 | sean-k-mooney | jaypipes: to your privous point however if we have a PCPU resouce and VCPU resouce then we can get rid fo the host depency | |
| 16:50:56 | cfriesen | jaypipes: currently with dedicated cpus it's deterministic because the emulator threads get a whole separate host CPU | |
| 16:50:56 | sean-k-mooney | actully no we cant. we would need to get teh host info to the scheduer filter still | |
| 16:51:03 | jaypipes | stephenfin: I *also* do not want to leak *any* of this crap to the end user. My whole spec is intended to convey a simple to understand concept about consumable CPU resources. | |
| 16:51:49 | jaypipes | stephenfin: i.e. the user wants a dedicated CPU, they ask for that. if they want 4 shared CPUs, they ask for that. | |
| 16:52:04 | cfriesen | jaypipes: If you set the CPU threading policy to "ISOLATE", then the number of host CPUs actually consumed will depend on whether the host has hyperthreading enabled or not. | |
| 16:52:22 | jaypipes | stephenfin: the problem is that for some reason, everyone wants to complicate the situation endlessly with hardware-specific doodads that nobody outside of Intel and a couple folks at Red Hat even understand. | |
| 16:53:05 | jaypipes | stephenfin: sorry if I'm frustrated, but by God, why is this stuff so complicated? | |
| 16:53:06 | sahid | jaypipes: i think you are making it more complicated by mixing VCPU/PCPU also that, two option overhed_dedicated/shared_set, we should keep all of that easy, you ask for PCPU or VCPU and you ask to isolate or not the overhead | |
| 16:53:29 | stephenfin | Yup ^ ISOLATE gives us an additional case, where a user ask for entire CPU including thread siblings | |
| 16:53:39 | sahid | one case that run the same as the guest pCPUs assigned and the other torun on CODN.cpu_shared_set | |
| 16:53:45 | jaypipes | sahid: how is that simpler than just having the user ask for a quantity of dedicated CPU and a quantity of shared CPU resources? | |
| 16:53:45 | cfriesen | jaypipes: realistically, it's complicated because people can use the complexity to get better performance. | |
| 16:53:46 | stephenfin | cores, in 'lscpu' terminology | |
| 16:53:49 | sean-k-mooney | stephenfin: ya isolate is teh real issue here | |
| 16:54:00 | sahid | because how are you doing the pinning | |
| 16:54:37 | stephenfin | We could kill the 'ISOLATE' feature, but then folks, me included, are going to wonder what we're gaining for the people that use this | |
| 16:54:45 | jaypipes | sahid: again, I don't care about the pinning. all I care about here is the resource accounting. how *many* of dedicated CPU and shared CPU resources are being consumed on the host. | |
| 16:54:47 | sean-k-mooney | jaypipes: and jay ya i know this is hardware defined infrastruct at its best. sorry to bring up the edgecases | |
| 16:55:35 | sean-k-mooney | stephenfin: well we could replace isolate with avoid. whic would meen we just make sure you land on a host that has no hyperthreads so you get teh same performance | |
| 16:55:40 | cfriesen | jaypipes: for the simple case I think it should be possible to make it simple. | |
| 16:55:47 | sean-k-mooney | that could be modled as a trait | |
| 16:55:55 | stephenfin | jaypipes: and it's understandably frustrating but it's telcos that are the main driver of this stuff. We're just enabling it | |
| 16:56:07 | jaypipes | sahid: and by "I", I mean "this spec doesn't care about the pinning". Not that the virt driver doesn't care. Just that the placement service doesn't care or know at all which guest CPU is pinned to which host CPU. | |
| 16:56:42 | sahid | jaypipes: i think we care about the pinnig, because the end user will not see the CPU that will be dedicated for running emultreads | |
| 16:56:52 | cfriesen | sean-k-mooney: then you're back to compute-node level granularity and may as well use aggregates | |
| 16:57:26 | stephenfin | aye, what sahid said is the main thing bothering me. The user isn't getting a CPU. They're getting their emulator threads offloaded | |
| 16:57:26 | jaypipes | stephenfin: telcos are *not* demanding an utterly incomprehensible way of configuring workloads. they are demanding that there high performance workloads be set up in a way that takes advantage of the hardware as much as possible. | |
| 16:57:27 | dansmith | mriedem: sorry I keep missing those comments.. unintentional | |
| 16:57:28 | sean-k-mooney | cfriesen: ya i know... | |
| 16:58:16 | jaypipes | stephenfin: trust me, nobody in Verizon HQ Planning likes trying to understand these CONF and extra spec options. In fact, they pretty much have to rely on RH's triple-o people to configure stuff for them because nobody can understand any of it. | |
| 16:58:25 | cfriesen | stephenphin: agreed. but *internally* we could model that by bumping the VCPU resource count (assuming we ran the emulator thread on the "shared" host cpus). | |
| 16:59:24 | cfriesen | stephenfin: this would basically be giving a rough estimate that the emulator work is roughly equal to a "shared" vcpu | |
| 16:59:29 | jaypipes | cfriesen: isn't that precisely what my spec is proposing? bumping the # of requested VCPU resources to account for emulator threads? | |
| 16:59:36 | sean-k-mooney | jaypipes: true but onap, ecomp osm and open mano all use these things today so we cant break existing users. well we can but we need to know what we are breaking | |
| 16:59:53 | johnthetubaguy | jaypipes: I still think we should have a guide that says "make me go predictably fast and isolated" and another that says "let me go as dense with the packing as I can to get max utilisation" | |
| 17:00:10 | jaypipes | sean-k-mooney: ONAP and eCOMP both go around nova and inventory the hosts entirely externally of nova. :( | |
| 17:00:10 | cfriesen | jaypipes: the difference is that I don't want the extra "VCPU" resource to be specified in the flavor. I want nova to add it if the flavor asks for emulator threads to be isolated. | |
| 17:00:38 | jaypipes | cfriesen: what is the difference? | |
| 17:00:53 | jaypipes | johnthetubaguy: totes agree. | |
| 17:00:55 | sean-k-mooney | jaypipes: ya i hate that too but they use the flavour extra specs also for things like emulator threads and cpu pinning | |
| 17:01:09 | jaypipes | johnthetubaguy: clearly you've never read the RH NFV OpenStack tuning guide. | |
| 17:01:17 | cfriesen | jaypipes: clarity to the user. If I look at a flavor that specifies 5 cpus, but then the resource in the extra specs asks for 6 cpus, I'm going to think there's a mistake | |
| 17:01:33 | johnthetubaguy | jaypipes: nope, should I? | |
| 17:01:59 | jaypipes | johnthetubaguy: no | |
| 17:02:02 | sahid | cfriesen: yes that is why the additional CPU used when using policy=isolate is not take into account in quota | |
| 17:02:09 | johnthetubaguy | jaypipes: cool, gtk | |
| 17:02:16 | sean-k-mooney | cfriesen: that depends. if the resouces asks for 6 cpus but 5 for the vm the the vms sees and 1 for emulator thread that it does not i think its ok | |
| 17:02:19 | stephenfin | jaypipes: Heh, yeah. I'm not saying it's _not_ all crap as it is (quite the opposite, tbh) but I just don't see yet how we can abstract this kind of stuff better than we do | |
| 17:02:26 | jaypipes | sahid: and that is a resource accounting problem, no? | |
| 17:02:32 | sahid | we ask for 4 dedicated pCPUs but actually 5 are dedicated | |
| 17:02:34 | stephenfin | I'm just seeing "if we do this, stuff that does work at the moment breaks" | |
| 17:02:58 | johnthetubaguy | stephenfin: what about having those two options I just mentioned? | |
| 17:03:06 | sahid | jaypipes: it is yes, but saying I want 5 vCPUs and the user only have 4vCPU is also a mistake | |
| 17:03:16 | sean-k-mooney | cfriesen: we accounted for the extra cpu for the emulator for in placement and the guest still got what it expected | |
| 17:03:46 | stephenfin | johnthetubaguy: The problem is, what VZW wants is different from what E/// want | |
| 17:03:48 | openstackgerrit | Merged openstack/osc-placement master: Do not depend on jenkins user in devstack gate https://review.openstack.org/552476 | |
| 17:03:58 | sahid | we don't have any unit in our resource management to take that extra CPU into account | |
| 17:04:14 | cfriesen | sean-k-mooney: I want to distinguish between what's in the flavor and what nova asks from placement. If the flavor says "4 VCPU and I want isolated emulator threads" then I'm fine with nova asking for 5 VCPU. | |
| 17:04:30 | cfriesen | sean-k-mooney: but I don't want the flavor to have to ask for 5 VCPU | |
| 17:04:41 | stephenfin | We could also stop treating emulator threads as a VCPU and add another resource class to that | |
| 17:04:44 | stephenfin | *for that | |
| 17:04:49 | sean-k-mooney | sahid: again it depends if the flavour vcpu field is set to 4 the that means the guest will see 4 cpus. if the resouce:VCPUS is 5 we will claim 5 cpus in placemnt 4 of which would be used by the guest and on that could be used for the emulator | |
| 17:04:50 | jaypipes | sahid: so perhaps the way forward is indeed to have a completely separate CPU_SHARED and CPU_DEDICATED resource class, separte from VCPU. VCPU would represent guest vCPU threads that are using *shared* host CPUs. CPU_SHARED would represent emulator threads (and anything else not guest-related) that are on shared host CPUs, and CPU_DEDICATED would be the amount of dedicated CPUs. | |
| 17:04:59 | stephenfin | (for the emulator_threads_policy issue) | |
| 17:05:01 | cfriesen | stephenfin: just to simplify things. :) | |
| 17:05:01 | johnthetubaguy | stephenfin: I guess I am trying to suggest we document the two extreme cases really well, then document a few ways you might want to dial back in either direction from the extreme? | |
| 17:05:30 | stephenfin | So CPU_SHARED, CPU_DEDICATED, and CPU_EMULATOR_THREADS | |
| 17:05:55 | jaypipes | stephenfin: CPU_EMULATOR_THREADS is not a consumable resource, though... | |
| 17:06:05 | jaypipes | stephenfin: CPU_SHARED is the consumable resource that emulator threads consume. | |
| 17:06:08 | cfriesen | stephenfin: fundamentally right now, emulator threads run on the same host CPUs as the guest CPUs do by default. | |
| 17:06:15 | sahid | jaypipes: yes basically i thought that was you idea since the beginning :) | |
| 17:06:39 | stephenfin | jaypipes: It could be though, if we added the 'emulator_pin_set' list to nova.conf that sahid suggests | |
| 17:06:49 | dansmith | sahid: I left you minor comments in that spec | |