Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-03-22
16:55:35 sean-k-mooney stephenfin: well we could replace isolate with avoid. whic would meen we just make sure you land on a host that has no hyperthreads so you get teh same performance
16:55:40 cfriesen jaypipes: for the simple case I think it should be possible to make it simple.
16:55:47 sean-k-mooney that could be modled as a trait
16:55:55 stephenfin jaypipes: and it's understandably frustrating but it's telcos that are the main driver of this stuff. We're just enabling it
16:56:07 jaypipes sahid: and by "I", I mean "this spec doesn't care about the pinning". Not that the virt driver doesn't care. Just that the placement service doesn't care or know at all which guest CPU is pinned to which host CPU.
16:56:42 sahid jaypipes: i think we care about the pinnig, because the end user will not see the CPU that will be dedicated for running emultreads
16:56:52 cfriesen sean-k-mooney: then you're back to compute-node level granularity and may as well use aggregates
16:57:26 stephenfin aye, what sahid said is the main thing bothering me. The user isn't getting a CPU. They're getting their emulator threads offloaded
16:57:26 jaypipes stephenfin: telcos are *not* demanding an utterly incomprehensible way of configuring workloads. they are demanding that there high performance workloads be set up in a way that takes advantage of the hardware as much as possible.
16:57:27 dansmith mriedem: sorry I keep missing those comments.. unintentional
16:57:28 sean-k-mooney cfriesen: ya i know...
16:58:16 jaypipes stephenfin: trust me, nobody in Verizon HQ Planning likes trying to understand these CONF and extra spec options. In fact, they pretty much have to rely on RH's triple-o people to configure stuff for them because nobody can understand any of it.
16:58:25 cfriesen stephenphin: agreed. but *internally* we could model that by bumping the VCPU resource count (assuming we ran the emulator thread on the "shared" host cpus).
16:59:24 cfriesen stephenfin: this would basically be giving a rough estimate that the emulator work is roughly equal to a "shared" vcpu
16:59:29 jaypipes cfriesen: isn't that precisely what my spec is proposing? bumping the # of requested VCPU resources to account for emulator threads?
16:59:36 sean-k-mooney jaypipes: true but onap, ecomp osm and open mano all use these things today so we cant break existing users. well we can but we need to know what we are breaking
16:59:53 johnthetubaguy jaypipes: I still think we should have a guide that says "make me go predictably fast and isolated" and another that says "let me go as dense with the packing as I can to get max utilisation"
17:00:10 jaypipes sean-k-mooney: ONAP and eCOMP both go around nova and inventory the hosts entirely externally of nova. :(
17:00:10 cfriesen jaypipes: the difference is that I don't want the extra "VCPU" resource to be specified in the flavor. I want nova to add it if the flavor asks for emulator threads to be isolated.
17:00:38 jaypipes cfriesen: what is the difference?
17:00:53 jaypipes johnthetubaguy: totes agree.
17:00:55 sean-k-mooney jaypipes: ya i hate that too but they use the flavour extra specs also for things like emulator threads and cpu pinning
17:01:09 jaypipes johnthetubaguy: clearly you've never read the RH NFV OpenStack tuning guide.
17:01:17 cfriesen jaypipes: clarity to the user. If I look at a flavor that specifies 5 cpus, but then the resource in the extra specs asks for 6 cpus, I'm going to think there's a mistake
17:01:33 johnthetubaguy jaypipes: nope, should I?
17:01:59 jaypipes johnthetubaguy: no
17:02:02 sahid cfriesen: yes that is why the additional CPU used when using policy=isolate is not take into account in quota
17:02:09 johnthetubaguy jaypipes: cool, gtk
17:02:16 sean-k-mooney cfriesen: that depends. if the resouces asks for 6 cpus but 5 for the vm the the vms sees and 1 for emulator thread that it does not i think its ok
17:02:19 stephenfin jaypipes: Heh, yeah. I'm not saying it's _not_ all crap as it is (quite the opposite, tbh) but I just don't see yet how we can abstract this kind of stuff better than we do
17:02:26 jaypipes sahid: and that is a resource accounting problem, no?
17:02:32 sahid we ask for 4 dedicated pCPUs but actually 5 are dedicated
17:02:34 stephenfin I'm just seeing "if we do this, stuff that does work at the moment breaks"
17:02:58 johnthetubaguy stephenfin: what about having those two options I just mentioned?
17:03:06 sahid jaypipes: it is yes, but saying I want 5 vCPUs and the user only have 4vCPU is also a mistake
17:03:16 sean-k-mooney cfriesen: we accounted for the extra cpu for the emulator for in placement and the guest still got what it expected
17:03:46 stephenfin johnthetubaguy: The problem is, what VZW wants is different from what E/// want
17:03:48 openstackgerrit Merged openstack/osc-placement master: Do not depend on jenkins user in devstack gate https://review.openstack.org/552476
17:03:58 sahid we don't have any unit in our resource management to take that extra CPU into account
17:04:14 cfriesen sean-k-mooney: I want to distinguish between what's in the flavor and what nova asks from placement. If the flavor says "4 VCPU and I want isolated emulator threads" then I'm fine with nova asking for 5 VCPU.
17:04:30 cfriesen sean-k-mooney: but I don't want the flavor to have to ask for 5 VCPU
17:04:41 stephenfin We could also stop treating emulator threads as a VCPU and add another resource class to that
17:04:44 stephenfin *for that
17:04:49 sean-k-mooney sahid: again it depends if the flavour vcpu field is set to 4 the that means the guest will see 4 cpus. if the resouce:VCPUS is 5 we will claim 5 cpus in placemnt 4 of which would be used by the guest and on that could be used for the emulator
17:04:50 jaypipes sahid: so perhaps the way forward is indeed to have a completely separate CPU_SHARED and CPU_DEDICATED resource class, separte from VCPU. VCPU would represent guest vCPU threads that are using *shared* host CPUs. CPU_SHARED would represent emulator threads (and anything else not guest-related) that are on shared host CPUs, and CPU_DEDICATED would be the amount of dedicated CPUs.
17:04:59 stephenfin (for the emulator_threads_policy issue)
17:05:01 cfriesen stephenfin: just to simplify things. :)
17:05:01 johnthetubaguy stephenfin: I guess I am trying to suggest we document the two extreme cases really well, then document a few ways you might want to dial back in either direction from the extreme?
17:05:30 stephenfin So CPU_SHARED, CPU_DEDICATED, and CPU_EMULATOR_THREADS
17:05:55 jaypipes stephenfin: CPU_EMULATOR_THREADS is not a consumable resource, though...
17:06:05 jaypipes stephenfin: CPU_SHARED is the consumable resource that emulator threads consume.
17:06:08 cfriesen stephenfin: fundamentally right now, emulator threads run on the same host CPUs as the guest CPUs do by default.
17:06:15 sahid jaypipes: yes basically i thought that was you idea since the beginning :)
17:06:39 stephenfin jaypipes: It could be though, if we added the 'emulator_pin_set' list to nova.conf that sahid suggests
17:06:49 dansmith sahid: I left you minor comments in that spec
17:07:10 sahid thanks dansmith i will address them
17:07:13 cfriesen stephenfin: I don't like the idea of having a separate dedicated pool just for emulator threads when the "shared" pool is perfectly fine
17:07:29 jaypipes stephenfin: you are conflating two different things. one is allocation of resources. the other is assignment of emulator threads to a particular set of host CPUs.
17:08:20 cfriesen I think accounting for the emulator threads as a VCPU (specifically if all the cpus in a guest are "dedicated") is a reasonable approximation
17:08:38 cfriesen If any of the cpus in a guest are "shared", then we can run the emulator threads with them.
17:09:00 jaypipes omg, chris you're killing me.
17:09:08 stephenfin jaypipes: Am I? We're saying for the resources for CPU_DEDICATED will come from CONF.cpu_dedicated_set. I'm saying the resources for CPU_EMULATOR_THREADS could come from CONF.cpu_emulator_thread_set
17:09:32 sean-k-mooney jaypipes: can i sugges we create an etherpad and start listing exampels and then clearly state what each means in terms of host,guest and placement allocatiosn
17:09:40 cfriesen jaypipes: ah, forget it. just account for the emulator threads as a VCPU.
17:09:41 stephenfin Then, as a guest, I want N CPU_EMULATOR_THREADS
17:10:08 stephenfin sean-k-mooney: Not a bad idea
17:10:12 jaypipes stephenfin: but according to cfriesen, you can't request that.
17:10:23 jaypipes stephenfin: you can't say "I want 3 emulator threads".
17:10:48 bauzas jaypipes: stephenfin: I'm done with my meeting
17:10:54 bauzas can we jump into a hangout ?
17:10:58 jaypipes bauzas: uhm,....
17:11:00 cfriesen stephenfin: yeah, there's just some random number of emulator threads that are all pinned the same.
17:11:05 jaypipes bauzas: you've missed quite a bit.
17:11:24 bauzas jaypipes: yeah, that's what I just see...
17:12:09 stephenfin Riight. OK, forget that
17:12:10 sean-k-mooney maybe this would help https://libvirt.org/formatdomain.html#elementsCPUTuning
17:12:24 sean-k-mooney and this https://libvirt.org/formatdomain.html#elementsIOThreadsAllocation
17:12:24 bauzas jaypipes: stephenfin: any possible tl;dr ?
17:12:25 jaypipes sean-k-mooney: that's the problem...
17:12:44 stephenfin You can tell libvirt what CPUs the emulator threads will be placed on. Currently we say to pin only to one dedicated CPU
17:12:52 jaypipes sean-k-mooney: I am not interested in the pinning questions. I am only concerned with how to do resource accounting.
17:13:02 stephenfin But in this new world, we'd be affining to the entire pool of shared CPUs
17:13:13 stephenfin That first bit was what I was mixing up
17:13:20 sean-k-mooney jaypipes: yes i know.
17:13:31 cfriesen jaypipes: if every guest CPU is "shared" then the emulator threads just float across the same set of host cpus. It's when you have "dedicated" guest CPUs that the question of how to account for the emulator thread work arises.
17:13:48 jaypipes cfriesen: ack, understood.
17:13:49 openstackgerrit Nguyen Hai proposed openstack/nova-specs master: Enhance nova-specs webpage and clean up repo https://review.openstack.org/551802
17:13:53 sean-k-mooney so looking at the docs you can only have 1 emulator thread. you can have multiple io thread and multiple vcpu trheads
17:14:13 cfriesen sean-k-mooney: don't we spawn another emulator thread when live migrating?
17:14:17 stephenfin bauzas: We're enraging jaypipes to the point that I fear for his pugs' lives
17:14:23 jaypipes cfriesen: but it's not like that is something dynamically calculated, right? I mean, a guest either has dedicated CPUs or it doesn.t'
17:14:26 jaypipes https://tenor.com/view/angry-panda-mascot-mad-gif-3456638
17:14:29 sean-k-mooney cfriesen: i dont know but in that case to we care
17:14:38 cfriesen jaypipes: correct.
17:14:41 bauzas stephenfin: hold my beer
17:14:45 sean-k-mooney cfriesen: its live migrating all sla are out the window at that point
17:14:53 stephenfin bauzas: Also, we're trying to figure out how we map things like emulator threads policy and the ISOLATE CPU threads policy features to placement
17:15:00 bauzas k
17:15:11 jaypipes cfriesen: so it would be a request for, say resources:CPU_DEDICATED=4 (the vCPU threads) and resources:CPU_SHARED=1 (the emulator thread)
17:15:50 openstackgerrit Nguyen Hai proposed openstack/nova-specs master: Enhance nova-specs webpage and clean up repo https://review.openstack.org/551802

Earlier   Later