Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-03-22
16:11:31 jaypipes cfriesen: I am attempting to create a new resource class that represents a physical dedicated CPU (sahid would rather this resource class be called CPU_DEDICATED insetad of PCPU) that can be used to inventory these kinds of resources (which are currently not tracked in the same way as other resources, much to my dismay)
16:11:51 efried s/';'/':'/
16:11:53 efried sigh
16:12:04 dansmith efried: ah okay, although that's also kindof already done, but yeah I see that now
16:12:06 sean-k-mooney jaypipes: yes, if we track the overhead sepreatly form the pcpus/vcpu resuces it could imply that the cpus the guest sees is the resouce:*CPU-overhead or we ask for resouce:*cpus + over head form placement
16:12:31 cfriesen jaypipes: the problem is if I have X "dedicated" guest vcpus, and I want to allocate X+1 physical cpus, and run the emulator threads on a physical CPU that isn't running any vCPU thread.
16:12:40 jaypipes sean-k-mooney: this has nothing to do with "what the guest sees". this is a resource accounting issue only.
16:12:42 efried dansmith: so let's move ahead with repeatable member_of. Human reading query string will interpret '&' as 'AND', as it should be.
16:12:42 coreycb mriedem: special sauce indeeed! my bad.
16:12:47 mriedem nice
16:13:30 cdent wait, efried, did you just make a new different decision? I thought we had gone the other way ten minutes ago?
16:13:39 efried vay
16:13:40 cfriesen jaypipes: I'd be fine with CPU_DEDICATED. Would we also have CPU_SHARED instead of VCPU?
16:14:01 efried cdent: Where's your purple thingy?
16:14:04 jaypipes sean-k-mooney: this is only about answering the question "how many dedicated CPUs are left on the host (or on a specific NUMA node)?" and "how many shared CPUs are left on the host (or specific NUMA node)"
16:14:06 sean-k-mooney jaypipes: well it could what im asking is the resouces we counting are track soly by the vaules of resources:PCPU and resources:VCPU and that the overhead extra spec dont effect how we claim in placement
16:14:19 cdent efried: do p!help
16:14:22 efried p!help
16:14:26 jaypipes cfriesen: no, we can't change VCPU to CPU_SHARED, unfortunately.
16:14:29 cdent p!help
16:14:46 efried p!log
16:14:53 jaypipes cfriesen: which is why I wanted PCPU to be the class name for the dedicated CPU resource, since it matched VCPU
16:15:05 cdent you can do it in a privmsg too, so we don't have to see it. there you can leave off the p! prefic
16:15:18 efried cdent: http://p.anticdent.org/logs/openstack-nova?dated=2018-03-22%2016:05:18.169011#4hT5
16:16:37 efried cdent: Basically, we collectively on average seem to hate the available options for intra-param punctuation more than we hate deviating from the rest of the API by supporting repeatable for this one new case.
16:16:52 sean-k-mooney jaypipes: can you clear up somthing for me. if i have resources:VCPU=4 resources:PCPU=1 and hw:overhead_dedicated_set=0 do i claim 5 cpus in placement (4 shared and 1 dedicated)
16:16:53 cdent oh, that's not how intepreted that
16:16:58 openstackgerrit Eric Young proposed openstack/nova master: Support extending attached ScaleIO volumes https://review.openstack.org/554679
16:17:23 cdent I got what I said here about being least disruptive: [t 3cRQ]
16:17:23 purplerbot <cdent> efried: "fixable problem" was the immediate thing of "being able to accept duplicates on keys in general". I agree that addressign dan's problem in the least disruptive way is probably syntax in the param [2018-03-22 15:53:38.038412] [n 3cRQ]
16:17:35 jaypipes sean-k-mooney: no, you claim 4 VCPU and 1 PCPU resources.
16:17:47 jaypipes sean-k-mooney: there's no such things as a generic "CPU resource"
16:17:51 sean-k-mooney jaypipes: yes that is what i ment
16:18:23 sean-k-mooney i just wanted to make sure we dont claim 4VCPUs and 2 PCPUS
16:18:50 cfriesen sean-k-mooney: jaypipes: hw:overhead_dedicated_set doesn't make sense since we may want to run emulator threads separate from all vCPU threads (this is what "isolate" does today)
16:19:05 sean-k-mooney and in this case the guest recived 5 logical cpus and the emulator tread is pinned to the PCPU that was allocated to the vm
16:19:11 cdent efried: it probably doesn't matter _that_ much as long as documentation etc
16:19:17 cdent (whichever choice we make)
16:19:31 jaypipes cfriesen: ack. I'm reworking this in the spec after your feedback.
16:19:56 sean-k-mooney cfriesen when you say isolate which isolate do you mean
16:20:00 jaypipes cfriesen, sean-k-mooney: can there ever be >1 emulator thread, though?
16:20:00 efried cdent, dansmith, edleafe: Okay, let's make a call one way or another. And send an email. And propose a spec delta. And fix the code. And close the books.
16:20:08 cfriesen jaypipes: yes
16:20:12 edleafe cdent: efried: I didn't realize that we didn't accept multiple resources= params. But I do agree that changing that now is not a good use of time
16:20:15 jaypipes sean-k-mooney: he is referring to hw:emulator_threads_policy=isolate
16:20:22 cfriesen jaypipes: and there can be multiple I/O threads too
16:20:50 jaypipes cfriesen: and how does "hw:emulator_threads_policy=isolate" account for >1 emulator thread? that was the whole point of the overhead_dedicated_set being a pinset -- allowing >1 emulator thread to be specified.
16:20:56 sean-k-mooney jaypipes: ah ok hw:cpu_threads_policy=isolate is different just makeing sure
16:21:11 cfriesen jaypipes: all the emulator threads get affined to a single physical cpu
16:21:44 jaypipes cfriesen: that sounds like a libvirt-specific assumption that will likely change at any given time (as most of these things tend to do)
16:22:03 sahid jaypipes: oh so you can't change VCPU? in that case i'm ok with PCPU
16:22:18 cfriesen jaypipes: the emulator threads don't usually have a lot of work to do (the exception is during live migration I think). the goal of "hw:emulator_threads_policy=isolate" is to ensure that the minimal work they do doesn't interrupt the guest cpu threads
16:22:25 jaypipes sahid: I mean, we *could*, but that would be a seriously annoying code change ;)
16:23:00 mriedem melwitt: if you approve a blueprint like https://blueprints.launchpad.net/nova/+spec/hurrah-for-privsep-again you should set the 'series goal' to rocky so it shows up on the actual rocky blueprints page: https://blueprints.launchpad.net/nova/rocky
16:23:36 melwitt mriedem: sorry, I missed that it wasn't set. setting now
16:24:05 jaypipes cfriesen: so it never makes sense to have emulator threads on dedicated CPUs and guest vCPU threads on shared CPUs, right?
16:24:12 cfriesen jaypipes: yes, I suppose that's qemu-specific
16:24:34 cfriesen jaypipes: no, that wouldn't make sense.
16:24:37 sahid jaypipes: yes i'm agreed with cfriesen, that never make sense
16:25:04 jaypipes cfriesen: furthermore, from what I gather from your and sean-k-mooney's feedback, it also would never make sense to have emulator threads policy set to *anything* if the guest vCPU threads were *not* pinned to dedicated CPUs.
16:25:28 cfriesen jaypipes: yes
16:25:35 jaypipes k
16:26:16 cfriesen jaypipes: but it might make sense to have vcpu threads on dedicated cpu, and emulator threads on shared cpu (lumped together with other emulator threads and shared vcpu threads)
16:27:01 stephenfin yeah, that's one of the ideas that came up in reviews of sahid's spec ^
16:27:08 cfriesen jaypipes: sahid: I'm just not sure if we need to keep the "allocate a whole host cpu for my emulator threads because I'm super important and don't want other guests to impact my emulator threads"
16:27:42 sahid cfriesen: yes, you have asked me, but did not have responded. I think that is not necessary
16:27:44 stephenfin assuming we gain 'pcpu_pin_set' option (or whatever tetsuro's spec called for)
16:28:22 sahid we just want in some cases the emulthreads to run somewhere
16:28:26 jaypipes stephenfin: CONF.cpu_dedicated_set
16:28:32 cfriesen If we don't need that, then the simplest thing would be to enforce that we always have at least one shared host CPU, and if you ask for hw:cpu_threads_policy=isolate it runs there.
16:28:33 stephenfin that's the one
16:28:44 jaypipes stephenfin: is my proposal, along with CONF.cpu_shared_set being the other disjoint set.
16:28:59 stephenfin or maybe not, but the idea's the same
16:29:13 jaypipes cfriesen: you mean emulator_threads_policy, not cpu_threads_policy.
16:29:18 cfriesen bah, yes
16:29:20 sahid cfriesen: i see you point, we can't have cpu_shared_set empty
16:29:29 jaypipes yet another reason all this stuff is so confusing...
16:30:09 stephenfin So we'd completely remove the ability to have one core per guest for emulator threads (e.g. what we currently have)
16:30:19 stephenfin not that I'm against that. Just making sure I understand
16:30:29 sahid stephenfin: yes, no need of that anymore
16:30:29 cfriesen stephenfin: that's really an implementation detail currently, but yes
16:30:37 stephenfin (y)
16:31:38 jaypipes stephenfin: I don't understand that last statement...
16:31:47 jaypipes stephenfin: why would we be removing that ability?
16:32:03 cfriesen jaypipes: if we assume that the emulator thread work scales with the number of vcpus, then we don't need to account for it separately, it's already in cpu_allocation_ratio. This would simplify things since you wouldn't need to ask for an extra VCPU from placement.
16:32:21 stephenfin jaypipes: Because you can't account for the extra per-guest CPU
16:32:24 cfriesen jaypipes: we're saying we'd remove the code that allocates a whole extra host CPU to run the emulator threads
16:32:36 stephenfin what cfriesen says
16:32:36 jaypipes cfriesen: that's exactly *not* what I want.
16:33:03 cfriesen jaypipes: sorry, which conversation thread is that for?
16:33:04 cfriesen :)
16:33:08 jaypipes cfriesen: *something* needs to account for CPU resources consumed by emulator threads. right now, nothing is accounted for in placement.
16:33:43 cfriesen jaypipes: there are two possibilities. 1) emulator thread work scales with the number of guests. 2) emulator thread work scales with the number of vcpus
16:34:06 stephenfin jaypipes: Does it though? I mean, if those threads are running on cores meant for 'shared' workloads, do we care?
16:34:40 cfriesen jaypipes: if 1 is true, then yes, we need to account for the work done by the emulator threads
16:34:41 stephenfin After all, we don't account for the overhead of emulator threads _without_ the policy applied. We could just ignore it
16:34:44 openstackgerrit Matt Riedemann proposed openstack/osc-placement master: Resolve nits from I552688b9ee32b719a576a7a9ed5e4d5aa31d7b3f https://review.openstack.org/537971
16:34:57 jaypipes stephenfin: what's the difference between an emulator thread running on a shared CPU and a guest vCPU thread running on a shared CPU, then?
16:35:15 stephenfin You charge for the latter
16:35:17 jaypipes stephenfin: we account for the latter, not the formter.

Earlier   Later