Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-05-22
16:00:20 mriedem the problem is no one is defining what 'reschedule' means here
16:00:26 jaypipes dansmith: I thought you were cool with getting rid of retries entirely?
16:00:33 dansmith lol
16:00:34 mriedem i think what jaypipes is talking about is calling select_destinations again
16:00:37 bauzas that's at least a bit unfortunate I spent my week on something I thought we had a consensus :)
16:00:50 bauzas without doing any reviews (sorry folks!)
16:01:00 mriedem i think dansmith is talking about deleting failed allocations and looping to the next host in our list, which is not the same
16:01:13 mriedem which we could do in conductor,
16:01:19 mriedem if the scheduler gives a list from select_destinations
16:01:30 bauzas mriedem: I implemented that in https://review.openstack.org/#/c/460455/15/nova/conductor/manager.py
16:01:31 jaypipes mriedem: no, I'm talking about the scheduler doing the "retry" of a claim/allocation.
16:01:33 mriedem "reschedule" is a broad ass term
16:01:51 bauzas jaypipes: please see at http://lists.openstack.org/pipermail/openstack-dev/2017-May/117039.html
16:01:58 dansmith how about a hangout at the bottom of the hour?
16:02:09 bauzas jaypipes: there are 4 reasons why I disagree with doing the allocation within the scheduler worker process
16:02:22 openstackgerrit Stephen Finucane proposed openstack/nova master: utils: Replace 'utf8' with oslo.utils equivalent https://review.openstack.org/466799
16:02:22 openstackgerrit Stephen Finucane proposed openstack/nova master: trivial: Remove dead code https://review.openstack.org/460662
16:02:29 mriedem dansmith: yeah need a hangout, so 1PM ET?
16:02:57 jaypipes dansmith, bauzas: I strongly think we should be calling placement's POST /allocations from the scheduler, not the conductor. Sorry if this surprises people.
16:03:03 dansmith mriedem: mriedem sure
16:03:26 dansmith jaypipes: yes that's very surprising.. we all had this conversation in boston and I was sure we had consensus
16:03:33 jaypipes I specifically remember saying "doing claims in the scheduler gets rid of the retry issues (and all associated "upcall shenanigans")
16:03:50 bauzas jaypipes: well, I do remember exatly operators nodding at dansmith's proposal of passing down a list of possible acceptable hosts
16:03:53 dansmith jaypipes: so, we also agreed we can't fully get rid of retries right away, which means conductor _will_ have to do allocation calls as well
16:03:56 dansmith right
16:03:57 mriedem note we've also been saying "claims in the scheduler*"
16:04:16 dansmith jaypipes: I'm opposed to doing it in scheduler first and conductor second.. I think that's a mess
16:04:23 melwitt mriedem: thanks. short answer is, you're right. replying on the review now
16:04:35 jaypipes dansmith: I see no reason to "do it in the conductor second".
16:04:45 mriedem melwitt: ok - so basically wtf for how this behaves, but it's not backward incompatible so ok
16:04:52 jaypipes dansmith: if the scheduler does the claim, there is no reason for anything else to do the claim.
16:05:00 dansmith jaypipes: reschedule
16:05:06 mriedem jaypipes: retries from the computes go up to the cell conductor
16:05:10 jaypipes dansmith: we should *not* support reschedule. period.
16:05:16 dansmith jaypipes: there was almost zero support in the room for eliminating reschedules
16:05:30 bauzas I thought we had that discussion in Boston :/
16:05:33 dansmith jaypipes: which was the whole point of the "send extra hosts down to the cell" discussion, which you were definitely there for
16:05:36 dansmith bauzas: we did
16:05:36 jaypipes dansmith: yes there was when I said "95% of the causes of reschedules currently will be eliminated"
16:05:44 dansmith *facepalm*
16:06:03 melwitt mriedem: yeah ... I'm trying to make all of the formats of count as dict consistent. before, I had some dict and some int so that only things that could have project and user would be dict, but people found that more confusing
16:06:05 bauzas anyway, I'm litterally 24 hours away of taking a long break
16:06:42 jaypipes dansmith: perhaps this is the session I had to leave early (I was standing in the back and right before I left I said "recschedules should not occur and the reason for them will be gone when claining in the scheduler"?
16:06:45 bauzas I'm trying hard to make sure we're all on the same page before I leave
16:06:58 dansmith jaypipes: bauzas: mriedem: I'm good for a hangout any time starting now, whenever you want, until 11:30 PDT
16:07:08 bauzas I can
16:07:09 jaypipes I can do a hangout whenever.
16:07:13 mriedem i can also do now
16:07:17 jaypipes ++
16:07:23 dansmith jaypipes: yeah, you left, but I thought you were still around for the consensus on reschedules.. however you _were_ there for the scheduling claims session where we talked about the extra hosts thing
16:07:30 melwitt eliminating reschedules would make things really suck for Ironic users in the absence of a porcelain API that does retries
16:07:47 jaypipes melwitt: you mean "standalone Ironic".
16:07:53 melwitt no, Ironic + Nova
16:08:15 mriedem plus i think numa came up
16:08:20 dansmith I'll get a link, just a sec
16:08:22 mriedem in the claims in the scheduler session
16:08:40 jaypipes mriedem: NUMA is handled in the existing NUMAtopologyFilter.
16:08:57 dansmith https://hangouts.google.com/call/e6jtzndusjf6rezqvwdatascvuu
16:09:12 bauzas joining
16:09:16 jaypipes mriedem: claims would occur after the NUMATopologyFilter selected a host. We could have the NUMAtopology that was selected (and currently thrown away by the filter) written to the compute_nodes table.
16:10:26 bauzas dansmith: you're not there in the hangout
16:11:15 jaypipes melwitt: honestly, I don't really care about use cases where an IPMI boot failure results in an ERROR state for an instance
16:12:30 sajeesh ping sdague
16:14:37 sdague sajeesh: what's up?
16:14:58 sajeesh ruleset looks like, so it can be turned into a worked example
16:14:58 sajeesh Hi sdague ..you have written in the mail Get someone from the CERN team to write down what their ideal
16:15:12 sdague sajeesh: yes
16:15:20 sajeesh by ruleset do you mean all use cases ?
16:15:51 sdague not use cases, how they think they would want hierarchical quotas to work
16:16:06 melwitt jaypipes: :\ I don't know the particulars on what is different about scheduling onto baremetal directly, but when you have enough nodes, enough of them could be flaky and if we fail and go to error, we're not absorbing any of that for the end user
16:16:06 sajeesh ok
16:17:03 sdague sajeesh: because the use cases here don't build the hierarchical model all that well, most people set a bunch of use cases together that actually are conflicting
16:17:33 sdague so, I want the model algorithm to be proposed by groups that are interested, then we'll work it through
16:18:00 sajeesh sdague:+1
16:21:44 ayoung Since we are rewriting how you guys do policy enforcement, maybe y'all should take a look: https://review.openstack.org/#/c/384148/
16:22:50 ayoung alaski, it has changed a bit since you last looked at it...in October
16:25:23 cdent jaypipes are you guys still on the hangout?
16:25:30 jaypipes cdent: yes
16:25:37 sdague ayoung: so, my concern with is_admin on project is that now if you give a user authority to add other users to a project, they also get to see the physical topology of your cloud, and perform live migrations. Because while is_admin is no longer fully global, it isn't service scoped, right?
16:26:41 alaski ayoung: I'm no longer actively involved in openstack and likely won't get a chance to look too closely at that change.
16:26:54 ayoung alaski, understood
16:27:06 ayoung sdague, let me parse what you are asking.
16:28:19 ayoung sdague, "my concern with" == "the reason I don't like this approach." " if you give a user authority to add other users to a project" should read " if you give a user authority to add other users to the admin project"
16:28:45 ayoung sdague, is that what you mean?
16:28:52 sdague ayoung: well, I'm trying to understand based on what's there
16:29:06 ayoung sdague, ok...so let me try to explain what we are doing here.
16:29:17 sdague so, what does the flow look like from openstack cli calls
16:29:27 ayoung We added a config option in Keystone that indicates one project is an admin project.
16:29:28 sdague for what an operator would be doing, and why
16:30:09 ayoung If that value is set, if a token is scoped to the admin project, the oslo-context will contain the value is_admin_project=True
16:30:47 ayoung the assumption is that, for operations that should be scoped to a proejct, we still need to honor the admin role, but that should be distinct from global operations that need the admin role
16:31:20 ayoung for global operations, such as adding a hypervisor, you limit access to tokens with is_admin_project=True
16:31:28 ayoung it is not scoped per service
16:31:46 ayoung you could do per-service via RBAC, but you would still want to make use of this mechanism
16:32:11 ayoung if you wanted a computeadmin role separate from admin, you could do that, but it would require custom policy
16:32:43 ayoung but you would still want to enforce that the user has computeadmin on the admin_project, and not accept tokens scoped to other projects
16:32:49 ayoung sdague, does that make sense?
16:33:11 sdague ok, I guess I don't understand how it moves things forward. I think I need an example of openstack cli commands about "now this is in place, this is what a keystone admin would do to give bob the following permissions, and this is what he could do after"
16:33:29 sdague I need an outside in view of this to make it make sense to me
16:33:45 ayoung sdague, fair enough

Earlier   Later