Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-05-22
15:57:43 bauzas because of the rescheduling issue
15:57:45 mriedem melwitt: random questions https://review.openstack.org/#/c/446239/7/nova/compute/api.py@5167
15:57:55 dansmith jaypipes: wat. really?
15:57:55 jaypipes bauzas: there are no rescheduling issues if we do the claims in the scheduler.
15:58:03 jaypipes dansmith: yes.
15:58:17 dansmith jaypipes: we totally said we were doing them in the conductor in boston :)
15:58:32 bauzas jaypipes: I explained why the scheduler sould be a wrong possibilty in some email
15:58:34 bauzas lemme find it
15:58:53 jaypipes dansmith, bauzas: reschedule issues go away if we do the claims in the scheduler.
15:59:02 dansmith jaypipes: eh?
15:59:16 bauzas jaypipes: http://lists.openstack.org/pipermail/openstack-dev/2017-May/117039.html
15:59:20 dansmith jaypipes: if we do it in the scheduler, then we end up with scheduler claiming first, then conductor claiming on reschedules
15:59:24 dansmith jaypipes: which I don't like
15:59:25 bauzas "There are multiple reasons why we chose to use conductor for that :"
15:59:30 jaypipes dansmith: you remember when I told operators that "95% or more of the reasons for retries go away when the scheduler does claims"?
15:59:51 jaypipes dansmith: there is no reason to do reschedules at all if we do claims in the scheduler.
15:59:52 dansmith jaypipes: sure, but we're talking about claiming during the scheduling phase
16:00:10 dansmith I can't believe we're having this conversation again :)
16:00:20 mriedem the problem is no one is defining what 'reschedule' means here
16:00:26 jaypipes dansmith: I thought you were cool with getting rid of retries entirely?
16:00:33 dansmith lol
16:00:34 mriedem i think what jaypipes is talking about is calling select_destinations again
16:00:37 bauzas that's at least a bit unfortunate I spent my week on something I thought we had a consensus :)
16:00:50 bauzas without doing any reviews (sorry folks!)
16:01:00 mriedem i think dansmith is talking about deleting failed allocations and looping to the next host in our list, which is not the same
16:01:13 mriedem which we could do in conductor,
16:01:19 mriedem if the scheduler gives a list from select_destinations
16:01:30 bauzas mriedem: I implemented that in https://review.openstack.org/#/c/460455/15/nova/conductor/manager.py
16:01:31 jaypipes mriedem: no, I'm talking about the scheduler doing the "retry" of a claim/allocation.
16:01:33 mriedem "reschedule" is a broad ass term
16:01:51 bauzas jaypipes: please see at http://lists.openstack.org/pipermail/openstack-dev/2017-May/117039.html
16:01:58 dansmith how about a hangout at the bottom of the hour?
16:02:09 bauzas jaypipes: there are 4 reasons why I disagree with doing the allocation within the scheduler worker process
16:02:22 openstackgerrit Stephen Finucane proposed openstack/nova master: trivial: Remove dead code https://review.openstack.org/460662
16:02:22 openstackgerrit Stephen Finucane proposed openstack/nova master: utils: Replace 'utf8' with oslo.utils equivalent https://review.openstack.org/466799
16:02:29 mriedem dansmith: yeah need a hangout, so 1PM ET?
16:02:57 jaypipes dansmith, bauzas: I strongly think we should be calling placement's POST /allocations from the scheduler, not the conductor. Sorry if this surprises people.
16:03:03 dansmith mriedem: mriedem sure
16:03:26 dansmith jaypipes: yes that's very surprising.. we all had this conversation in boston and I was sure we had consensus
16:03:33 jaypipes I specifically remember saying "doing claims in the scheduler gets rid of the retry issues (and all associated "upcall shenanigans")
16:03:50 bauzas jaypipes: well, I do remember exatly operators nodding at dansmith's proposal of passing down a list of possible acceptable hosts
16:03:53 dansmith jaypipes: so, we also agreed we can't fully get rid of retries right away, which means conductor _will_ have to do allocation calls as well
16:03:56 dansmith right
16:03:57 mriedem note we've also been saying "claims in the scheduler*"
16:04:16 dansmith jaypipes: I'm opposed to doing it in scheduler first and conductor second.. I think that's a mess
16:04:23 melwitt mriedem: thanks. short answer is, you're right. replying on the review now
16:04:35 jaypipes dansmith: I see no reason to "do it in the conductor second".
16:04:45 mriedem melwitt: ok - so basically wtf for how this behaves, but it's not backward incompatible so ok
16:04:52 jaypipes dansmith: if the scheduler does the claim, there is no reason for anything else to do the claim.
16:05:00 dansmith jaypipes: reschedule
16:05:06 mriedem jaypipes: retries from the computes go up to the cell conductor
16:05:10 jaypipes dansmith: we should *not* support reschedule. period.
16:05:16 dansmith jaypipes: there was almost zero support in the room for eliminating reschedules
16:05:30 bauzas I thought we had that discussion in Boston :/
16:05:33 dansmith jaypipes: which was the whole point of the "send extra hosts down to the cell" discussion, which you were definitely there for
16:05:36 jaypipes dansmith: yes there was when I said "95% of the causes of reschedules currently will be eliminated"
16:05:36 dansmith bauzas: we did
16:05:44 dansmith *facepalm*
16:06:03 melwitt mriedem: yeah ... I'm trying to make all of the formats of count as dict consistent. before, I had some dict and some int so that only things that could have project and user would be dict, but people found that more confusing
16:06:05 bauzas anyway, I'm litterally 24 hours away of taking a long break
16:06:42 jaypipes dansmith: perhaps this is the session I had to leave early (I was standing in the back and right before I left I said "recschedules should not occur and the reason for them will be gone when claining in the scheduler"?
16:06:45 bauzas I'm trying hard to make sure we're all on the same page before I leave
16:06:58 dansmith jaypipes: bauzas: mriedem: I'm good for a hangout any time starting now, whenever you want, until 11:30 PDT
16:07:08 bauzas I can
16:07:09 jaypipes I can do a hangout whenever.
16:07:13 mriedem i can also do now
16:07:17 jaypipes ++
16:07:23 dansmith jaypipes: yeah, you left, but I thought you were still around for the consensus on reschedules.. however you _were_ there for the scheduling claims session where we talked about the extra hosts thing
16:07:30 melwitt eliminating reschedules would make things really suck for Ironic users in the absence of a porcelain API that does retries
16:07:47 jaypipes melwitt: you mean "standalone Ironic".
16:07:53 melwitt no, Ironic + Nova
16:08:15 mriedem plus i think numa came up
16:08:20 dansmith I'll get a link, just a sec
16:08:22 mriedem in the claims in the scheduler session
16:08:40 jaypipes mriedem: NUMA is handled in the existing NUMAtopologyFilter.
16:08:57 dansmith https://hangouts.google.com/call/e6jtzndusjf6rezqvwdatascvuu
16:09:12 bauzas joining
16:09:16 jaypipes mriedem: claims would occur after the NUMATopologyFilter selected a host. We could have the NUMAtopology that was selected (and currently thrown away by the filter) written to the compute_nodes table.
16:10:26 bauzas dansmith: you're not there in the hangout
16:11:15 jaypipes melwitt: honestly, I don't really care about use cases where an IPMI boot failure results in an ERROR state for an instance
16:12:30 sajeesh ping sdague
16:14:37 sdague sajeesh: what's up?
16:14:58 sajeesh Hi sdague ..you have written in the mail Get someone from the CERN team to write down what their ideal
16:14:58 sajeesh ruleset looks like, so it can be turned into a worked example
16:15:12 sdague sajeesh: yes
16:15:20 sajeesh by ruleset do you mean all use cases ?
16:15:51 sdague not use cases, how they think they would want hierarchical quotas to work
16:16:06 sajeesh ok
16:16:06 melwitt jaypipes: :\ I don't know the particulars on what is different about scheduling onto baremetal directly, but when you have enough nodes, enough of them could be flaky and if we fail and go to error, we're not absorbing any of that for the end user
16:17:03 sdague sajeesh: because the use cases here don't build the hierarchical model all that well, most people set a bunch of use cases together that actually are conflicting
16:17:33 sdague so, I want the model algorithm to be proposed by groups that are interested, then we'll work it through
16:18:00 sajeesh sdague:+1
16:21:44 ayoung Since we are rewriting how you guys do policy enforcement, maybe y'all should take a look: https://review.openstack.org/#/c/384148/
16:22:50 ayoung alaski, it has changed a bit since you last looked at it...in October
16:25:23 cdent jaypipes are you guys still on the hangout?
16:25:30 jaypipes cdent: yes
16:25:37 sdague ayoung: so, my concern with is_admin on project is that now if you give a user authority to add other users to a project, they also get to see the physical topology of your cloud, and perform live migrations. Because while is_admin is no longer fully global, it isn't service scoped, right?
16:26:41 alaski ayoung: I'm no longer actively involved in openstack and likely won't get a chance to look too closely at that change.
16:26:54 ayoung alaski, understood
16:27:06 ayoung sdague, let me parse what you are asking.

Earlier   Later