| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-05-22 | |||
| 16:41:58 | ayoung | sdague, determining which operations are global vs scoped is a decision for the Nova engineering team to make | |
| 16:42:08 | ayoung | as is figuring out how to enforce the scoping | |
| 16:42:22 | ayoung | what Keystione is providing is the answer to "is this token valid for global operations" | |
| 16:42:44 | ayoung | this patch is using that answer to start differentiating between scoped and global operationws | |
| 16:42:57 | ayoung | could it be done in middleware? Not today | |
| 16:43:03 | ayoung | Not without a lot more tooling. | |
| 16:43:35 | sdague | ayoung: so, I'll repeat, this isn't going to make any sense to me without those other diagrams and flows. So I'll stay opted out of the conversation if those don't exist. | |
| 16:43:54 | ayoung | sdague, huh? | |
| 16:44:08 | ayoung | I...I don't think I can make it any more simple than that. | |
| 16:44:22 | ayoung | In order to perform global admin operations, you need a token scoped to the admin project. | |
| 16:46:18 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: Scheduler returning alternatives per destination https://review.openstack.org/465175 | |
| 16:46:18 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: Delete existing allocations when retrying https://review.openstack.org/460433 | |
| 16:46:19 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: Don't delete allocation if instance being scheduled https://review.openstack.org/460177 | |
| 16:46:19 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: WIP: Allocate instance by the conductor https://review.openstack.org/460455 | |
| 16:46:20 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: Fix ChanceScheduler returning a wrong node https://review.openstack.org/466725 | |
| 16:54:23 | ayoung | sdague, what exactly would you require in order to move this forward? Considering that a comparable change needs to go into every openstack project, I'm guessing that whatever we provided for the Nova team would help move it forward across the board. The current implementations are really broken, due to Keystone not having provided a way to distinguish between global and scoped tokens. But that mistake goes back to pre-Ke | |
| 16:54:23 | ayoung | ystone Nova's token system as well. | |
| 16:54:42 | ayoung | Just because it has been broken since Cactus does not mean it is not broken. | |
| 17:04:09 | bauzas | jaypipes: mriedem: dansmith: cdent: melwitt: edleafe: FWIW, given the outcome of our hangout, I'm going to stop working on my series, in particular given I'll need to disappear starting tomorrow | |
| 17:04:23 | bauzas | edleafe: feel free to takeover any change you want | |
| 17:04:57 | bauzas | edleafe: I just removed my -2s that I gave you previously | |
| 17:05:09 | bauzas | tl;dr: all is clear for iterating fast | |
| 17:05:32 | bauzas | I'll spend my day tomorrow for bug triaging and reviews | |
| 17:09:21 | ayoung | sdague, here is my RBAC talk. https://www.openstack.org/videos/boston-2017/per-api-role-based-access-control please watch it when you have the time. In the meanwhile, I will try to put together the explanation of why we are approach the fix to 968696 the way we are. | |
| 17:15:24 | bauzas | mriedem: melwitt: do you want me to publish a new rev for the ReqSpec delete where I would call the deletion in https://review.openstack.org/#/c/391060/9/nova/compute/api.py@2046 for example ? | |
| 17:15:33 | bauzas | I can do that quickly so you both could review | |
| 17:16:24 | cdent | bauzas: ack on the all clear for iterating | |
| 17:16:57 | mriedem | bauzas: i don't think that's the place to do it | |
| 17:17:05 | mriedem | _local_delete is the place to do it in the api | |
| 17:17:20 | bauzas | mriedem: I saw you wanted to restrict for local deletes | |
| 17:17:20 | mriedem | bauzas: but i won't be reviewing anything else for awhile, i'm on patch 1 of the quota series | |
| 17:17:37 | mriedem | bauzas: the _local_delete method is where local delete happens | |
| 17:17:41 | bauzas | mriedem: like I said in the comment, I think it's pretty harmless | |
| 17:18:08 | bauzas | mriedem: well, I think we would loose a huge number of deletions if we only do that for local instance deletes | |
| 17:18:15 | bauzas | right? | |
| 17:18:16 | melwitt | bauzas: I didn't know about the on-the-fly reqspec it can come up with if it doesn't find one, so it's not catastrophic in the case we described | |
| 17:19:36 | melwitt | it would still be something like "user can see and do actions on instance but instance may not honor original scheduling spec" | |
| 17:19:57 | mriedem | we are still going to need something to cleanup api db orphans, like instance_mappings | |
| 17:20:05 | bauzas | melwitt: eg. https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L294 | |
| 17:20:11 | mriedem | idk, i can't context switch on this right now | |
| 17:20:53 | bauzas | okay, let's postpone that for later | |
| 17:20:58 | bauzas | it's a bugfix, we can land it later | |
| 17:21:15 | bauzas | I'll review jaypipes's series for shared-rps | |
| 17:36:17 | edmondsw | johnthetubaguy can you point me to the patch you mentioned at the summit to swap network routing before pause/resume on live migrate? | |
| 17:44:43 | mriedem | melwitt: just so you're aware, i posted comments in the base patch but i'm still working through the rest | |
| 17:45:46 | gyee | mriedem: can you please take a look whenever you have a chance? https://review.openstack.org/#/c/466143/ | |
| 17:45:48 | melwitt | mriedem: cool, thanks | |
| 17:46:24 | openstackgerrit | Steve Noyes proposed openstack/nova master: WIP: update live migration to use v3 cinder api https://review.openstack.org/463987 | |
| 17:54:40 | jaypipes | mriedem, dansmith, melwitt, cdent, bauzas: ok, ML post sent about getting rid of automated reschedule behaviour. | |
| 17:55:15 | dansmith | ack | |
| 17:56:35 | jaypipes | mriedem, dansmith, melwitt, cdent, bauzas: pls let me know if I accurately explained the issue and the request. | |
| 17:56:48 | melwitt | out of curiosity I checked how kubernetes scheduling works and whether it retries, to be aware of what others are doing, | |
| 17:57:22 | melwitt | and it sounds like it does retry forever "remains in pending state until suitable machine becomes available" according to this post. I didn't find a doc about it on k8s site yet https://stackoverflow.com/questions/28857993/how-does-kubernetes-scheduler-work/28874577#28874577 | |
| 17:57:41 | jaypipes | melwitt: well, yeah, because k8s is an orchestrator :) | |
| 17:57:56 | dansmith | yeah, totally different | |
| 17:58:06 | jaypipes | melwitt: which fits nicely into my "nova shouldn't be an orchestrator" point ;) | |
| 17:58:28 | melwitt | but openstack doesn't have an orchestrator at all, does it? | |
| 17:58:57 | melwitt | so our thing just performs worse | |
| 17:59:25 | jaypipes | melwitt: heat? | |
| 17:59:27 | dansmith | heat | |
| 17:59:28 | dansmith | heh | |
| 17:59:36 | jaypipes | melwitt: or tacker for NFV. | |
| 18:00:45 | melwitt | I think heat is a manual orchestrator, you (the end user) have to tell it everything to do, I thought | |
| 18:01:28 | jaypipes | melwitt: well, k8s is the same. you declare a pod definition and k8s tries constantly to make your pod definition a reality. | |
| 18:01:51 | jaypipes | melwitt: or you just meant heat doesn't constantly try to do things? | |
| 18:03:04 | melwitt | jaypipes: more the latter. I'm not a heat expert, maybe there's a way to tell it to keep retrying a definition | |
| 18:03:29 | jaypipes | melwitt: yeah, I'm not a Heat expert either. | |
| 18:17:08 | cdent | jaypipes, melwitt: if want to play the odds, it is more likely that a "keep retrying orchestrator" (that actually does the job well) will come into existence if we do not include bad orchestration in nova. | |
| 18:17:36 | jaypipes | cdent: ++ | |
| 18:17:54 | cdent | it is also the case that the odds of "good orchestration" merging sooner (in nova) than something else trying to do it (outside of nova's constraints) are low | |
| 18:20:43 | melwitt | cdent: maybe. I'm not sure how many people would want to use an orchestrator-less system, but that works as long as no other viable options are available | |
| 18:21:37 | cdent | melwitt: Is what we have today orchestrator-less or bad-orchestrator? | |
| 18:22:25 | melwitt | cdent: bad-orchestrator | |
| 18:22:51 | cdent | as in makes a half hearted effort at retrying and then bails..., yeah? | |
| 18:23:23 | melwitt | yes | |
| 18:25:47 | jaypipes | dansmith: you were +2 previously. I'm good on https://review.openstack.org/#/c/460177/ now with bauzas' latest revision. | |
| 18:26:48 | melwitt | cdent: with the half hearted effort, an operators can detect a bad baremetal node and take it out of rotation without exposing end users to the fail. without it, the end user would have to suffer the failure at least once before the operator can take it out of rotation. unless the operator builds a porcelain API that maybe uses heat behind it and has end users point at that. is what I'm thinking | |
| 18:29:43 | melwitt | I can also see the point of, you can't serve everybody perfectly, and if it's really only 5% of people that would be affected, then I get that. we'll see what they say on the ML | |
| 18:35:38 | mriedem | melwitt: ok https://review.openstack.org/#/c/446239/ | |
| 18:35:44 | mriedem | melwitt: i didn't go through tests | |
| 18:36:38 | melwitt | mriedem: thanks. reading through | |
| 18:38:56 | mriedem | i have successfully finished a 60+ ounce bag of tootsie rolls from halloween | |
| 18:39:02 | mriedem | it was a grind, but it's done | |
| 18:40:47 | cdent | my jaw hurts now | |
| 18:54:07 | jaypipes | melwitt: hmm, I wonder if some combination of beer and cat toys will be enough to bribe your James into my worldview... | |
| 18:55:47 | mriedem | jaypipes: not until you can chug a beer faster than him | |
| 18:56:04 | mriedem | which from fenway we know isn't going to happen without proper training with a montage | |
| 18:56:52 | melwitt | heh | |
| 19:02:12 | melwitt | mriedem: replied on quotas | |
| 19:18:09 | cdent | mriedem, jaypipes, melwitt, dansmith: Does the idea that was bounced around on the call of claim in the scheduler, re-claim (when things are bad) in the conductor address the issues james is pointing out? Trying to clarify my confused brain. | |
| 19:18:42 | mriedem | cdent: yes and is what we discussed in boston | |
| 19:19:08 | mriedem | super conductor passes alternate hosts to the cell; if something fails, we reschedule via conductor within the cell using those alternate hosts | |
| 19:19:10 | cdent | i knew that the boston plan did, but wasn't clear on if the claim from two different places idea did | |
| 19:19:20 | cdent | gotchya, thanks | |
| 19:19:44 | mriedem | claiming in two places has the same long-term functionality if we have to retry within a cell | |
| 19:20:26 | mriedem | i think the idea of doing claims in the scheduler AND conductor is if we do it in the scheduler, we have (a) tighter retry loop and (b) if we eventually drop reschedules in the cell, then we've already got the work done in the scheduler - as dan pointed out on the call | |
| 19:20:26 | cdent | I think I'm leaning that way because of the "claim as early as possible" stuff I mentioned in one of my comments | |
| 19:20:30 | mriedem | the short-term downside is you have duplication | |
| 19:21:01 | mriedem | i do'nt really see how that's not hidden in the scheduler report client though | |
| 19:21:07 | cdent | DRY is what breeds and deeply nested object hierarchies and that makes me cry | |