| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-11-05 | |||
| 13:52:16 | johnthetubaguy | tssurya: currently the operator can specify per user limits, there are no plans to support this post unified limits | |
| 13:52:49 | johnthetubaguy | well basically, the plan is hierarchical limits replace that feature | |
| 13:52:51 | sean-k-mooney | johnthetubaguy: really | |
| 13:52:58 | tssurya | johnthetubaguy: ah thanks, so we are going to drop user quotas altogether ? | |
| 13:53:03 | leakypipes | johnthetubaguy: I'm getting close... | |
| 13:53:13 | johnthetubaguy | tssurya: that is what I propose in the spec and my patches, yeah | |
| 13:53:14 | leakypipes | johnthetubaguy: sigh, these quota unit test are ugly-as-f... | |
| 13:53:26 | sean-k-mooney | johnthetubaguy: so each user would have there own project in that case as a subproject of there teams project | |
| 13:53:30 | johnthetubaguy | leakypipes: not been in them yet! | |
| 13:53:32 | tssurya | johnthetubaguy: thanks I will go read the spec | |
| 13:53:51 | jaypipes | johnthetubaguy: bring a decanter of scotch. | |
| 13:53:54 | jaypipes | you'll need it. | |
| 13:54:05 | jaypipes | hopefully I'll have the worst of them cleaned up shortly. | |
| 13:54:22 | jaypipes | 9 failures left to fix (after converting them from test.TestCase to test.NoDBTestCase) | |
| 13:54:31 | johnthetubaguy | OK, nice | |
| 13:58:06 | sean-k-mooney | jaypipes: johnthetubaguy so has there been any push back on the fact that users with acess to multiple project will nolong have a global limit on the resouce that user can use | |
| 13:59:15 | sean-k-mooney | i know you can kindo of model that with per user project nested under a larger project but with only 2 levels of nesting that is not as flexible as what could be done before | |
| 13:59:39 | johnthetubaguy | sean-k-mooney: so I should be more precise, it was per user within a project, if my memory is correct | |
| 14:00:42 | jaypipes | sean-k-mooney: no, no pushback. | |
| 14:00:43 | johnthetubaguy | sean-k-mooney: the only users we know about, prefer and want to move to two level project quotas (they used the user thing as a workaround) | |
| 14:00:55 | sean-k-mooney | johnthetubaguy: wait could you have a per user qoat that was different form the users global quota for a specific porject before | |
| 14:00:57 | jaypipes | sean-k-mooney: what johnthetubaguy said. | |
| 14:00:57 | johnthetubaguy | yeah, the opposite, we have support for this direction | |
| 14:01:25 | jaypipes | sean-k-mooney: what's a "qoat"? Quota Of All Time? :) | |
| 14:01:33 | johnthetubaguy | sean-k-mooney: there was no global user quota, it was per user within a specific project. Maybe I answered tssurya's question badly | |
| 14:02:15 | sean-k-mooney | oh then i totally missunder stood how that option worked and always gave my team memebr more instance then i planned lol | |
| 14:03:01 | tssurya | johnthetubaguy: no I think you answered my question correctly. we currently support per user quota within projects, after unified limits we won't right ? | |
| 14:03:12 | johnthetubaguy | tssurya: yeah, that's right | |
| 14:03:31 | sean-k-mooney | i assumed the user quota applied across all project so when i gave them acess to a second project outside there main one i used the user quotat to limit there use of th unlimited project | |
| 14:03:54 | tssurya | yea so I guess sean-k-mooney was worried of this to be a big bad regression ? | |
| 14:04:00 | johnthetubaguy | tssurya: the folks who used that also modified policy so you couldn't delete someone elses instance, so its all a bit strange around those parts | |
| 14:04:33 | tssurya | johnthetubaguy: oh yea that is strange if its all in the same project | |
| 14:04:45 | sean-k-mooney | tssurya: from what johnthetubaguy and jaypipes explained there is no regression that i can see so all good | |
| 14:04:45 | johnthetubaguy | sean-k-mooney: yeah, I thought that too till I reviewed melwitt's patches, turns out it doesn't do that | |
| 14:05:58 | sean-k-mooney | johnthetubaguy: well it was a private developer cloud so no harm in my case but ya | |
| 14:06:43 | sean-k-mooney | having to adjust both the project and user limits was always a bit of a pain so it soundls like its an improvment overall | |
| 14:06:54 | johnthetubaguy | cool | |
| 14:06:58 | tssurya | sean-k-mooney: okay :) | |
| 14:44:27 | efried | tssurya: I saw Belmiro respond on the ML; does this mean y'all have tried the patch in your env somewhere? | |
| 14:45:14 | tssurya | efried: well no all we do it increase the config's value to a very high number, ever since we added the config in queens | |
| 14:45:26 | tssurya | so if you allow setting the config to "0" | |
| 14:45:35 | tssurya | that will disable it its good for us | |
| 14:45:40 | efried | tssurya: The patch actually goes quite a bit further than that. | |
| 14:46:07 | efried | I believe with the refresh interval very high, you're still getting the inventory polls. | |
| 14:46:14 | efried | Now you won't even get those anymore. | |
| 14:46:41 | openstackgerrit | John Garbutt proposed openstack/nova-specs master: Add Unified Limits Spec https://review.openstack.org/602201 | |
| 14:46:45 | efried | ...I think. | |
| 14:47:32 | tssurya | efried: oh just read the commit (message) | |
| 14:47:55 | tssurya | yea we don't currently have any of those yet downstream (meaning our code is the same as upstream at the tracker level) | |
| 14:48:15 | tssurya | but it would be cool to optimize the calls to placement to the maximum | |
| 14:49:01 | sean-k-mooney | efried: by the way if you have time can you review https://review.openstack.org/#/c/610034/ | |
| 14:49:07 | tssurya | because in rocky you have more stuff than in queens right ? the way the updates are done with pulling the prodiver tree info and everything (meaing the new update_to_placement function)? | |
| 14:49:35 | efried | tssurya: What I think we'll be really looking for in order to have enough confidence to merge this is for deployments such as yours and mnaser's to put this in place and verify, not so much that the number of placement calls drops off a cliff - that should be a given - but that we don't wind up with the RT getting out of sync with placement's view. | |
| 14:49:43 | efried | tssurya: I don't believe there's much more in rocky than queens, no. | |
| 14:50:11 | tssurya | we are also investigating on our end | |
| 14:50:28 | tssurya | I can see if we can test this patch in our deployment | |
| 14:50:37 | tssurya | and give feedback | |
| 14:51:11 | efried | One thing about mnaser's comments, which I will also mention in a response on the thread, is that this should not affect *allocations* at all. IIUC, it was allocations getting out of sync that mnaser observed. I will be interested to know if he or you have ever observed anything else (inventories, etc.) getting out of sync. | |
| 14:51:29 | efried | Especially you, since you've been running the long-poll in production for a while, right? | |
| 14:51:52 | tssurya | efried: well the long poll used to only switch off traits and aggregates | |
| 14:52:21 | efried | tssurya: It would also affect inventory updates. | |
| 14:52:26 | efried | oh | |
| 14:52:41 | efried | yeah, I see what you're saying now. I think you're right. | |
| 14:52:50 | efried | and we're not doing much with those in q/r. | |
| 14:53:00 | tssurya | yea I am pretty sure it was only the traits and aggregates syn that was off | |
| 14:53:09 | tssurya | the inventory sync was like normal for us | |
| 14:53:28 | efried | the poll you're stretching would also update inventories, but there was an additional, separate inventory update that was happening outside of that one, so yeah. | |
| 14:53:45 | tssurya | ah yea that makes sense | |
| 14:54:00 | tssurya | because we didn't have that much out of syn inventory issues | |
| 14:54:04 | tssurya | sync* | |
| 14:55:16 | efried | tssurya: The theory behind the patch is that you shouldn't have those issues anyway, even if we *never* poll. | |
| 14:55:20 | efried | but | |
| 14:55:30 | efried | I'm not sure I hit the code paths that do that separate inventory update | |
| 14:55:32 | efried | sad face | |
| 14:55:53 | jaypipes | three more unit test failures to address... closing in... | |
| 14:55:56 | tssurya | haha, I am trying to go through your commit now... but | |
| 14:56:08 | tssurya | so idea is a inventory refresh only if something changes right ? | |
| 14:57:08 | efried | tssurya: well, at least in this code path, yes. Basically, if you set the refresh interval to zero, the only time you would get a refresh is if e.g. the virt driver pushes a change via update_provider_tree. | |
| 14:57:31 | efried | jaypipes: IYO, should there be a bp and/or spec for this? And should it be multiple patches? | |
| 14:58:06 | jaypipes | efried: the cache change stuff? | |
| 14:58:09 | efried | y | |
| 14:58:44 | jaypipes | efried: I think it would be useful to have a bp for tracking purposes, sure. spec, not so much. | |
| 14:58:56 | efried | ight, we'll start there. | |
| 15:02:31 | mnaser | tssurya: I can give you a script I wrote to audit placement (I think it’s in a paste somewhere). Interesting to see if you see things get out of sync too | |
| 15:02:50 | tssurya | efried: hmm yea makes sense as for the syncing issues which I am sure we would hit considering our size, can't we slowly build a heal/sync tool like we have for allocations/aggregates that deployments can run when they want ? | |
| 15:03:12 | openstackgerrit | John Garbutt proposed openstack/nova master: WIP: Integrating with unified limits https://review.openstack.org/615180 | |
| 15:03:20 | tssurya | like a placement sync tool for plausible things that would go out of sync | |
| 15:04:01 | tssurya | mnaser: ah thanks so far we haven't like disabled any major sync updates | |
| 15:04:29 | tssurya | but we surely do have out of sync issues | |
| 15:04:31 | efried | tssurya: I think healing allocations is a separate issue. If we can show that inventories/traits/aggregates don't get out of sync when we don't refresh them, that's goodness. And then separately, if we can write something to heal allocations - or better yet, to identify why they're getting out of sync in the first place and close that gap - also goodness. | |
| 15:04:41 | mnaser | tssurya: this is for allocations being out of sync tho, so it’d be interesting if this is something we have broken I guess | |
| 15:05:52 | efried | mnaser: Do you have any suspicions (or better) about where the allocation drift is happening? Like, is it on instances that fail a migration or similar? Resizes? Evacuations? Or (eek) just steady state? | |
| 15:06:12 | openstackgerrit | Merged openstack/nova master: Minimal construct plumbing for nova show when a cell is down https://review.openstack.org/591658 | |
| 15:06:53 | mnaser | I think it might be around live migrations, I suspect that if live migrations fail on a machine that somehow already has issues talking to placement then it won’t be able to revert the allocation or whatever | |
| 15:07:04 | mnaser | Do the compute nodes make all the claims during a resize or live migration? | |
| 15:07:25 | mnaser | I.e is it possible scheduler does something that compute cannot revert because of an intermittent issue | |
| 15:07:52 | tssurya | efried, mnaser: there is no way we don't have out of sync issues I am literally already working on a consistency tool but I can confirm more after looking further about the statistics regarding allocations/inventories - about how much those are out of sync | |
| 15:07:59 | efried | mnaser: certainly, although I thought we had (a bunch of really ugly) code to clean up that mess. | |
| 15:08:17 | sean-k-mooney | mnaser: i was under the impression the cpu ram and disk were still claimed in the schduler on migration | |
| 15:08:30 | sean-k-mooney | but pci device would be cliamied by the compute nodes | |