| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-11-05 | |||
| 14:52:50 | efried | and we're not doing much with those in q/r. | |
| 14:53:00 | tssurya | yea I am pretty sure it was only the traits and aggregates syn that was off | |
| 14:53:09 | tssurya | the inventory sync was like normal for us | |
| 14:53:28 | efried | the poll you're stretching would also update inventories, but there was an additional, separate inventory update that was happening outside of that one, so yeah. | |
| 14:53:45 | tssurya | ah yea that makes sense | |
| 14:54:00 | tssurya | because we didn't have that much out of syn inventory issues | |
| 14:54:04 | tssurya | sync* | |
| 14:55:16 | efried | tssurya: The theory behind the patch is that you shouldn't have those issues anyway, even if we *never* poll. | |
| 14:55:20 | efried | but | |
| 14:55:30 | efried | I'm not sure I hit the code paths that do that separate inventory update | |
| 14:55:32 | efried | sad face | |
| 14:55:53 | jaypipes | three more unit test failures to address... closing in... | |
| 14:55:56 | tssurya | haha, I am trying to go through your commit now... but | |
| 14:56:08 | tssurya | so idea is a inventory refresh only if something changes right ? | |
| 14:57:08 | efried | tssurya: well, at least in this code path, yes. Basically, if you set the refresh interval to zero, the only time you would get a refresh is if e.g. the virt driver pushes a change via update_provider_tree. | |
| 14:57:31 | efried | jaypipes: IYO, should there be a bp and/or spec for this? And should it be multiple patches? | |
| 14:58:06 | jaypipes | efried: the cache change stuff? | |
| 14:58:09 | efried | y | |
| 14:58:44 | jaypipes | efried: I think it would be useful to have a bp for tracking purposes, sure. spec, not so much. | |
| 14:58:56 | efried | ight, we'll start there. | |
| 15:02:31 | mnaser | tssurya: I can give you a script I wrote to audit placement (I think it’s in a paste somewhere). Interesting to see if you see things get out of sync too | |
| 15:02:50 | tssurya | efried: hmm yea makes sense as for the syncing issues which I am sure we would hit considering our size, can't we slowly build a heal/sync tool like we have for allocations/aggregates that deployments can run when they want ? | |
| 15:03:12 | openstackgerrit | John Garbutt proposed openstack/nova master: WIP: Integrating with unified limits https://review.openstack.org/615180 | |
| 15:03:20 | tssurya | like a placement sync tool for plausible things that would go out of sync | |
| 15:04:01 | tssurya | mnaser: ah thanks so far we haven't like disabled any major sync updates | |
| 15:04:29 | tssurya | but we surely do have out of sync issues | |
| 15:04:31 | efried | tssurya: I think healing allocations is a separate issue. If we can show that inventories/traits/aggregates don't get out of sync when we don't refresh them, that's goodness. And then separately, if we can write something to heal allocations - or better yet, to identify why they're getting out of sync in the first place and close that gap - also goodness. | |
| 15:04:41 | mnaser | tssurya: this is for allocations being out of sync tho, so it’d be interesting if this is something we have broken I guess | |
| 15:05:52 | efried | mnaser: Do you have any suspicions (or better) about where the allocation drift is happening? Like, is it on instances that fail a migration or similar? Resizes? Evacuations? Or (eek) just steady state? | |
| 15:06:12 | openstackgerrit | Merged openstack/nova master: Minimal construct plumbing for nova show when a cell is down https://review.openstack.org/591658 | |
| 15:06:53 | mnaser | I think it might be around live migrations, I suspect that if live migrations fail on a machine that somehow already has issues talking to placement then it won’t be able to revert the allocation or whatever | |
| 15:07:04 | mnaser | Do the compute nodes make all the claims during a resize or live migration? | |
| 15:07:25 | mnaser | I.e is it possible scheduler does something that compute cannot revert because of an intermittent issue | |
| 15:07:52 | tssurya | efried, mnaser: there is no way we don't have out of sync issues I am literally already working on a consistency tool but I can confirm more after looking further about the statistics regarding allocations/inventories - about how much those are out of sync | |
| 15:07:59 | efried | mnaser: certainly, although I thought we had (a bunch of really ugly) code to clean up that mess. | |
| 15:08:17 | sean-k-mooney | mnaser: i was under the impression the cpu ram and disk were still claimed in the schduler on migration | |
| 15:08:30 | sean-k-mooney | but pci device would be cliamied by the compute nodes | |
| 15:08:43 | sean-k-mooney | well that is a bad example | |
| 15:08:44 | efried | mnaser: I can't think of a way inventories would get out of sync unless a third party is mucking with them (e.g. CLI). | |
| 15:08:56 | efried | mnaser: But failed migrations of various types - absolutely. | |
| 15:09:05 | efried | ^^ for allocations | |
| 15:09:10 | mnaser | tssurya: let me share a tool that does exactly that! | |
| 15:09:25 | mnaser | Yeah I don’t think I ever had issue with inventories | |
| 15:09:41 | tssurya | mnaser: that would be great then I can get some statistics | |
| 15:09:58 | efried | mnaser: But as we were discussing above, even if you max out the refresh interval, we are still polling (and "healing") the inventories every periodic. | |
| 15:10:08 | efried | so if the inventories did drift, we would have fixed them. | |
| 15:10:16 | tssurya | efried: true | |
| 15:10:17 | efried | So there's really no way to know if drift issues exist there. | |
| 15:10:19 | mnaser | yeah but i dont think those would drift in our use cases | |
| 15:10:22 | efried | ...until I kill that. | |
| 15:10:23 | mnaser | even if they didnt sync | |
| 15:10:24 | mnaser | :P | |
| 15:10:57 | efried | I think I have a pretty major rework to do on this patch now that I'm thinking in terms of the inventory refreshes... | |
| 15:11:05 | mriedem | tssurya: https://bugs.launchpad.net/nova/+bug/1793569 | |
| 15:11:06 | openstack | Launchpad bug 1793569 in OpenStack Compute (nova) "Add placement audit commands" [Wishlist,Confirmed] | |
| 15:11:31 | tssurya | mriedem: thanks | |
| 15:12:30 | mnaser | http://paste.openstack.org/show/734146/ | |
| 15:12:35 | mnaser | easier to digest paste because of launchpad's wrapping | |
| 15:12:44 | mnaser | tssurya: ^ | |
| 15:13:07 | openstackgerrit | John Garbutt proposed openstack/nova-specs master: Add Unified Limits Spec https://review.openstack.org/602201 | |
| 15:18:21 | tssurya | mnaser: thanks :) | |
| 15:18:39 | mriedem | efried: fwiw, i would at least split out the change to disable the refresh interval (config value of 0) | |
| 15:18:47 | mriedem | since that's pretty straight forward i imagine | |
| 15:18:54 | efried | mriedem: ack, thx | |
| 15:50:48 | openstackgerrit | Maciej Jozefczyk proposed openstack/nova master: Add fill_virtual_interface_list online_data_migration script https://review.openstack.org/614167 | |
| 15:50:48 | openstackgerrit | Maciej Jozefczyk proposed openstack/nova master: Force refresh instance info_cache during heal https://review.openstack.org/591607 | |
| 15:55:39 | openstackgerrit | Maciej Jozefczyk proposed openstack/nova master: Add fill_virtual_interface_list online_data_migration script https://review.openstack.org/614167 | |
| 16:11:26 | openstackgerrit | Eric Fried proposed openstack/nova master: DNM: Trust the report client cache more https://review.openstack.org/614886 | |
| 16:33:03 | efried | stephenfin: Does :oslo.config:option: not work in renos yet? | |
| 16:33:55 | stephenfin | efried: https://github.com/openstack/nova/blob/master/doc/source/conf.py vs. https://github.com/openstack/nova/blob/master/releasenotes/source/conf.py | |
| 16:34:11 | stephenfin | efried: It's a different build. You'd need to configure it for reno | |
| 16:34:22 | stephenfin | But I strongly advise against doing so | |
| 16:34:31 | efried | stephenfin: ack. I think I don't care that much. | |
| 16:34:36 | efried | why would you advise against it? | |
| 16:34:49 | stephenfin | If we removed/renamed the option in the future, we'd break the build | |
| 16:35:05 | efried | as we should | |
| 16:35:07 | efried | oh | |
| 16:35:15 | efried | the reno should be able to stay forever | |
| 16:35:17 | efried | got it. | |
| 16:35:20 | stephenfin | Yup | |
| 16:35:28 | efried | butbutbut | |
| 16:35:37 | sean-k-mooney | i was about to say we cant fix rennos after teh fact right | |
| 16:35:50 | efried | shouldn't it therefore point to the respective version of the config.html? | |
| 16:35:57 | sean-k-mooney | or at least its non trivail to change them after a release is done | |
| 16:36:33 | efried | sean-k-mooney: You asked me to review https://review.openstack.org/#/c/610034/ -- I'm out of my depth on that one, I'm afraid. | |
| 16:36:36 | sean-k-mooney | efried: it would if you had the commit checked out | |
| 16:36:37 | stephenfin | sean-k-mooney: We couldn't in the past but we can now (reno fixed that). That said, why make life hard for yourself | |
| 16:36:42 | stephenfin | *? | |
| 16:36:56 | stephenfin | efried: I guess it should, yes | |
| 16:36:57 | sean-k-mooney | efried: ok no worries | |
| 16:37:12 | stephenfin | But that ship has long since sailed. All our doc links in renos point to latest | |
| 16:37:39 | efried | sean-k-mooney: You spelled "outer" wrong! -2! | |
| 16:38:01 | efried | that's about the best I can do on that one. | |
| 16:38:04 | sean-k-mooney | where in the patch | |
| 16:38:18 | efried | https://review.openstack.org/#/c/610034/10/nova/utils.py@1322 | |
| 16:38:48 | sean-k-mooney | ... | |
| 16:39:23 | sean-k-mooney | yes i did ... | |
| 16:40:26 | sean-k-mooney | that means its incorrect in the placement version too which is a pain. | |
| 16:40:50 | sean-k-mooney | ill repin hopefully for the last time | |
| 16:41:22 | sean-k-mooney | efried: thanks for finding it it woudl have been even more of a pain when backporting this | |