| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-11-05 | |||
| 15:06:12 | openstackgerrit | Merged openstack/nova master: Minimal construct plumbing for nova show when a cell is down https://review.openstack.org/591658 | |
| 15:06:53 | mnaser | I think it might be around live migrations, I suspect that if live migrations fail on a machine that somehow already has issues talking to placement then it won’t be able to revert the allocation or whatever | |
| 15:07:04 | mnaser | Do the compute nodes make all the claims during a resize or live migration? | |
| 15:07:25 | mnaser | I.e is it possible scheduler does something that compute cannot revert because of an intermittent issue | |
| 15:07:52 | tssurya | efried, mnaser: there is no way we don't have out of sync issues I am literally already working on a consistency tool but I can confirm more after looking further about the statistics regarding allocations/inventories - about how much those are out of sync | |
| 15:07:59 | efried | mnaser: certainly, although I thought we had (a bunch of really ugly) code to clean up that mess. | |
| 15:08:17 | sean-k-mooney | mnaser: i was under the impression the cpu ram and disk were still claimed in the schduler on migration | |
| 15:08:30 | sean-k-mooney | but pci device would be cliamied by the compute nodes | |
| 15:08:43 | sean-k-mooney | well that is a bad example | |
| 15:08:44 | efried | mnaser: I can't think of a way inventories would get out of sync unless a third party is mucking with them (e.g. CLI). | |
| 15:08:56 | efried | mnaser: But failed migrations of various types - absolutely. | |
| 15:09:05 | efried | ^^ for allocations | |
| 15:09:10 | mnaser | tssurya: let me share a tool that does exactly that! | |
| 15:09:25 | mnaser | Yeah I don’t think I ever had issue with inventories | |
| 15:09:41 | tssurya | mnaser: that would be great then I can get some statistics | |
| 15:09:58 | efried | mnaser: But as we were discussing above, even if you max out the refresh interval, we are still polling (and "healing") the inventories every periodic. | |
| 15:10:08 | efried | so if the inventories did drift, we would have fixed them. | |
| 15:10:16 | tssurya | efried: true | |
| 15:10:17 | efried | So there's really no way to know if drift issues exist there. | |
| 15:10:19 | mnaser | yeah but i dont think those would drift in our use cases | |
| 15:10:22 | efried | ...until I kill that. | |
| 15:10:23 | mnaser | even if they didnt sync | |
| 15:10:24 | mnaser | :P | |
| 15:10:57 | efried | I think I have a pretty major rework to do on this patch now that I'm thinking in terms of the inventory refreshes... | |
| 15:11:05 | mriedem | tssurya: https://bugs.launchpad.net/nova/+bug/1793569 | |
| 15:11:06 | openstack | Launchpad bug 1793569 in OpenStack Compute (nova) "Add placement audit commands" [Wishlist,Confirmed] | |
| 15:11:31 | tssurya | mriedem: thanks | |
| 15:12:30 | mnaser | http://paste.openstack.org/show/734146/ | |
| 15:12:35 | mnaser | easier to digest paste because of launchpad's wrapping | |
| 15:12:44 | mnaser | tssurya: ^ | |
| 15:13:07 | openstackgerrit | John Garbutt proposed openstack/nova-specs master: Add Unified Limits Spec https://review.openstack.org/602201 | |
| 15:18:21 | tssurya | mnaser: thanks :) | |
| 15:18:39 | mriedem | efried: fwiw, i would at least split out the change to disable the refresh interval (config value of 0) | |
| 15:18:47 | mriedem | since that's pretty straight forward i imagine | |
| 15:18:54 | efried | mriedem: ack, thx | |
| 15:50:48 | openstackgerrit | Maciej Jozefczyk proposed openstack/nova master: Force refresh instance info_cache during heal https://review.openstack.org/591607 | |
| 15:50:48 | openstackgerrit | Maciej Jozefczyk proposed openstack/nova master: Add fill_virtual_interface_list online_data_migration script https://review.openstack.org/614167 | |
| 15:55:39 | openstackgerrit | Maciej Jozefczyk proposed openstack/nova master: Add fill_virtual_interface_list online_data_migration script https://review.openstack.org/614167 | |
| 16:11:26 | openstackgerrit | Eric Fried proposed openstack/nova master: DNM: Trust the report client cache more https://review.openstack.org/614886 | |
| 16:33:03 | efried | stephenfin: Does :oslo.config:option: not work in renos yet? | |
| 16:33:55 | stephenfin | efried: https://github.com/openstack/nova/blob/master/doc/source/conf.py vs. https://github.com/openstack/nova/blob/master/releasenotes/source/conf.py | |
| 16:34:11 | stephenfin | efried: It's a different build. You'd need to configure it for reno | |
| 16:34:22 | stephenfin | But I strongly advise against doing so | |
| 16:34:31 | efried | stephenfin: ack. I think I don't care that much. | |
| 16:34:36 | efried | why would you advise against it? | |
| 16:34:49 | stephenfin | If we removed/renamed the option in the future, we'd break the build | |
| 16:35:05 | efried | as we should | |
| 16:35:07 | efried | oh | |
| 16:35:15 | efried | the reno should be able to stay forever | |
| 16:35:17 | efried | got it. | |
| 16:35:20 | stephenfin | Yup | |
| 16:35:28 | efried | butbutbut | |
| 16:35:37 | sean-k-mooney | i was about to say we cant fix rennos after teh fact right | |
| 16:35:50 | efried | shouldn't it therefore point to the respective version of the config.html? | |
| 16:35:57 | sean-k-mooney | or at least its non trivail to change them after a release is done | |
| 16:36:33 | efried | sean-k-mooney: You asked me to review https://review.openstack.org/#/c/610034/ -- I'm out of my depth on that one, I'm afraid. | |
| 16:36:36 | sean-k-mooney | efried: it would if you had the commit checked out | |
| 16:36:37 | stephenfin | sean-k-mooney: We couldn't in the past but we can now (reno fixed that). That said, why make life hard for yourself | |
| 16:36:42 | stephenfin | *? | |
| 16:36:56 | stephenfin | efried: I guess it should, yes | |
| 16:36:57 | sean-k-mooney | efried: ok no worries | |
| 16:37:12 | stephenfin | But that ship has long since sailed. All our doc links in renos point to latest | |
| 16:37:39 | efried | sean-k-mooney: You spelled "outer" wrong! -2! | |
| 16:38:01 | efried | that's about the best I can do on that one. | |
| 16:38:04 | sean-k-mooney | where in the patch | |
| 16:38:18 | efried | https://review.openstack.org/#/c/610034/10/nova/utils.py@1322 | |
| 16:38:48 | sean-k-mooney | ... | |
| 16:39:23 | sean-k-mooney | yes i did ... | |
| 16:40:26 | sean-k-mooney | that means its incorrect in the placement version too which is a pain. | |
| 16:40:50 | sean-k-mooney | ill repin hopefully for the last time | |
| 16:41:22 | sean-k-mooney | efried: thanks for finding it it woudl have been even more of a pain when backporting this | |
| 16:42:05 | efried | sean-k-mooney: Well, it's really not a big deal, especially since that method isn't exposed anywhere outside of the decorator. But sure, if you're going to respin; I imagine the other non-blocking nits are worse than that :) | |
| 16:42:39 | sean-k-mooney | well i wasnt but i can if needed | |
| 16:43:20 | sean-k-mooney | this has been part of an a downstream ci blocking bug for almost a month so i would like to finally get it fixed | |
| 16:48:05 | openstackgerrit | Lance Bragstad proposed openstack/nova master: WIP: experiment with oslo.limit interface https://review.openstack.org/615602 | |
| 16:48:38 | lbragstad | johnthetubaguy i took a wild crack at bringing the oslo.limit logic closer to the actual enforcement point, based on the commit you have ^ | |
| 16:49:07 | johnthetubaguy | lbragstad: ah, interesting, will have a look | |
| 16:49:38 | lbragstad | i guess i wanted to try and see if it was possible to make oslo.limit simple enough to not require a nova.limit module for dynamic limits | |
| 16:49:48 | lbragstad | but - what you have there for static limits totally makes sense | |
| 16:50:28 | lbragstad | i was just thinking that the original direction of the context manager might be hard to take advantage of if we go from api code -> limit module -> oslo limit | |
| 16:50:57 | lbragstad | since oslo.limit and the Enforcer context manager was written to be as close to the actual code consuming resources as possible | |
| 16:51:20 | johnthetubaguy | the problem is the retry check is always in another process at the moment | |
| 16:51:44 | johnthetubaguy | so not sure where the context manager will help, I am hoping jaypipes's patch will be different though | |
| 16:52:35 | lbragstad | dumb question, but what do you mean by the retry logic being in a different process? | |
| 16:52:58 | lbragstad | i was under the assumption that retry logic was in the same area of code that the resource consumption happens | |
| 16:53:00 | sean-k-mooney | what are people feeilngs about adding more debug logs in the schduler | |
| 16:53:15 | sean-k-mooney | or rather in the numa_toplogy_filter to be specific | |
| 16:54:06 | sean-k-mooney | im trying to debug a no valid host form schduler logs and the is nothing to go on to determin why the numa toplogy filter decied it was invlaid | |
| 16:54:18 | johnthetubaguy | lbragstad: it isn't at the moment sadly | |
| 16:54:55 | lbragstad | is the retry logic the thing that protects against race conditions between clients/ | |
| 16:55:00 | lbragstad | or is that something else? | |
| 16:55:06 | jaypipes | sean-k-mooney: if a deployer is using the NUMATopologyFilter, they don't care about quickness of the scheduler. I say go for it. | |
| 16:55:39 | johnthetubaguy | sean-k-mooney: I prefer debug logs only on the reject path, if possible, but what jaypipes said too | |
| 16:55:58 | johnthetubaguy | lbragstad: it is that thing, let me link to an example | |
| 16:56:17 | sean-k-mooney | ya it wanted to pring the host toplogy and requested guest topology as a debug message only on failure | |
| 16:56:31 | johnthetubaguy | lbragstad: this is the recheck for build requests: https://review.openstack.org/#/c/615180/5/nova/conductor/manager.py | |
| 16:57:08 | sean-k-mooney | at the moment we jsut asy it did not fit | |
| 16:57:23 | sean-k-mooney | that could be a little noisy however... | |
| 16:57:30 | johnthetubaguy | lbragstad: the first check is in the API process, here: https://github.com/openstack/nova/blob/8d089111c8554e94e117ada3a7f51a42df59e84f/nova/compute/api.py#L868 | |
| 16:57:36 | lbragstad | ahhh | |