| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-01-24 | |||
| 13:13:21 | efried | ...which will still have stale entries because we only removed the nodes from the ProviderTree; we didn't remove the cache timeout entry for the descendants. | |
| 13:14:02 | efried | So force=True as mentioned above would work; but the more correct solution would be to remove all those other association refresh times when we invalidate that root from the cache. | |
| 13:14:06 | rgerganov | efried, do you want me to try it? | |
| 13:14:22 | efried | rgerganov If you've got a setup all ready to go, please do. It would take me some time to get to that point. | |
| 13:14:32 | efried | rgerganov Meanwhile, let me come up with the other solution real quick... | |
| 13:14:33 | rgerganov | ok | |
| 13:17:53 | rgerganov | efried, with force=True it refreshed the associations for the root provider but it didn't add the nested RP in the tree | |
| 13:20:06 | efried | rgerganov At a glance, I don't see (and I don't offhand remember) how tree-associated providers get refreshed. While I poke at that, would you please try this: https://review.openstack.org/#/c/533821/7/nova/scheduler/client/report.py@1381 | |
| 13:21:19 | rgerganov | efried, ok, trying | |
| 13:21:20 | efried | Okay, that's not gonna work either. | |
| 13:21:28 | efried | But try it anyway, for grins. | |
| 13:21:33 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Transform instance.resize_prep notification https://review.openstack.org/465081 | |
| 13:24:47 | efried | rgerganov https://review.openstack.org/#/c/526541/19/nova/scheduler/client/report.py@631 and @639 are the things that *should* be repopulating the _provider_tree from placement. | |
| 13:25:00 | rgerganov | efried, AttributeError: 'SchedulerReportClient' object has no attribute 'association_refresh_time' | |
| 13:25:17 | efried | rgerganov Oh, are you sitting on the very top of the series? | |
| 13:25:29 | efried | Or did I forget the underscore :) | |
| 13:27:33 | openstackgerrit | Gao Fei proposed openstack/nova master: Update VMWare vSphere link address https://review.openstack.org/535244 | |
| 13:27:58 | rgerganov | efried, my patches are based on top of "Move refresh time from report client to prov tree" | |
| 13:27:58 | gibi | is it only me or gerrit is slow like hell today | |
| 13:28:01 | gibi | ? | |
| 13:28:11 | rgerganov | gibi, yes, it's really slow | |
| 13:28:13 | efried | gibi Not just you. Slow as hell. Started grinding yesterday. | |
| 13:28:21 | efried | Maybe ask -infra to bounce it. | |
| 13:28:26 | gibi | thanks for the info | |
| 13:28:53 | efried | rgerganov Okay, that patch got rid of _association_refresh_time in the report client, and would have made my suggested fix moot anyway. | |
| 13:29:51 | efried | rgerganov So what I can't figure out is why, in _ensure_resource_provider, the _get_providers_in_tree => populate_from_iterable isn't restoring the descendants to the cache. | |
| 13:30:03 | efried | Are you pdb'ing? | |
| 13:30:07 | rgerganov | efried, yes | |
| 13:30:58 | efried | Can you break at https://review.openstack.org/#/c/536902/1/nova/scheduler/client/report.py@626 at make sure rps_to_refresh contains all the descendants? | |
| 13:31:24 | rgerganov | with or without my patch? | |
| 13:31:35 | efried | Shouldn't matter. | |
| 13:31:52 | rgerganov | ok, tracing | |
| 13:32:13 | efried | Because as it turns out, _refresh_associations isn't where we pick up tree-associated providers. It's in fact in this chunk of _ensure_resource_provider. | |
| 13:33:43 | rgerganov | efried, it does matter because without my patch we don't get there | |
| 13:34:03 | efried | Oh, because 'return uuid'... | |
| 13:34:07 | rgerganov | correct | |
| 13:34:24 | efried | ...but that .exists(uuid) should be False! | |
| 13:34:28 | efried | Because we removed that guy. | |
| 13:34:40 | rgerganov | no, we pass the parent uuid here | |
| 13:34:45 | rgerganov | and it is there | |
| 13:35:19 | artom | Am I doing this right? http://logstash.openstack.org/#dashboard/file/logstash.json?query=libvirtError%3A%20Cannot%20recv%20data | |
| 13:35:30 | rgerganov | efried, _ensure_resource_provider is called with the uuid of the root RP | |
| 13:35:45 | artom | It says 0 hits, but we clearly got some here: http://logs.openstack.org/97/536897/2/check/legacy-tempest-dsvm-cells/fc9986a/logs/screen-n-cpu.txt.gz?level=ERROR#_Jan_23_23_16_41_428802 | |
| 13:35:53 | efried | rgerganov I see, and you were trying to add the bogus trait to the child? | |
| 13:36:02 | rgerganov | efried, correct | |
| 13:36:05 | efried | So we invalidated the child, but not the root. | |
| 13:36:11 | rgerganov | yup | |
| 13:36:13 | efried | Okay, it's coming together. | |
| 13:36:44 | rgerganov | could you please tell why my patch won't refresh the tree? | |
| 13:37:52 | efried | rgerganov No I can't. It should. | |
| 13:38:04 | efried | oh | |
| 13:42:17 | efried | rgerganov So with your patch, is rps_to_refresh populated at that breakpoint? | |
| 13:43:26 | rgerganov | efried, yes | |
| 13:43:46 | efried | rgerganov And after populate_from_iterable, is the _provider_tree populated properly? | |
| 13:43:47 | rgerganov | it contains both parent and child | |
| 13:44:05 | rgerganov | as far as I can see, yes | |
| 13:44:29 | mvenesio | Hi guys i'm trying to set nova to use SSL for the database connection, i set the mysql connection as well as i set it for the rest of the projects like cinder and glance, but for nova it does not work and i got an OperationalError. Any idea about how to set it right ? | |
| 13:45:49 | efried | mvenesio I think that's a better question for #openstack (see channel topic). | |
| 13:46:10 | efried | rgerganov So if the child is in the cache... where do we run into a problem? | |
| 13:46:17 | openstackgerrit | Lee Yarwood proposed openstack/nova master: rbd: flatten images when creating/unshelving an instance https://review.openstack.org/457886 | |
| 13:46:36 | lyarwood | melwitt: https://review.openstack.org/#/c/457886/ ^ I wonder if this is something we could land by the rc | |
| 13:47:26 | mvenesio | efried: ok i'll do it, thanks | |
| 13:47:27 | rgerganov | efried, the child gets in the cache with my patch and then we don't have a problem (at least not now :) ) | |
| 13:48:20 | efried | rgerganov I must have misunderstood "could you please tell why my patch won't refresh the tree?" | |
| 13:49:38 | rgerganov | efried, I asked that because you said "As written, that change will *not* refresh the tree, though" | |
| 13:49:40 | rgerganov | nevermind | |
| 13:50:08 | efried | rgerganov Okay, yeah, I was wrong. I had missed the 'return uuid' part and was misunderstanding my own code :( | |
| 13:51:06 | efried | rgerganov So your solution will work, but it's skipping a nontrivial optimization that I would like to keep if possible. | |
| 13:52:13 | efried | rgerganov One possible alternative would be for that cache invalidation to in fact kill the whole tree. But the implications of that are probably too far-reaching to be practical. | |
| 13:52:58 | efried | rgerganov Other than that patch, do you have any local code on the series? | |
| 13:53:19 | efried | rgerganov In particular, we should definitely commit whatever test case you're running to hit this problem. | |
| 13:53:30 | openstackgerrit | Merged openstack/nova master: Transform instance.resize_confirm notification https://review.openstack.org/482557 | |
| 13:53:45 | efried | rgerganov Though I would like to do it as part of the update_from_provider_tree patch. | |
| 13:54:13 | rgerganov | efried, so basically the test case should be virt driver adding an incorrect trait | |
| 13:54:33 | efried | rgerganov ...to a child | |
| 13:54:36 | efried | right? | |
| 13:54:39 | rgerganov | yes | |
| 13:55:03 | efried | rgerganov How close to your EOD are you? | |
| 13:55:16 | rgerganov | efried, I will head out in 2 hours | |
| 13:57:15 | efried | rgerganov So I've *almost* got this code path in my test_report_client work in progress. | |
| 13:57:56 | efried | rgerganov I've got one piece setting a bogus trait on the root. And another setting inventory in a bogus *resource class* on a descendant. The latter of which should *probably* have the same effect. | |
| 13:58:21 | efried | rgerganov But since we've identified this exact issue, it wouldn't go amiss to have a small, isolated test case that's separate from that big one. | |
| 13:58:46 | efried | rgerganov Do you have the time/inclination/know-how to write that test case before you leave? | |
| 13:58:57 | rgerganov | efried, I will give it a try | |
| 13:59:05 | efried | rgerganov Thanks! | |
| 13:59:22 | rgerganov | efried, do you want me to update an existing patch or start a new patch? | |
| 14:00:49 | mriedem | alex_xu: i'm fine with checking the version in the Selection object or just checking if the allocation request is the list or dict format and adjusting properly, up to you, but i think we have to handle both cases in queens, we can then probably remove that check later in rocky | |
| 14:02:24 | artom | mriedem, 'morning - did we figure out if the libvirt connection reset errors were a real bug or not? | |
| 14:02:29 | efried | rgerganov New one. We can always slot it into the series, or squash it into the existing commit. | |
| 14:02:57 | efried | rgerganov And that way I can continue working on these test cases locally in parallel. | |
| 14:03:09 | artom | Also - I think I'm using logstash wrong - http://logstash.openstack.org/#dashboard/file/logstash.json?query=Connection%20reset%20by%20peer is turning up nothing for the past month | |
| 14:03:11 | rgerganov | efried, ok cool | |
| 14:03:48 | bauzas | mriedem: when you said "sneaky" in https://review.openstack.org/#/c/535693/, you meant okay for that or not ? | |
| 14:04:15 | Roamer` | actually, yeah, I've been meaning to ask - is there some documentation on using logstash somewhere? I've seen people compose nice queries, like "a message that looks almost like this in this set of files", but I'd like to know more :) | |
| 14:04:15 | mriedem | artom: message:"Connection reset by peer" AND tags:"screen-n-cpu.txt" | |
| 14:04:23 | mriedem | bauzas: it's ok | |
| 14:04:24 | mriedem | just sneaky | |
| 14:04:30 | Roamer` | ah... that's more or less exactly it | |
| 14:04:32 | artom | mriedem, cheers :) | |
| 14:05:07 | mriedem | Roamer`: artom: http://lucene.apache.org/core/4_0_0/queryparser/org/apache/lucene/queryparser/classic/package-summary.html#package_description | |