| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-07-15 | |||
| 10:06:12 | sean-k-mooney | it is in subunit parser | |
| 10:06:20 | sean-k-mooney | but the subunit protocal | |
| 10:06:33 | cdent | so finding and turning off the deprecation warnings (or simply fixing them) is probably the easiest thing | |
| 10:06:35 | sean-k-mooney | allows large packets to be fragmented | |
| 10:06:47 | sean-k-mooney | i think the issue is we are not fragmenting the stream | |
| 10:07:04 | sean-k-mooney | cdent: ya that is how we are fixing it upstream | |
| 10:07:18 | sean-k-mooney | but downstream that is more of a challange | |
| 10:07:43 | sean-k-mooney | it either requires backporting things that may not be backported upstream or downstream only patches | |
| 10:13:35 | sean-k-mooney | cdent: i was wondering if there was a way to redirect all deperecation warnings to a seperate file so that it does not end up in the test output | |
| 10:14:05 | sean-k-mooney | we should be able to install a python logging filter i think but i havent tried | |
| 10:15:15 | cdent | yeah, that does appear to be the case | |
| 10:18:43 | sean-k-mooney | the trick would be using a generic enough regex to match deprecations without redirecting real errors. we would also want the gate jobs to copy the deprecation file so that we can actully track how big it is and squash them | |
| 10:22:57 | sean-k-mooney | it lookse like we already filter a bunch of waring here https://github.com/openstack/nova/blob/master/nova/tests/fixtures.py#L815-L877 | |
| 10:24:02 | sean-k-mooney | cdent: actully https://github.com/openstack/nova/blob/master/nova/tests/fixtures.py#L820 should be makeing deprecation warning print once | |
| 10:24:10 | sean-k-mooney | but maybe that is per test? | |
| 10:24:49 | sean-k-mooney | i would have assumed it would be per logger but maybe we dont use the warning filter in all places we should | |
| 10:26:10 | cdent | could be that it is not turned on everywhere? there are some tests that don't use the usual base? | |
| 10:27:34 | sean-k-mooney | the test that is exploding downswtream is the test_instace_action functional test which runns a bunch of tests interall https://github.com/openstack/nova/blob/7279d6fa009c6e276188bcad0ad5a1832849a4f9/nova/tests/functional/notification_sample_tests/test_instance.py#L357-L377 | |
| 10:28:13 | sean-k-mooney | it is not direcly using the fixtre but ill look at its inheritance tree | |
| 10:28:26 | sean-k-mooney | maybe we dont use it in the functional tests | |
| 10:29:37 | gibi | sean-k-mooney: does not this solved the problem https://review.opendev.org/#/c/656844/ ? | |
| 10:30:56 | sean-k-mooney | i can check if we have that on osp 15 which is stine but migi was saying the backport he has tried sofar do not fix it for us | |
| 10:31:28 | sean-k-mooney | we dont have auto import of backports so its possible we are missing that one | |
| 10:36:17 | sean-k-mooney | gibi: ya we have that and no it does not fix it for us | |
| 10:43:45 | sean-k-mooney | cdent: looks like the waring filter is installed for the functional tests | |
| 10:43:54 | sean-k-mooney | so i guess we are still looking too much | |
| 10:44:21 | sean-k-mooney | part of the issue is i think stestr is creating a log stream per worker not per test | |
| 12:56:31 | hongda | Hello everyone. Can you help to review: https://review.opendev.org/#/c/670016/ and https://review.opendev.org/#/c/669867/ ? They tried to fix live-migration failure when token expires. Thanks a lot. XD | |
| 13:00:46 | sean-k-mooney | we would have to fix this on master first before https://review.opendev.org/#/c/670016/ can be applied | |
| 13:00:54 | sean-k-mooney | why is that stable only by the way | |
| 13:01:54 | efried | alex_xu: Talk to me | |
| 13:05:34 | sean-k-mooney | hongda: im not sure we should just blindly use the admin context | |
| 13:06:10 | sean-k-mooney | we could be it would proably be better to only use the admin context if the token had expired | |
| 13:07:15 | sean-k-mooney | hongda: it also looks like you are reusing an old bug that was closed in 2017 | |
| 13:07:24 | efried | sean-k-mooney: I'm not paying much attention here, but service auth was made for a situation where you start off with a user token for a long-running operation and then it expires somewhere in the middle. | |
| 13:07:29 | sean-k-mooney | hongda: it would be better to file a new noen and reference it | |
| 13:07:30 | efried | wrap the user auth in a service auth and you're good. | |
| 13:07:34 | efried | what service is this for? | |
| 13:08:02 | sean-k-mooney | efried: hongda patches https://review.opendev.org/#/c/669867/1 | |
| 13:08:12 | sean-k-mooney | and this stable only patch https://review.opendev.org/#/c/670016/ | |
| 13:08:21 | openstack | Launchpad bug 1647451 in OpenStack Compute (nova) newton "Post live migration step could fail due to auth errors" [Medium,Fix committed] - Assigned to Lee Yarwood (lyarwood) | |
| 13:08:21 | sean-k-mooney | its related to https://bugs.launchpad.net/nova/+bug/1647451 | |
| 13:09:08 | sean-k-mooney | efried: so instead of juat createing an admin client here https://review.opendev.org/#/c/669867/1/nova/network/neutronv2/api.py we shoudl use the service auth thing right | |
| 13:10:28 | efried | sean-k-mooney: I'm going to have to take a closer look at this a bit later. What's the operation that starts this flow? | |
| 13:10:41 | efried | i.e. why does the user token have the opportunity to expire before list_ports? | |
| 13:11:37 | sean-k-mooney | efried: i ltrally just started looking at this 10 mins ago. but i belive its the admin does openstack server migrate --live near the end of the lifetime of the token | |
| 13:11:51 | sean-k-mooney | and it expires midway | |
| 13:11:57 | openstackgerrit | Merged openstack/python-novaclient master: Remove deprecated methods and properties https://review.opendev.org/667762 | |
| 13:12:00 | sean-k-mooney | or you know the live migration just take a while | |
| 13:12:17 | efried | at a glance, it appears as though the rest of the stuff in this flow is using admin auth | |
| 13:12:20 | sean-k-mooney | in either case it expires by the time it gets to post live migrate | |
| 13:12:45 | efried | but I don't want to discount the possibility that the list_ports is being done under user auth specifically to guard against someone kicking off this operation when they shouldn't. | |
| 13:12:47 | sean-k-mooney | well live-migate is an admin only op so its propably fine | |
| 13:14:01 | efried | okay. There's some more opportunities to reuse the admin client in this flow as well. | |
| 13:14:08 | efried | Let me come back to this after my mtg | |
| 13:14:40 | sean-k-mooney | sure i was more worried about breaking the api request id tracking stuff | |
| 13:15:09 | sean-k-mooney | but i guess that will be preseved in the keystone context so it prably fine | |
| 13:15:20 | sean-k-mooney | *proably | |
| 13:18:17 | efried | sean-k-mooney: If get_client with admin=True is breaking the global request ID, then that's broken all over the place. Would be a separate issue, if it's an issue at all. | |
| 13:18:55 | sean-k-mooney | efried: it proably isnt a problem | |
| 13:19:20 | efried | nope, I'm looking at get_client itself now and it's properly handling the global request ID. | |
| 13:19:27 | sean-k-mooney | but that is just want i wanted to confrim before i +/-1'd and reviewd it properly | |
| 13:19:46 | sean-k-mooney | efried: arent you ment to be in a meeting :P | |
| 13:21:02 | efried | Yeah, if we're happy that this is supposed to be an admin-only flow anyway, then this change makes sense to me. | |
| 13:22:49 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: API microversion 2.75: Add 'power-update' external event https://review.opendev.org/645611 | |
| 13:23:07 | artom | I don't think I'm doing this right | |
| 13:23:30 | sean-k-mooney | life? openstack? irc? | |
| 13:23:30 | artom | I'm trying to see if test_server_connectivity_cold_migration_revert started failing recently-ish | |
| 13:23:36 | artom | sean-k-mooney, I mean, yes, but: | |
| 13:23:42 | artom | Here's my logstash query: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22test_server_connectivity_cold_migration_revert*FAILED%5C%22%20and%20tags%3A%5C%22job-output.txt%5C%22 | |
| 13:24:15 | openstackgerrit | Merged openstack/python-novaclient master: Deprecate cells v1 and extension commands and APIs https://review.opendev.org/669597 | |
| 13:24:16 | artom | Looks like I got the wildcard wrong - but I need it, because there's a timestamp in there | |
| 13:24:16 | openstackgerrit | Merged openstack/python-novaclient master: Add a guide to add a new microversion support https://review.opendev.org/667002 | |
| 13:24:30 | artom | For example: tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_cold_migration_revert [221.049469s] ... FAILED | |
| 13:26:37 | artom | Context is, we may have merged https://review.opendev.org/#/c/663405/ too soon | |
| 13:27:20 | artom | That recently un-skipped test is failing pretty consistently (though not 100%) on https://review.opendev.org/#/c/668631/ | |
| 13:27:31 | openstackgerrit | Surya Seetharaman proposed openstack/python-novaclient master: API microversion 2.75: Add 'power-update' external event https://review.opendev.org/666792 | |
| 13:27:35 | sean-k-mooney | too soon in that its still broken or we need to receck things | |
| 13:27:55 | artom | But everything looks right from the Nova events POV, so maybe there's something else, and we need to re-skip test_server_connectivity_cold_migration_revert | |
| 13:28:33 | artom | But step 1 is determining whether those failures started happening on all runs right after we merged the un-skip patch, or whether it's just my patch that's causing trouble | |
| 13:28:40 | artom | Hence the logstash query | |
| 13:29:06 | sean-k-mooney | ya | |
| 13:29:18 | sean-k-mooney | ill see if i can hack something to work too quickly | |
| 13:31:04 | sean-k-mooney | artom: it look like your query is being ignored more or less | |
| 13:31:18 | artom | sean-k-mooney, I know :( | |
| 13:32:38 | sean-k-mooney | do you have an example failure | |
| 13:33:14 | artom | sean-k-mooney, http://logs.openstack.org/31/668631/7/check/tempest-slow-py3/a16d7d9/job-output.txt.gz#_2019-07-14_15_36_51_145040 | |
| 13:33:37 | sean-k-mooney | thanks | |
| 13:34:10 | artom | The message operator, it's per line, right? | |
| 13:34:41 | sean-k-mooney | yes it should be | |
| 13:35:07 | sean-k-mooney | well yes and no | |
| 13:35:25 | sean-k-mooney | the log stream should chunk it per new line | |
| 13:35:55 | sean-k-mooney | sorry not per line but per log output | |
| 13:36:30 | sean-k-mooney | e.g. if you log multi line message teh log/parser/setream will stream that full log messge to logstash in one go | |
| 13:38:11 | artom | sean-k-mooney, oh, I think it splits on the _ | |
| 13:41:30 | sean-k-mooney | its shouldnt | |
| 13:41:44 | sean-k-mooney | it might be but it shouldn't | |
| 13:41:56 | alex_xu | efried: looking for some feedback on this direction https://review.opendev.org/#/q/status:open+project:openstack/nova+branch:master+topic:claim_for_instance before I'm going futher. | |
| 13:41:59 | efried | artom: (still) not paying a lot of attention here, but there ought to be lots of good examples in the elastic-recheck project | |
| 13:42:16 | artom | efried, ack, thanks | |