| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-07-15 | |||
| 10:28:13 | sean-k-mooney | it is not direcly using the fixtre but ill look at its inheritance tree | |
| 10:28:26 | sean-k-mooney | maybe we dont use it in the functional tests | |
| 10:29:37 | gibi | sean-k-mooney: does not this solved the problem https://review.opendev.org/#/c/656844/ ? | |
| 10:30:56 | sean-k-mooney | i can check if we have that on osp 15 which is stine but migi was saying the backport he has tried sofar do not fix it for us | |
| 10:31:28 | sean-k-mooney | we dont have auto import of backports so its possible we are missing that one | |
| 10:36:17 | sean-k-mooney | gibi: ya we have that and no it does not fix it for us | |
| 10:43:45 | sean-k-mooney | cdent: looks like the waring filter is installed for the functional tests | |
| 10:43:54 | sean-k-mooney | so i guess we are still looking too much | |
| 10:44:21 | sean-k-mooney | part of the issue is i think stestr is creating a log stream per worker not per test | |
| 12:56:31 | hongda | Hello everyone. Can you help to review: https://review.opendev.org/#/c/670016/ and https://review.opendev.org/#/c/669867/ ? They tried to fix live-migration failure when token expires. Thanks a lot. XD | |
| 13:00:46 | sean-k-mooney | we would have to fix this on master first before https://review.opendev.org/#/c/670016/ can be applied | |
| 13:00:54 | sean-k-mooney | why is that stable only by the way | |
| 13:01:54 | efried | alex_xu: Talk to me | |
| 13:05:34 | sean-k-mooney | hongda: im not sure we should just blindly use the admin context | |
| 13:06:10 | sean-k-mooney | we could be it would proably be better to only use the admin context if the token had expired | |
| 13:07:15 | sean-k-mooney | hongda: it also looks like you are reusing an old bug that was closed in 2017 | |
| 13:07:24 | efried | sean-k-mooney: I'm not paying much attention here, but service auth was made for a situation where you start off with a user token for a long-running operation and then it expires somewhere in the middle. | |
| 13:07:29 | sean-k-mooney | hongda: it would be better to file a new noen and reference it | |
| 13:07:30 | efried | wrap the user auth in a service auth and you're good. | |
| 13:07:34 | efried | what service is this for? | |
| 13:08:02 | sean-k-mooney | efried: hongda patches https://review.opendev.org/#/c/669867/1 | |
| 13:08:12 | sean-k-mooney | and this stable only patch https://review.opendev.org/#/c/670016/ | |
| 13:08:21 | openstack | Launchpad bug 1647451 in OpenStack Compute (nova) newton "Post live migration step could fail due to auth errors" [Medium,Fix committed] - Assigned to Lee Yarwood (lyarwood) | |
| 13:08:21 | sean-k-mooney | its related to https://bugs.launchpad.net/nova/+bug/1647451 | |
| 13:09:08 | sean-k-mooney | efried: so instead of juat createing an admin client here https://review.opendev.org/#/c/669867/1/nova/network/neutronv2/api.py we shoudl use the service auth thing right | |
| 13:10:28 | efried | sean-k-mooney: I'm going to have to take a closer look at this a bit later. What's the operation that starts this flow? | |
| 13:10:41 | efried | i.e. why does the user token have the opportunity to expire before list_ports? | |
| 13:11:37 | sean-k-mooney | efried: i ltrally just started looking at this 10 mins ago. but i belive its the admin does openstack server migrate --live near the end of the lifetime of the token | |
| 13:11:51 | sean-k-mooney | and it expires midway | |
| 13:11:57 | openstackgerrit | Merged openstack/python-novaclient master: Remove deprecated methods and properties https://review.opendev.org/667762 | |
| 13:12:00 | sean-k-mooney | or you know the live migration just take a while | |
| 13:12:17 | efried | at a glance, it appears as though the rest of the stuff in this flow is using admin auth | |
| 13:12:20 | sean-k-mooney | in either case it expires by the time it gets to post live migrate | |
| 13:12:45 | efried | but I don't want to discount the possibility that the list_ports is being done under user auth specifically to guard against someone kicking off this operation when they shouldn't. | |
| 13:12:47 | sean-k-mooney | well live-migate is an admin only op so its propably fine | |
| 13:14:01 | efried | okay. There's some more opportunities to reuse the admin client in this flow as well. | |
| 13:14:08 | efried | Let me come back to this after my mtg | |
| 13:14:40 | sean-k-mooney | sure i was more worried about breaking the api request id tracking stuff | |
| 13:15:09 | sean-k-mooney | but i guess that will be preseved in the keystone context so it prably fine | |
| 13:15:20 | sean-k-mooney | *proably | |
| 13:18:17 | efried | sean-k-mooney: If get_client with admin=True is breaking the global request ID, then that's broken all over the place. Would be a separate issue, if it's an issue at all. | |
| 13:18:55 | sean-k-mooney | efried: it proably isnt a problem | |
| 13:19:20 | efried | nope, I'm looking at get_client itself now and it's properly handling the global request ID. | |
| 13:19:27 | sean-k-mooney | but that is just want i wanted to confrim before i +/-1'd and reviewd it properly | |
| 13:19:46 | sean-k-mooney | efried: arent you ment to be in a meeting :P | |
| 13:21:02 | efried | Yeah, if we're happy that this is supposed to be an admin-only flow anyway, then this change makes sense to me. | |
| 13:22:49 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: API microversion 2.75: Add 'power-update' external event https://review.opendev.org/645611 | |
| 13:23:07 | artom | I don't think I'm doing this right | |
| 13:23:30 | sean-k-mooney | life? openstack? irc? | |
| 13:23:30 | artom | I'm trying to see if test_server_connectivity_cold_migration_revert started failing recently-ish | |
| 13:23:36 | artom | sean-k-mooney, I mean, yes, but: | |
| 13:23:42 | artom | Here's my logstash query: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22test_server_connectivity_cold_migration_revert*FAILED%5C%22%20and%20tags%3A%5C%22job-output.txt%5C%22 | |
| 13:24:15 | openstackgerrit | Merged openstack/python-novaclient master: Deprecate cells v1 and extension commands and APIs https://review.opendev.org/669597 | |
| 13:24:16 | artom | Looks like I got the wildcard wrong - but I need it, because there's a timestamp in there | |
| 13:24:16 | openstackgerrit | Merged openstack/python-novaclient master: Add a guide to add a new microversion support https://review.opendev.org/667002 | |
| 13:24:30 | artom | For example: tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_cold_migration_revert [221.049469s] ... FAILED | |
| 13:26:37 | artom | Context is, we may have merged https://review.opendev.org/#/c/663405/ too soon | |
| 13:27:20 | artom | That recently un-skipped test is failing pretty consistently (though not 100%) on https://review.opendev.org/#/c/668631/ | |
| 13:27:31 | openstackgerrit | Surya Seetharaman proposed openstack/python-novaclient master: API microversion 2.75: Add 'power-update' external event https://review.opendev.org/666792 | |
| 13:27:35 | sean-k-mooney | too soon in that its still broken or we need to receck things | |
| 13:27:55 | artom | But everything looks right from the Nova events POV, so maybe there's something else, and we need to re-skip test_server_connectivity_cold_migration_revert | |
| 13:28:33 | artom | But step 1 is determining whether those failures started happening on all runs right after we merged the un-skip patch, or whether it's just my patch that's causing trouble | |
| 13:28:40 | artom | Hence the logstash query | |
| 13:29:06 | sean-k-mooney | ya | |
| 13:29:18 | sean-k-mooney | ill see if i can hack something to work too quickly | |
| 13:31:04 | sean-k-mooney | artom: it look like your query is being ignored more or less | |
| 13:31:18 | artom | sean-k-mooney, I know :( | |
| 13:32:38 | sean-k-mooney | do you have an example failure | |
| 13:33:14 | artom | sean-k-mooney, http://logs.openstack.org/31/668631/7/check/tempest-slow-py3/a16d7d9/job-output.txt.gz#_2019-07-14_15_36_51_145040 | |
| 13:33:37 | sean-k-mooney | thanks | |
| 13:34:10 | artom | The message operator, it's per line, right? | |
| 13:34:41 | sean-k-mooney | yes it should be | |
| 13:35:07 | sean-k-mooney | well yes and no | |
| 13:35:25 | sean-k-mooney | the log stream should chunk it per new line | |
| 13:35:55 | sean-k-mooney | sorry not per line but per log output | |
| 13:36:30 | sean-k-mooney | e.g. if you log multi line message teh log/parser/setream will stream that full log messge to logstash in one go | |
| 13:38:11 | artom | sean-k-mooney, oh, I think it splits on the _ | |
| 13:41:30 | sean-k-mooney | its shouldnt | |
| 13:41:44 | sean-k-mooney | it might be but it shouldn't | |
| 13:41:56 | alex_xu | efried: looking for some feedback on this direction https://review.opendev.org/#/q/status:open+project:openstack/nova+branch:master+topic:claim_for_instance before I'm going futher. | |
| 13:41:59 | efried | artom: (still) not paying a lot of attention here, but there ought to be lots of good examples in the elastic-recheck project | |
| 13:42:16 | artom | efried, ack, thanks | |
| 13:42:43 | efried | alex_xu: Okay, I saw a bunch of patches come in this morning. Were you planning to put up a spec for this? | |
| 13:42:44 | sean-k-mooney | ya https://github.com/openstack-infra/elastic-recheck/tree/master/queries i was looking at those | |
| 13:42:49 | sean-k-mooney | artom: ^ | |
| 13:44:12 | artom | sean-k-mooney, yeah... and I appear to be doing everything right :( | |
| 13:45:13 | sean-k-mooney | i dont see any example of actully match on the tempest output | |
| 13:45:14 | artom | Wait, do I need to capitalize AND? | |
| 13:45:24 | sean-k-mooney | maybe | |
| 13:45:44 | sean-k-mooney | yes | |
| 13:46:42 | artom | *facepalm* | |
| 13:46:48 | artom | OK, now it's turning up nothing | |
| 13:47:14 | sean-k-mooney | artom: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_cold_migration_revert%5C%22%20AND%20message%3A%5C%22FAILED%5C%22%20AND%20tags%3A%5C%22job-output.txt%5C%22 | |
| 13:47:19 | sean-k-mooney | set it to 30days | |
| 13:47:24 | sean-k-mooney | and things show up | |
| 13:48:25 | sean-k-mooney | the last failure was about an hour ago | |
| 13:49:10 | sean-k-mooney | looks like it start to show up on teh 13th | |
| 13:50:24 | artom | sean-k-mooney, that sounds about right | |
| 13:50:56 | artom | Means https://review.opendev.org/#/c/663405/ is causing it | |
| 13:51:47 | stephenfin | melwitt: If you're around today, could you take a look at this doc fix? https://review.opendev.org/#/c/670125/ | |