| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-07-15 | |||
| 13:07:34 | efried | what service is this for? | |
| 13:08:02 | sean-k-mooney | efried: hongda patches https://review.opendev.org/#/c/669867/1 | |
| 13:08:12 | sean-k-mooney | and this stable only patch https://review.opendev.org/#/c/670016/ | |
| 13:08:21 | sean-k-mooney | its related to https://bugs.launchpad.net/nova/+bug/1647451 | |
| 13:08:21 | openstack | Launchpad bug 1647451 in OpenStack Compute (nova) newton "Post live migration step could fail due to auth errors" [Medium,Fix committed] - Assigned to Lee Yarwood (lyarwood) | |
| 13:09:08 | sean-k-mooney | efried: so instead of juat createing an admin client here https://review.opendev.org/#/c/669867/1/nova/network/neutronv2/api.py we shoudl use the service auth thing right | |
| 13:10:28 | efried | sean-k-mooney: I'm going to have to take a closer look at this a bit later. What's the operation that starts this flow? | |
| 13:10:41 | efried | i.e. why does the user token have the opportunity to expire before list_ports? | |
| 13:11:37 | sean-k-mooney | efried: i ltrally just started looking at this 10 mins ago. but i belive its the admin does openstack server migrate --live near the end of the lifetime of the token | |
| 13:11:51 | sean-k-mooney | and it expires midway | |
| 13:11:57 | openstackgerrit | Merged openstack/python-novaclient master: Remove deprecated methods and properties https://review.opendev.org/667762 | |
| 13:12:00 | sean-k-mooney | or you know the live migration just take a while | |
| 13:12:17 | efried | at a glance, it appears as though the rest of the stuff in this flow is using admin auth | |
| 13:12:20 | sean-k-mooney | in either case it expires by the time it gets to post live migrate | |
| 13:12:45 | efried | but I don't want to discount the possibility that the list_ports is being done under user auth specifically to guard against someone kicking off this operation when they shouldn't. | |
| 13:12:47 | sean-k-mooney | well live-migate is an admin only op so its propably fine | |
| 13:14:01 | efried | okay. There's some more opportunities to reuse the admin client in this flow as well. | |
| 13:14:08 | efried | Let me come back to this after my mtg | |
| 13:14:40 | sean-k-mooney | sure i was more worried about breaking the api request id tracking stuff | |
| 13:15:09 | sean-k-mooney | but i guess that will be preseved in the keystone context so it prably fine | |
| 13:15:20 | sean-k-mooney | *proably | |
| 13:18:17 | efried | sean-k-mooney: If get_client with admin=True is breaking the global request ID, then that's broken all over the place. Would be a separate issue, if it's an issue at all. | |
| 13:18:55 | sean-k-mooney | efried: it proably isnt a problem | |
| 13:19:20 | efried | nope, I'm looking at get_client itself now and it's properly handling the global request ID. | |
| 13:19:27 | sean-k-mooney | but that is just want i wanted to confrim before i +/-1'd and reviewd it properly | |
| 13:19:46 | sean-k-mooney | efried: arent you ment to be in a meeting :P | |
| 13:21:02 | efried | Yeah, if we're happy that this is supposed to be an admin-only flow anyway, then this change makes sense to me. | |
| 13:22:49 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: API microversion 2.75: Add 'power-update' external event https://review.opendev.org/645611 | |
| 13:23:07 | artom | I don't think I'm doing this right | |
| 13:23:30 | artom | I'm trying to see if test_server_connectivity_cold_migration_revert started failing recently-ish | |
| 13:23:30 | sean-k-mooney | life? openstack? irc? | |
| 13:23:36 | artom | sean-k-mooney, I mean, yes, but: | |
| 13:23:42 | artom | Here's my logstash query: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22test_server_connectivity_cold_migration_revert*FAILED%5C%22%20and%20tags%3A%5C%22job-output.txt%5C%22 | |
| 13:24:15 | openstackgerrit | Merged openstack/python-novaclient master: Deprecate cells v1 and extension commands and APIs https://review.opendev.org/669597 | |
| 13:24:16 | openstackgerrit | Merged openstack/python-novaclient master: Add a guide to add a new microversion support https://review.opendev.org/667002 | |
| 13:24:16 | artom | Looks like I got the wildcard wrong - but I need it, because there's a timestamp in there | |
| 13:24:30 | artom | For example: tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_cold_migration_revert [221.049469s] ... FAILED | |
| 13:26:37 | artom | Context is, we may have merged https://review.opendev.org/#/c/663405/ too soon | |
| 13:27:20 | artom | That recently un-skipped test is failing pretty consistently (though not 100%) on https://review.opendev.org/#/c/668631/ | |
| 13:27:31 | openstackgerrit | Surya Seetharaman proposed openstack/python-novaclient master: API microversion 2.75: Add 'power-update' external event https://review.opendev.org/666792 | |
| 13:27:35 | sean-k-mooney | too soon in that its still broken or we need to receck things | |
| 13:27:55 | artom | But everything looks right from the Nova events POV, so maybe there's something else, and we need to re-skip test_server_connectivity_cold_migration_revert | |
| 13:28:33 | artom | But step 1 is determining whether those failures started happening on all runs right after we merged the un-skip patch, or whether it's just my patch that's causing trouble | |
| 13:28:40 | artom | Hence the logstash query | |
| 13:29:06 | sean-k-mooney | ya | |
| 13:29:18 | sean-k-mooney | ill see if i can hack something to work too quickly | |
| 13:31:04 | sean-k-mooney | artom: it look like your query is being ignored more or less | |
| 13:31:18 | artom | sean-k-mooney, I know :( | |
| 13:32:38 | sean-k-mooney | do you have an example failure | |
| 13:33:14 | artom | sean-k-mooney, http://logs.openstack.org/31/668631/7/check/tempest-slow-py3/a16d7d9/job-output.txt.gz#_2019-07-14_15_36_51_145040 | |
| 13:33:37 | sean-k-mooney | thanks | |
| 13:34:10 | artom | The message operator, it's per line, right? | |
| 13:34:41 | sean-k-mooney | yes it should be | |
| 13:35:07 | sean-k-mooney | well yes and no | |
| 13:35:25 | sean-k-mooney | the log stream should chunk it per new line | |
| 13:35:55 | sean-k-mooney | sorry not per line but per log output | |
| 13:36:30 | sean-k-mooney | e.g. if you log multi line message teh log/parser/setream will stream that full log messge to logstash in one go | |
| 13:38:11 | artom | sean-k-mooney, oh, I think it splits on the _ | |
| 13:41:30 | sean-k-mooney | its shouldnt | |
| 13:41:44 | sean-k-mooney | it might be but it shouldn't | |
| 13:41:56 | alex_xu | efried: looking for some feedback on this direction https://review.opendev.org/#/q/status:open+project:openstack/nova+branch:master+topic:claim_for_instance before I'm going futher. | |
| 13:41:59 | efried | artom: (still) not paying a lot of attention here, but there ought to be lots of good examples in the elastic-recheck project | |
| 13:42:16 | artom | efried, ack, thanks | |
| 13:42:43 | efried | alex_xu: Okay, I saw a bunch of patches come in this morning. Were you planning to put up a spec for this? | |
| 13:42:44 | sean-k-mooney | ya https://github.com/openstack-infra/elastic-recheck/tree/master/queries i was looking at those | |
| 13:42:49 | sean-k-mooney | artom: ^ | |
| 13:44:12 | artom | sean-k-mooney, yeah... and I appear to be doing everything right :( | |
| 13:45:13 | sean-k-mooney | i dont see any example of actully match on the tempest output | |
| 13:45:14 | artom | Wait, do I need to capitalize AND? | |
| 13:45:24 | sean-k-mooney | maybe | |
| 13:45:44 | sean-k-mooney | yes | |
| 13:46:42 | artom | *facepalm* | |
| 13:46:48 | artom | OK, now it's turning up nothing | |
| 13:47:14 | sean-k-mooney | artom: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_cold_migration_revert%5C%22%20AND%20message%3A%5C%22FAILED%5C%22%20AND%20tags%3A%5C%22job-output.txt%5C%22 | |
| 13:47:19 | sean-k-mooney | set it to 30days | |
| 13:47:24 | sean-k-mooney | and things show up | |
| 13:48:25 | sean-k-mooney | the last failure was about an hour ago | |
| 13:49:10 | sean-k-mooney | looks like it start to show up on teh 13th | |
| 13:50:24 | artom | sean-k-mooney, that sounds about right | |
| 13:50:56 | artom | Means https://review.opendev.org/#/c/663405/ is causing it | |
| 13:51:47 | stephenfin | melwitt: If you're around today, could you take a look at this doc fix? https://review.opendev.org/#/c/670125/ | |
| 13:52:04 | artom | All on different changes, too | |
| 13:52:19 | sean-k-mooney | the first failure was https://review.opendev.org/#/c/627765/ | |
| 13:53:23 | sean-k-mooney | it might be caused by https://review.opendev.org/#/c/663405/ but it looks like a neutron issue to me | |
| 13:55:01 | artom | sean-k-mooney, it has to be "cause" by https://review.opendev.org/#/c/663405/ because before that landed we just didn't run that test :) | |
| 13:55:51 | alex_xu | efried: if we need a spec, I can write up one | |
| 13:56:00 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Integrate 'pre-commit' https://review.opendev.org/665518 | |
| 13:56:02 | sean-k-mooney | well we dont know that https://review.opendev.org/#/c/663405/ would fix the issue in that test | |
| 13:56:22 | sean-k-mooney | we speculated that it should | |
| 13:56:47 | artom | sean-k-mooney, you mean https://review.opendev.org/#/c/667177/? | |
| 13:56:51 | sean-k-mooney | sorry https://review.opendev.org/#/c/663405/ is the tempest chage | |
| 13:56:59 | artom | Yeah, we thought it'd be fine | |
| 13:57:03 | artom | Apparently not | |
| 13:57:23 | artom | And like I said, I checked logs for 2 failing runs, everything seems OK from the Nova POV | |
| 13:57:26 | sean-k-mooney | right so https://review.opendev.org/#/c/667177/ is likely not enough | |
| 13:57:39 | artom | We correctly wait for plug-time events, we received them, we finish booting up the guest | |
| 13:57:40 | sean-k-mooney | it look like the neutron l3 agent has not correctly set up the floating ip | |
| 13:57:50 | sean-k-mooney | and that is why the ssh connection is not working | |
| 13:58:00 | artom | sean-k-mooney, ping is failing | |
| 13:58:15 | artom | But with no working fip we'd expect both ping and SSH to fail | |