| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-01-18 | |||
| 16:02:45 | EmilienM | so puppet managed keystone resources at 17h33 but nova compute tries to reach them at 17h31 | |
| 16:02:57 | melwitt | mriedem: looking ... compute/api will call the rpcapi next though | |
| 16:03:12 | EmilienM | bauzas: nova compute should retry later no? | |
| 16:03:32 | bauzas | EmilienM: I'm verifying that | |
| 16:03:48 | bauzas | in theory, we auto-heal every 60 secs | |
| 16:04:18 | EmilienM | even if we could say "tell puppet to create keystone resource BEFORE starting nova compute", this is odd and nova-compute should retry later | |
| 16:04:41 | melwitt | mriedem: I don't think that's calling nova-api though, is it? self.compute_api is an interface to making that rpc call, which will I guess just call the same nova-compute | |
| 16:04:48 | bauzas | EmilienM: we do | |
| 16:05:11 | bauzas | EmilienM: ie. the failing trace I showed you comes from a method that is running every 60 secs | |
| 16:05:15 | openstackgerrit | Artom Lifshitz proposed openstack/nova: Plumbing for tagged nic attachment https://review.openstack.org/415622 | |
| 16:05:16 | openstackgerrit | Artom Lifshitz proposed openstack/nova: Plumbing for tagged volume attachment https://review.openstack.org/391941 | |
| 16:05:16 | openstackgerrit | Artom Lifshitz proposed openstack/nova: Libvirt support for tagged volume attachment https://review.openstack.org/418100 | |
| 16:05:17 | openstackgerrit | Artom Lifshitz proposed openstack/nova: API support for tagged device attachment https://review.openstack.org/376021 | |
| 16:05:18 | openstackgerrit | Artom Lifshitz proposed openstack/nova: Libvirt support for tagged nic attachment https://review.openstack.org/415520 | |
| 16:05:23 | bauzas | EmilienM: and that method is creating the RP in memory | |
| 16:05:50 | bauzas | EmilienM: but I'm trying to identify all the workflow | |
| 16:06:00 | bauzas | so we could possibly have missed something | |
| 16:06:06 | bauzas | in case keystone is back after a flip | |
| 16:06:15 | bauzas | s/flip/blip | |
| 16:10:31 | openstackgerrit | Vasyl Saienko proposed openstack/nova: Ensure we mark baremetal links as phy links https://review.openstack.org/422068 | |
| 16:10:31 | openstackgerrit | Vasyl Saienko proposed openstack/nova: Support Ironic interface attach/detach in nova virt https://review.openstack.org/364413 | |
| 16:12:58 | bauzas | EmilienM: I got a shit ton of "placement API endpoint not found" even after keystone is started http://logs.openstack.org/66/421366/1/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/f138bcc/logs/nova/nova-compute.txt.gz#_2017-01-17_17_38_02_282 | |
| 16:13:14 | markus_z | adreznec: I think I have it up and running http://nova-dpm.readthedocs.io/en/latest/ Thanks again for your time! | |
| 16:13:26 | bauzas | EmilienM: correct me if I'm wrong but you told me that keystone is ready by 17:35 ? | |
| 16:13:47 | bauzas | oh, even 17:33 | |
| 16:13:49 | bauzas | "(17:02:45) EmilienM: so puppet managed keystone resources at 17h33 but nova compute tries to reach them at 17h31" | |
| 16:13:56 | bauzas | that's... weird | |
| 16:14:09 | adreznec | markus_z: Awesome, looks good! | |
| 16:14:15 | bauzas | EmilienM: because it seems we litterally fail to find the correct endpoint even after that | |
| 16:14:17 | openstackgerrit | Eric Fried proposed openstack/nova: WIP(2): Introduce base PowerVM driver (disk drv) https://review.openstack.org/409401 | |
| 16:14:48 | bauzas | EmilienM: ie. keystoneclient returns an EndpointNotFound exception | |
| 16:14:54 | EmilienM | bauzas: 17h33 | |
| 16:15:10 | EmilienM | bauzas: let's see how my endpoint is structured | |
| 16:15:13 | EmilienM | maybe it's wrong | |
| 16:16:23 | EmilienM | bauzas: http://paste.openstack.org/show/H7pXYOKWuBjQROqkZjQm/ | |
| 16:16:50 | EmilienM | cdent: ^ | |
| 16:17:10 | bauzas | EmilienM: maybe the placement service failed to start | |
| 16:17:25 | bauzas | because the URL seems correct | |
| 16:17:27 | EmilienM | bauzas: apache would have failed | |
| 16:17:32 | EmilienM | let's see apache logs | |
| 16:17:39 | cdent | the wsgi process shows up in ps.txt | |
| 16:18:05 | EmilienM | yeah | |
| 16:18:14 | bauzas | I mean, something is wrong with that HTTP call | |
| 16:18:20 | cdent | I'll need to review the wsgi config, but I'm surprised to see: http://127.0.0.1:8778/placement instead of either | |
| 16:18:36 | bauzas | so, either it's the service catalog which is incorrect or the placement service wsgi stack not fully up | |
| 16:18:38 | cdent | http://127.0.0.1/placement or http://127.0.0.1:8778/ | |
| 16:18:42 | cdent | however | |
| 16:18:50 | bauzas | cdent: you're correct, that's the reason | |
| 16:19:05 | bauzas | EmilienM: I need to see your apache conf | |
| 16:19:10 | cdent | except that the exception message is that the endpoint is never found | |
| 16:19:12 | bauzas | because I guess the root cause is ^ | |
| 16:19:20 | openstackgerrit | Gábor Antal proposed openstack/nova: Transform instance.trigger_crash_dump notification https://review.openstack.org/396225 | |
| 16:19:24 | cdent | and the placement-api logs never start, which suggests the wsgi app is never contacted | |
| 16:19:24 | bauzas | cdent: yeah, because it could fail | |
| 16:19:30 | bauzas | I mean getting the VirtualHost | |
| 16:19:35 | bauzas | cdent: exactly | |
| 16:19:46 | bauzas | FWIW http://logs.openstack.org/66/421366/1/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/f138bcc/logs/apache_config/10-placement_wsgi.conf.txt.gz | |
| 16:19:56 | openstackgerrit | Scott DAngelo proposed openstack/nova: Fix Nova to allow using cinder v3 endpoint https://review.openstack.org/422078 | |
| 16:20:23 | bauzas | I think this configuration is incorrect | |
| 16:20:56 | cdent | bauzas: except that is the same as a previous working config, I _think_ | |
| 16:21:02 | cdent | but that ought to be confirmed | |
| 16:21:05 | EmilienM | so if I look at Puppet CI this morning it seems to work but not in TripleO | |
| 16:21:20 | EmilienM | I don't see placement-api logs http://logs.openstack.org/98/417898/15/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/ee7ff1b/logs/overcloud-controller-0/var/log/nova/ | |
| 16:21:44 | EmilienM | but I know the vhost has been created, see http://logs.openstack.org/98/417898/15/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/ee7ff1b/logs/overcloud-controller-0/var/log/httpd/ | |
| 16:22:20 | cdent | the critical point on the placement-api logs is that they will not exist until at least one request is made to the placement API | |
| 16:22:27 | cdent | whether or not it is up | |
| 16:22:37 | EmilienM | ok | |
| 16:22:59 | EmilienM | eg: when you spawn a VM, I guess | |
| 16:23:06 | bauzas | yup | |
| 16:23:15 | bauzas | https://github.com/openstack-dev/devstack/blob/master/files/apache-placement-api.template is the devstack config | |
| 16:23:29 | bauzas | there is an alias that is out of the virtualhost | |
| 16:23:32 | EmilienM | so in tripleo, we spawn overcloud nodes with nova/ironic and i don't see placement logs: http://logs.openstack.org/98/417898/15/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/ee7ff1b/logs/undercloud/var/log/nova/ | |
| 16:23:46 | bauzas | hence why you get either :8778/ or :80/placement | |
| 16:24:08 | bauzas | we share the same wsgi process app, but that's two different directives | |
| 16:24:36 | EmilienM | because we use HAproxy and 80 is used for horizon | |
| 16:24:39 | bauzas | EmilienM: yeah, got the problem | |
| 16:24:52 | bauzas | EmilienM: in devstack, we alias on / for the vhost https://github.com/openstack-dev/devstack/blob/master/files/apache-placement-api.template#L8 | |
| 16:25:12 | bauzas | EmilienM: in http://logs.openstack.org/66/421366/1/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/f138bcc/logs/apache_config/10-placement_wsgi.conf.txt.gz you alias on /placement | |
| 16:25:20 | openstackgerrit | Gábor Antal proposed openstack/nova: Transform instance.trigger_crash_dump notification https://review.openstack.org/396225 | |
| 16:25:22 | bauzas | but you don't provide the location directive | |
| 16:25:24 | EmilienM | yep | |
| 16:25:33 | cdent | if that's the case why did it work before? | |
| 16:25:48 | EmilienM | yeah and I just noticed it works in some CI jobs this morning | |
| 16:25:59 | EmilienM | let me grab the more recent one | |
| 16:26:23 | EmilienM | http://logs.openstack.org/80/421980/3/check/gate-puppet-openstack-integration-4-scenario001-tempest-centos-7/d4b019b/logs/nova/ | |
| 16:26:39 | cdent | I wonder if the problem here is that we don't understand well enough how keystoneauth1 is working, and thus the pathways to that exception we are seeing may be more complex than we know? | |
| 16:28:05 | openstack | Launchpad bug 1656075 in OpenStack Compute (nova) "DiscoveryFailure when trying to get resource providers from the scheduler report client" [Low,Confirmed] - Assigned to Roman Dobosz (roman-dobosz) | |
| 16:28:05 | _gryf | seems similar to https://bugs.launchpad.net/nova/+bug/1656075 | |
| 16:28:13 | bauzas | EmilienM: looks all good for that one http://logs.openstack.org/80/421980/3/check/gate-puppet-openstack-integration-4-scenario001-tempest-centos-7/d4b019b/logs/nova/nova-compute.txt.gz#_2017-01-18_15_44_36_293 | |
| 16:28:50 | EmilienM | ok so good news: it works in Puppet OpenStack CI this morning. Now trying to figure why it doesn't work in TripleO | |
| 16:29:03 | EmilienM | bauzas: I swear it didn't work yesterday | |
| 16:29:08 | bauzas | :) | |
| 16:29:14 | bauzas | I haven't touched anything ! | |
| 16:29:15 | openstack | Launchpad bug 1657467 in tripleo "placement: unable to refresh compute resource provider record" [High,Triaged] | |
| 16:29:15 | EmilienM | if you look at https://bugs.launchpad.net/tripleo/+bug/1657467 | |
| 16:29:21 | EmilienM | logs are from yesterday and it didn't work | |
| 16:29:38 | bauzas | I'd look at the vhost config | |
| 16:29:41 | mriedem | melwitt: yeah good point, nova.compute.manager calling nova.compute.api to call compute rpcapi to cast the | |
| 16:29:42 | bauzas | and compare both | |
| 16:29:45 | mriedem | 'stop' back to the same compute | |