| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-01-18 | |||
| 15:55:40 | EmilienM | bauzas: yes, it should (and it did a few days ago) work when puppet creates keystone resources for placement (service, endpoint, etc) | |
| 15:55:57 | EmilienM | bauzas: in a few min, I'll have output of keystone resources | |
| 15:56:05 | kashyap | markus_z: But if you have other priority things, feel free to ignore it. Because, if the submitter like to get this merged, they should engage more actively as well | |
| 15:56:11 | bauzas | EmilienM: what looks weird to me is that we don't have "Updated inventory" logs in http://logs.openstack.org/66/421366/1/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/f138bcc/logs/nova/nova-compute.txt.gz#_2017-01-17_17_32_39_092 | |
| 15:56:14 | markus_z | kashyap: I put it on my todo list for tomorrow. | |
| 15:56:45 | markus_z | kashyap: TBH, having a test job for that is far too long on my todo list but I tabled it week after week. :-/ | |
| 15:56:49 | kashyap | Sure. markus_z - Just a heads-up: Tomorrow I'll only be working til 13 CET. Then I'll be AFK, travelling the whole day | |
| 15:56:57 | bauzas | EmilienM: just a brief explanation, we basically cache the list of RPs we create in a memory object to reduce the number of HTTP calls | |
| 15:57:13 | markus_z | kashyap: cool, no biggie | |
| 15:57:15 | kashyap | markus_z: You're not alone in feeling that guilt, I know it all too well | |
| 15:57:15 | bauzas | EmilienM: the log you mention is because we can't find the mapping in the cache for the RP | |
| 15:57:46 | bauzas | EmilienM: if the cache key corresponding to the compute is not found, that's probably because we failed updating the placement service, hence that log | |
| 15:58:04 | bauzas | EmilienM: if we would update that placement service, we put the result in the cach | |
| 15:58:06 | bauzas | cache | |
| 15:59:11 | EmilienM | bauzas: what could cause this mapping failing? | |
| 16:00:18 | bauzas | EmilienM: excellent question | |
| 16:00:25 | mriedem | melwitt: so on this auto-register thing for cells, i just remembered that nova-compute actually does already make calls to nova-api https://github.com/openstack/nova/blob/045f08ab8a57b90b00e4777ca71a20b5603c296c/nova/compute/manager.py#L6421 | |
| 16:00:27 | bauzas | EmilienM: that I'm trying to answer by looking at the logs | |
| 16:00:38 | mriedem | not even through rpc | |
| 16:01:21 | bauzas | EmilienM: mmmm http://logs.openstack.org/66/421366/1/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/f138bcc/logs/nova/nova-compute.txt.gz#_2017-01-17_17_31_39_862 | |
| 16:01:46 | EmilienM | bauzas: I think this one is because the keystone resources are not created yet | |
| 16:02:02 | EmilienM | bauzas: let's see timestamps, gimme a sec | |
| 16:02:27 | EmilienM | http://logs.openstack.org/66/421366/1/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/f138bcc/logs/nova/nova-compute.txt.gz#_2017-01-17_17_31_39_862 | |
| 16:02:45 | EmilienM | so puppet managed keystone resources at 17h33 but nova compute tries to reach them at 17h31 | |
| 16:02:57 | melwitt | mriedem: looking ... compute/api will call the rpcapi next though | |
| 16:03:12 | EmilienM | bauzas: nova compute should retry later no? | |
| 16:03:32 | bauzas | EmilienM: I'm verifying that | |
| 16:03:48 | bauzas | in theory, we auto-heal every 60 secs | |
| 16:04:18 | EmilienM | even if we could say "tell puppet to create keystone resource BEFORE starting nova compute", this is odd and nova-compute should retry later | |
| 16:04:41 | melwitt | mriedem: I don't think that's calling nova-api though, is it? self.compute_api is an interface to making that rpc call, which will I guess just call the same nova-compute | |
| 16:04:48 | bauzas | EmilienM: we do | |
| 16:05:11 | bauzas | EmilienM: ie. the failing trace I showed you comes from a method that is running every 60 secs | |
| 16:05:15 | openstackgerrit | Artom Lifshitz proposed openstack/nova: Plumbing for tagged nic attachment https://review.openstack.org/415622 | |
| 16:05:16 | openstackgerrit | Artom Lifshitz proposed openstack/nova: Plumbing for tagged volume attachment https://review.openstack.org/391941 | |
| 16:05:16 | openstackgerrit | Artom Lifshitz proposed openstack/nova: Libvirt support for tagged volume attachment https://review.openstack.org/418100 | |
| 16:05:17 | openstackgerrit | Artom Lifshitz proposed openstack/nova: API support for tagged device attachment https://review.openstack.org/376021 | |
| 16:05:18 | openstackgerrit | Artom Lifshitz proposed openstack/nova: Libvirt support for tagged nic attachment https://review.openstack.org/415520 | |
| 16:05:23 | bauzas | EmilienM: and that method is creating the RP in memory | |
| 16:05:50 | bauzas | EmilienM: but I'm trying to identify all the workflow | |
| 16:06:00 | bauzas | so we could possibly have missed something | |
| 16:06:06 | bauzas | in case keystone is back after a flip | |
| 16:06:15 | bauzas | s/flip/blip | |
| 16:10:31 | openstackgerrit | Vasyl Saienko proposed openstack/nova: Ensure we mark baremetal links as phy links https://review.openstack.org/422068 | |
| 16:10:31 | openstackgerrit | Vasyl Saienko proposed openstack/nova: Support Ironic interface attach/detach in nova virt https://review.openstack.org/364413 | |
| 16:12:58 | bauzas | EmilienM: I got a shit ton of "placement API endpoint not found" even after keystone is started http://logs.openstack.org/66/421366/1/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/f138bcc/logs/nova/nova-compute.txt.gz#_2017-01-17_17_38_02_282 | |
| 16:13:14 | markus_z | adreznec: I think I have it up and running http://nova-dpm.readthedocs.io/en/latest/ Thanks again for your time! | |
| 16:13:26 | bauzas | EmilienM: correct me if I'm wrong but you told me that keystone is ready by 17:35 ? | |
| 16:13:47 | bauzas | oh, even 17:33 | |
| 16:13:49 | bauzas | "(17:02:45) EmilienM: so puppet managed keystone resources at 17h33 but nova compute tries to reach them at 17h31" | |
| 16:13:56 | bauzas | that's... weird | |
| 16:14:09 | adreznec | markus_z: Awesome, looks good! | |
| 16:14:15 | bauzas | EmilienM: because it seems we litterally fail to find the correct endpoint even after that | |
| 16:14:17 | openstackgerrit | Eric Fried proposed openstack/nova: WIP(2): Introduce base PowerVM driver (disk drv) https://review.openstack.org/409401 | |
| 16:14:48 | bauzas | EmilienM: ie. keystoneclient returns an EndpointNotFound exception | |
| 16:14:54 | EmilienM | bauzas: 17h33 | |
| 16:15:10 | EmilienM | bauzas: let's see how my endpoint is structured | |
| 16:15:13 | EmilienM | maybe it's wrong | |
| 16:16:23 | EmilienM | bauzas: http://paste.openstack.org/show/H7pXYOKWuBjQROqkZjQm/ | |
| 16:16:50 | EmilienM | cdent: ^ | |
| 16:17:10 | bauzas | EmilienM: maybe the placement service failed to start | |
| 16:17:25 | bauzas | because the URL seems correct | |
| 16:17:27 | EmilienM | bauzas: apache would have failed | |
| 16:17:32 | EmilienM | let's see apache logs | |
| 16:17:39 | cdent | the wsgi process shows up in ps.txt | |
| 16:18:05 | EmilienM | yeah | |
| 16:18:14 | bauzas | I mean, something is wrong with that HTTP call | |
| 16:18:20 | cdent | I'll need to review the wsgi config, but I'm surprised to see: http://127.0.0.1:8778/placement instead of either | |
| 16:18:36 | bauzas | so, either it's the service catalog which is incorrect or the placement service wsgi stack not fully up | |
| 16:18:38 | cdent | http://127.0.0.1/placement or http://127.0.0.1:8778/ | |
| 16:18:42 | cdent | however | |
| 16:18:50 | bauzas | cdent: you're correct, that's the reason | |
| 16:19:05 | bauzas | EmilienM: I need to see your apache conf | |
| 16:19:10 | cdent | except that the exception message is that the endpoint is never found | |
| 16:19:12 | bauzas | because I guess the root cause is ^ | |
| 16:19:20 | openstackgerrit | Gábor Antal proposed openstack/nova: Transform instance.trigger_crash_dump notification https://review.openstack.org/396225 | |
| 16:19:24 | cdent | and the placement-api logs never start, which suggests the wsgi app is never contacted | |
| 16:19:24 | bauzas | cdent: yeah, because it could fail | |
| 16:19:30 | bauzas | I mean getting the VirtualHost | |
| 16:19:35 | bauzas | cdent: exactly | |
| 16:19:46 | bauzas | FWIW http://logs.openstack.org/66/421366/1/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/f138bcc/logs/apache_config/10-placement_wsgi.conf.txt.gz | |
| 16:19:56 | openstackgerrit | Scott DAngelo proposed openstack/nova: Fix Nova to allow using cinder v3 endpoint https://review.openstack.org/422078 | |
| 16:20:23 | bauzas | I think this configuration is incorrect | |
| 16:20:56 | cdent | bauzas: except that is the same as a previous working config, I _think_ | |
| 16:21:02 | cdent | but that ought to be confirmed | |
| 16:21:05 | EmilienM | so if I look at Puppet CI this morning it seems to work but not in TripleO | |
| 16:21:20 | EmilienM | I don't see placement-api logs http://logs.openstack.org/98/417898/15/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/ee7ff1b/logs/overcloud-controller-0/var/log/nova/ | |
| 16:21:44 | EmilienM | but I know the vhost has been created, see http://logs.openstack.org/98/417898/15/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/ee7ff1b/logs/overcloud-controller-0/var/log/httpd/ | |
| 16:22:20 | cdent | the critical point on the placement-api logs is that they will not exist until at least one request is made to the placement API | |
| 16:22:27 | cdent | whether or not it is up | |
| 16:22:37 | EmilienM | ok | |
| 16:22:59 | EmilienM | eg: when you spawn a VM, I guess | |
| 16:23:06 | bauzas | yup | |
| 16:23:15 | bauzas | https://github.com/openstack-dev/devstack/blob/master/files/apache-placement-api.template is the devstack config | |
| 16:23:29 | bauzas | there is an alias that is out of the virtualhost | |
| 16:23:32 | EmilienM | so in tripleo, we spawn overcloud nodes with nova/ironic and i don't see placement logs: http://logs.openstack.org/98/417898/15/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/ee7ff1b/logs/undercloud/var/log/nova/ | |
| 16:23:46 | bauzas | hence why you get either :8778/ or :80/placement | |
| 16:24:08 | bauzas | we share the same wsgi process app, but that's two different directives | |
| 16:24:36 | EmilienM | because we use HAproxy and 80 is used for horizon | |
| 16:24:39 | bauzas | EmilienM: yeah, got the problem | |
| 16:24:52 | bauzas | EmilienM: in devstack, we alias on / for the vhost https://github.com/openstack-dev/devstack/blob/master/files/apache-placement-api.template#L8 | |