Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-01-18
15:52:50 markus_z ;)
15:52:57 kashyap markus_z: I'm actually burnt looking at it, as it's been dragged for over a year
15:53:00 EmilienM bauzas: that's what log says, not sure it's true.
15:53:10 kashyap (It is spun off from a different review.)
15:53:17 EmilienM bauzas: the keystone resources are ready *after* compute service starts
15:53:30 kashyap markus_z: I think Michele should engage in IRC here if he really wants it in, and solicit feedback, and answer questions in real-time
15:53:36 EmilienM bauzas: dansmith fixed the issue by adding a loop, it worked but now it fails to register in the loop
15:53:38 bauzas EmilienM: mmm, okay
15:53:46 kashyap Too much lag via email I feel.
15:53:58 bauzas EmilienM: yeah, exactly, we changed from a try-once into a recheck
15:54:10 bauzas okay, so it would fail once but works after that
15:54:26 markus_z kashyap: Want me to have a look at it?
15:55:18 kashyap markus_z: If you have time, sure. But I am willing to test it and see what falls through. Just spinning up a new DevStack env with this patch on my slow test machine next to me
15:55:40 EmilienM bauzas: yes, it should (and it did a few days ago) work when puppet creates keystone resources for placement (service, endpoint, etc)
15:55:57 EmilienM bauzas: in a few min, I'll have output of keystone resources
15:56:05 kashyap markus_z: But if you have other priority things, feel free to ignore it. Because, if the submitter like to get this merged, they should engage more actively as well
15:56:11 bauzas EmilienM: what looks weird to me is that we don't have "Updated inventory" logs in http://logs.openstack.org/66/421366/1/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/f138bcc/logs/nova/nova-compute.txt.gz#_2017-01-17_17_32_39_092
15:56:14 markus_z kashyap: I put it on my todo list for tomorrow.
15:56:45 markus_z kashyap: TBH, having a test job for that is far too long on my todo list but I tabled it week after week. :-/
15:56:49 kashyap Sure. markus_z - Just a heads-up: Tomorrow I'll only be working til 13 CET. Then I'll be AFK, travelling the whole day
15:56:57 bauzas EmilienM: just a brief explanation, we basically cache the list of RPs we create in a memory object to reduce the number of HTTP calls
15:57:13 markus_z kashyap: cool, no biggie
15:57:15 kashyap markus_z: You're not alone in feeling that guilt, I know it all too well
15:57:15 bauzas EmilienM: the log you mention is because we can't find the mapping in the cache for the RP
15:57:46 bauzas EmilienM: if the cache key corresponding to the compute is not found, that's probably because we failed updating the placement service, hence that log
15:58:04 bauzas EmilienM: if we would update that placement service, we put the result in the cach
15:58:06 bauzas cache
15:59:11 EmilienM bauzas: what could cause this mapping failing?
16:00:18 bauzas EmilienM: excellent question
16:00:25 mriedem melwitt: so on this auto-register thing for cells, i just remembered that nova-compute actually does already make calls to nova-api https://github.com/openstack/nova/blob/045f08ab8a57b90b00e4777ca71a20b5603c296c/nova/compute/manager.py#L6421
16:00:27 bauzas EmilienM: that I'm trying to answer by looking at the logs
16:00:38 mriedem not even through rpc
16:01:21 bauzas EmilienM: mmmm http://logs.openstack.org/66/421366/1/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/f138bcc/logs/nova/nova-compute.txt.gz#_2017-01-17_17_31_39_862
16:01:46 EmilienM bauzas: I think this one is because the keystone resources are not created yet
16:02:02 EmilienM bauzas: let's see timestamps, gimme a sec
16:02:27 EmilienM http://logs.openstack.org/66/421366/1/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/f138bcc/logs/nova/nova-compute.txt.gz#_2017-01-17_17_31_39_862
16:02:45 EmilienM so puppet managed keystone resources at 17h33 but nova compute tries to reach them at 17h31
16:02:57 melwitt mriedem: looking ... compute/api will call the rpcapi next though
16:03:12 EmilienM bauzas: nova compute should retry later no?
16:03:32 bauzas EmilienM: I'm verifying that
16:03:48 bauzas in theory, we auto-heal every 60 secs
16:04:18 EmilienM even if we could say "tell puppet to create keystone resource BEFORE starting nova compute", this is odd and nova-compute should retry later
16:04:41 melwitt mriedem: I don't think that's calling nova-api though, is it? self.compute_api is an interface to making that rpc call, which will I guess just call the same nova-compute
16:04:48 bauzas EmilienM: we do
16:05:11 bauzas EmilienM: ie. the failing trace I showed you comes from a method that is running every 60 secs
16:05:15 openstackgerrit Artom Lifshitz proposed openstack/nova: Plumbing for tagged nic attachment https://review.openstack.org/415622
16:05:16 openstackgerrit Artom Lifshitz proposed openstack/nova: Plumbing for tagged volume attachment https://review.openstack.org/391941
16:05:16 openstackgerrit Artom Lifshitz proposed openstack/nova: Libvirt support for tagged volume attachment https://review.openstack.org/418100
16:05:17 openstackgerrit Artom Lifshitz proposed openstack/nova: API support for tagged device attachment https://review.openstack.org/376021
16:05:18 openstackgerrit Artom Lifshitz proposed openstack/nova: Libvirt support for tagged nic attachment https://review.openstack.org/415520
16:05:23 bauzas EmilienM: and that method is creating the RP in memory
16:05:50 bauzas EmilienM: but I'm trying to identify all the workflow
16:06:00 bauzas so we could possibly have missed something
16:06:06 bauzas in case keystone is back after a flip
16:06:15 bauzas s/flip/blip
16:10:31 openstackgerrit Vasyl Saienko proposed openstack/nova: Ensure we mark baremetal links as phy links https://review.openstack.org/422068
16:10:31 openstackgerrit Vasyl Saienko proposed openstack/nova: Support Ironic interface attach/detach in nova virt https://review.openstack.org/364413
16:12:58 bauzas EmilienM: I got a shit ton of "placement API endpoint not found" even after keystone is started http://logs.openstack.org/66/421366/1/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/f138bcc/logs/nova/nova-compute.txt.gz#_2017-01-17_17_38_02_282
16:13:14 markus_z adreznec: I think I have it up and running http://nova-dpm.readthedocs.io/en/latest/ Thanks again for your time!
16:13:26 bauzas EmilienM: correct me if I'm wrong but you told me that keystone is ready by 17:35 ?
16:13:47 bauzas oh, even 17:33
16:13:49 bauzas "(17:02:45) EmilienM: so puppet managed keystone resources at 17h33 but nova compute tries to reach them at 17h31"
16:13:56 bauzas that's... weird
16:14:09 adreznec markus_z: Awesome, looks good!
16:14:15 bauzas EmilienM: because it seems we litterally fail to find the correct endpoint even after that
16:14:17 openstackgerrit Eric Fried proposed openstack/nova: WIP(2): Introduce base PowerVM driver (disk drv) https://review.openstack.org/409401
16:14:48 bauzas EmilienM: ie. keystoneclient returns an EndpointNotFound exception
16:14:54 EmilienM bauzas: 17h33
16:15:10 EmilienM bauzas: let's see how my endpoint is structured
16:15:13 EmilienM maybe it's wrong
16:16:23 EmilienM bauzas: http://paste.openstack.org/show/H7pXYOKWuBjQROqkZjQm/
16:16:50 EmilienM cdent: ^
16:17:10 bauzas EmilienM: maybe the placement service failed to start
16:17:25 bauzas because the URL seems correct
16:17:27 EmilienM bauzas: apache would have failed
16:17:32 EmilienM let's see apache logs
16:17:39 cdent the wsgi process shows up in ps.txt
16:18:05 EmilienM yeah
16:18:14 bauzas I mean, something is wrong with that HTTP call
16:18:20 cdent I'll need to review the wsgi config, but I'm surprised to see: http://127.0.0.1:8778/placement instead of either
16:18:36 bauzas so, either it's the service catalog which is incorrect or the placement service wsgi stack not fully up
16:18:38 cdent http://127.0.0.1/placement or http://127.0.0.1:8778/
16:18:42 cdent however
16:18:50 bauzas cdent: you're correct, that's the reason
16:19:05 bauzas EmilienM: I need to see your apache conf
16:19:10 cdent except that the exception message is that the endpoint is never found
16:19:12 bauzas because I guess the root cause is ^
16:19:20 openstackgerrit Gábor Antal proposed openstack/nova: Transform instance.trigger_crash_dump notification https://review.openstack.org/396225
16:19:24 cdent and the placement-api logs never start, which suggests the wsgi app is never contacted
16:19:24 bauzas cdent: yeah, because it could fail
16:19:30 bauzas I mean getting the VirtualHost
16:19:35 bauzas cdent: exactly
16:19:46 bauzas FWIW http://logs.openstack.org/66/421366/1/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/f138bcc/logs/apache_config/10-placement_wsgi.conf.txt.gz
16:19:56 openstackgerrit Scott DAngelo proposed openstack/nova: Fix Nova to allow using cinder v3 endpoint https://review.openstack.org/422078
16:20:23 bauzas I think this configuration is incorrect
16:20:56 cdent bauzas: except that is the same as a previous working config, I _think_
16:21:02 cdent but that ought to be confirmed
16:21:05 EmilienM so if I look at Puppet CI this morning it seems to work but not in TripleO
16:21:20 EmilienM I don't see placement-api logs http://logs.openstack.org/98/417898/15/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/ee7ff1b/logs/overcloud-controller-0/var/log/nova/
16:21:44 EmilienM but I know the vhost has been created, see http://logs.openstack.org/98/417898/15/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/ee7ff1b/logs/overcloud-controller-0/var/log/httpd/

Earlier   Later