Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-02-09
14:39:41 openstackgerrit Stephen Finucane proposed openstack/nova master: conf: Deprecate 'firewall_driver' https://review.openstack.org/412390
14:39:42 openstackgerrit Stephen Finucane proposed openstack/nova master: conf: Deprecate 'send_arp_for_ha', 'send_arp_for_ha_count' https://review.openstack.org/431021
14:40:18 cdent EmilienM: Looks like a similar situation to what we were seeing before: n-cpu is not finding the placement endpoint, never records inventory for the compute-node. When n-scheduler starts talking to the placement-api that's the first time it is being talked to, but there's no data there
14:40:43 EmilienM cdent: what is weird is that it passed on the other scenario
14:40:48 EmilienM cdent: which means we're racy now
14:41:04 EmilienM cdent: all our scenarios deploy nova/placement the same way
14:41:15 sfinucan johnthetubaguy, sdague: I think the first 10 or so of those are noddy but helpful nonetheless. Would appreciate ye taking a glance and (hopefully) sending them on their way
14:41:17 EmilienM owalsh: ^ fyi
14:41:42 EmilienM cdent: i'm currently investigating puppet catalog and resources orchestration
14:42:12 sfinucan They should clean up and mark virtually all of the nova-network opts for future removal, while the latter ones untangle the repurposed nova-net opts
14:42:14 cdent EmilienM: it would be great if we could get a summary, with times, of when things happen, because trying to inspect multiple logs at once is proving rather difficult
14:42:37 johnthetubaguy cdent: I wondering if we only try fetching the service catalog once? would we log something for that?
14:43:12 EmilienM cdent: sure let me prepare that: etherpad.openstack.org/p/puppet-nova-trunk-debug
14:43:40 cdent johnthetubaguy: I don't know. This is part of why I've been saying that I think our understanding of keystoneauth1 is incomplete. I personally haven't had a chance to fix that for myself yet.
14:44:43 johnthetubaguy cdent: yeah, I have been digging in there recently for the service token stuff, I know I keep getting lost in there still
14:46:40 EmilienM cdent: I tried to grep useful things : http://paste.fedoraproject.org/551748/66515901/
14:47:22 EmilienM cdent: if it's too noisy I can reduce the file
14:47:36 EmilienM cdent: so that file is the catalog of a job that failed
14:49:39 openstackgerrit Takashi NATSUME proposed openstack/nova master: Enable cold migration with target host(1/2) https://review.openstack.org/408955
14:50:20 openstackgerrit Takashi NATSUME proposed openstack/nova master: Enable cold migration with target host(2/2) https://review.openstack.org/408964
14:50:34 EmilienM cdent: I reduced the catalog: http://paste.fedoraproject.org/551757/86651825/
14:50:39 openstackgerrit Takashi NATSUME proposed openstack/nova master: Add functional tests for cold migration to same host https://review.openstack.org/414926
14:50:57 openstackgerrit Takashi NATSUME proposed openstack/nova master: api-ref: Add parameters in cold migrate action https://review.openstack.org/410042
14:51:26 cdent EmilienM: I'm not sure that providing a log of the puppet activity (that is what that is isn't it?0 really tells us what we need to know. We need to understand the interactions of the various services, in a timeline, which means extracting signal from the logs of the various services, unforutnately. What I mean is:
14:52:50 EmilienM cdent: what gives the catalog is timestamps of when service run
14:52:56 cdent at time X something is registered in the catalog, at time Y something starts, at time Z n-cpu tries to talk to placement for the first time, at time A the placement service records its first hit, at time B n-cpu asks for a service catalog
14:52:59 EmilienM cdent: for ex, you can see when we configure cells, etc
14:53:01 cdent when the services _run_ is not enough
14:53:27 cdent it's when the actions happen that we're trying to detangle, isn't it? I'm not entirely sure.
14:53:36 EmilienM right
14:57:37 sdague the service catalog is embedded in the keystone token
14:58:03 mordred keypairs are per-AZ?
14:58:39 sdague johnthetubaguy / cdent perhaps there is a placement service call before it's registered?
14:58:45 EmilienM cdent: damn, now we promoted tripleo CI, I also see jobs flapping with placement service
14:58:53 EmilienM cdent: it's working sometimes but failing very often
14:59:25 nicolasbock Hi, I am trying to setup the placement-api
14:59:35 cdent sdague: would it not "try again"?
14:59:53 nicolasbock according to http://docs.openstack.org/developer/nova/placement.html there should be a wsgi script somewhere in the package
14:59:57 sdague cdent: while the token is valid?
15:00:15 sdague I thought typically for service users that token is only rotated on expiration
15:00:16 nicolasbock but I can't find that script
15:00:28 nicolasbock could someone give me a pointer where I can find it?
15:00:29 cdent sdague: i don't know, see above about not knowing enough
15:00:59 sdague mordred: there is nothing I know about in the keypair data model that makes it per AZ
15:01:28 johnthetubaguy mordred: +1 what sdague said, its per region, AFAIK
15:01:43 mordred cool. that's what I thought
15:01:58 mordred I got this bug report https://github.com/ansible/ansible/issues/21191 and it seemed crazy to me - I'll dig more, thanks
15:02:04 dansmith yep, and stored with the instance now, so not per cell either
15:02:12 EmilienM nicolasbock: /usr/bin/nova-placement-api
15:02:23 nicolasbock EmilienM, ah, cool!
15:02:25 cdent nicolasbock: a file called nova-placement-api will be automatically created as part of the install process. that is the wsgi script. where it ends up depends on your python setup
15:02:33 cdent aw, EmilienM beat me to it
15:02:39 nicolasbock cdent, thanks :)
15:02:53 sdague mordred: http://developer.openstack.org/api-ref/compute/?expanded=create-or-import-keypair-detail#create-or-import-keypair
15:03:03 nicolasbock I was expecting something like app.wsgi as it is done in some other services
15:03:23 sdague mordred: I have no idea why they believe that setting per AZ is a thing
15:03:23 EmilienM dansmith: hey good morning - we promoted TripleO Ci with nova/trunk because jobs passed but it's racy, I see some failure now. I have debug on http://logs.openstack.org/15/424915/13/check/gate-puppet-openstack-integration-4-scenario001-tempest-centos-7/42fa9b9/logs/nova/nova-placement-api.txt.gz
15:03:35 cdent nicolasbock: what's being done with placement is an artifact of pbr
15:03:59 dansmith EmilienM: yeah, I've been watching. I have no ideas at the moment
15:04:13 EmilienM dansmith: ok, I keep digging
15:04:17 nicolasbock cdent, I see. I
15:04:36 nicolasbock is there a timetable for when placement will be split off into a separate entity?
15:04:45 cdent dansmith, EmilienM: if sdague is right that a valid token will not try again to get a service catalog and that service catalog doesn't yet have a placement entry, then that would explain things?
15:05:03 cdent nicolasbock: no, not yet. It's on the agenda for ptg, but nothing solid or detailed yet
15:05:10 mordred sdague: what confuses me more is that they seem to be getting availability_zone back in their response dict
15:05:11 sdague mordred could probably answer that keystoneauth1 assumption best
15:05:13 nicolasbock cdent, ok
15:05:24 sdague mordred: from what provider?
15:05:39 sdague mordred: that's definitely no where near upstream code
15:06:00 mordred sdague: unknown. I'll be asking- and telling them I will not support az for that resource
15:06:08 mordred sdague: what keystoneauth assumption?
15:06:09 EmilienM cdent, sdague: http://logs.openstack.org/15/424915/13/check/gate-puppet-openstack-integration-4-scenario001-tempest-centos-7/42fa9b9/logs/keystone-resources.txt.gz
15:06:39 cdent EmilienM: but at what time is that list made?
15:06:43 mordred yes. keystoneauth1 will not get a new token if the current token is valid
15:06:48 EmilienM cdent: after CI run
15:06:54 EmilienM cdent: it's in post-ci thing, when we gather logs
15:06:57 cdent if it is late in the process then it doesn't tell us anything
15:06:59 EmilienM cdent: so wayyyyy after tempest ran
15:07:11 cdent if the catalog doesn't have placement in it when n-cpu starts up, then boom?
15:07:24 EmilienM that's a good question ^
15:07:38 EmilienM because nova-cpu starts BEFORE httpd, which means before endpoints/services are created
15:08:00 mordred that seems like ordering you'll need to change
15:08:07 EmilienM so if nova-cpu is not able to retry by itself, indeed this is a problem
15:08:11 mordred as more and more things decide to use the catalog to get references to each other
15:08:12 EmilienM mordred: I disagree
15:08:23 EmilienM mordred: nova compute should be able to retry by itself
15:09:04 sdague EmilienM: you really need to have layed out your service catalog before any services start
15:09:05 EmilienM and placement registration is already retrying in nova-compute (dan wrote it iirc)
15:09:07 mordred EmilienM: but if the catalog is required to find out where other apis are, then one should register services with keystone before starting dependent services no?
15:09:14 sdague mordred: ++
15:09:38 dansmith agree, catalog should be available first
15:09:54 EmilienM ok I disagree with your approach but I'll follow your suggestion and tell nova compute to start after
15:10:08 sdague this is a pretty basic flow assumption
15:10:08 EmilienM I'll let you know how it works very shortly
15:10:26 sdague bring up keystone, create all service users, create all endpoint definitions, bring up rest of services
15:10:48 dansmith just like start apache after you have network up so it can bind to addresses, etc
15:11:07 EmilienM sdague: ok, let me change that now.
15:11:11 mordred so - that brings to mind a question though...
15:11:12 mriedem johnthetubaguy: thanks for running the meeting, looks like it was a short one
15:11:14 openstackgerrit Jeremy Liu proposed openstack/nova master: [3/3]Replace six.iteritems() with .items() https://review.openstack.org/415080

Earlier   Later