| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-02-09 | |||
| 14:39:40 | openstackgerrit | Stephen Finucane proposed openstack/nova master: conf: Deprecate various 'iptables_' options https://review.openstack.org/430915 | |
| 14:39:40 | openstackgerrit | Stephen Finucane proposed openstack/nova master: conf: Deprecate 'fake_network' https://review.openstack.org/431020 | |
| 14:39:41 | openstackgerrit | Stephen Finucane proposed openstack/nova master: conf: Deprecate 'firewall_driver' https://review.openstack.org/412390 | |
| 14:39:41 | openstackgerrit | Stephen Finucane proposed openstack/nova master: conf: Deprecate 'metadata_host', 'metadata_port' https://review.openstack.org/430914 | |
| 14:39:42 | openstackgerrit | Stephen Finucane proposed openstack/nova master: conf: Deprecate 'send_arp_for_ha', 'send_arp_for_ha_count' https://review.openstack.org/431021 | |
| 14:40:18 | cdent | EmilienM: Looks like a similar situation to what we were seeing before: n-cpu is not finding the placement endpoint, never records inventory for the compute-node. When n-scheduler starts talking to the placement-api that's the first time it is being talked to, but there's no data there | |
| 14:40:43 | EmilienM | cdent: what is weird is that it passed on the other scenario | |
| 14:40:48 | EmilienM | cdent: which means we're racy now | |
| 14:41:04 | EmilienM | cdent: all our scenarios deploy nova/placement the same way | |
| 14:41:15 | sfinucan | johnthetubaguy, sdague: I think the first 10 or so of those are noddy but helpful nonetheless. Would appreciate ye taking a glance and (hopefully) sending them on their way | |
| 14:41:17 | EmilienM | owalsh: ^ fyi | |
| 14:41:42 | EmilienM | cdent: i'm currently investigating puppet catalog and resources orchestration | |
| 14:42:12 | sfinucan | They should clean up and mark virtually all of the nova-network opts for future removal, while the latter ones untangle the repurposed nova-net opts | |
| 14:42:14 | cdent | EmilienM: it would be great if we could get a summary, with times, of when things happen, because trying to inspect multiple logs at once is proving rather difficult | |
| 14:42:37 | johnthetubaguy | cdent: I wondering if we only try fetching the service catalog once? would we log something for that? | |
| 14:43:12 | EmilienM | cdent: sure let me prepare that: etherpad.openstack.org/p/puppet-nova-trunk-debug | |
| 14:43:40 | cdent | johnthetubaguy: I don't know. This is part of why I've been saying that I think our understanding of keystoneauth1 is incomplete. I personally haven't had a chance to fix that for myself yet. | |
| 14:44:43 | johnthetubaguy | cdent: yeah, I have been digging in there recently for the service token stuff, I know I keep getting lost in there still | |
| 14:46:40 | EmilienM | cdent: I tried to grep useful things : http://paste.fedoraproject.org/551748/66515901/ | |
| 14:47:22 | EmilienM | cdent: if it's too noisy I can reduce the file | |
| 14:47:36 | EmilienM | cdent: so that file is the catalog of a job that failed | |
| 14:49:39 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Enable cold migration with target host(1/2) https://review.openstack.org/408955 | |
| 14:50:20 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Enable cold migration with target host(2/2) https://review.openstack.org/408964 | |
| 14:50:34 | EmilienM | cdent: I reduced the catalog: http://paste.fedoraproject.org/551757/86651825/ | |
| 14:50:39 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Add functional tests for cold migration to same host https://review.openstack.org/414926 | |
| 14:50:57 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: api-ref: Add parameters in cold migrate action https://review.openstack.org/410042 | |
| 14:51:26 | cdent | EmilienM: I'm not sure that providing a log of the puppet activity (that is what that is isn't it?0 really tells us what we need to know. We need to understand the interactions of the various services, in a timeline, which means extracting signal from the logs of the various services, unforutnately. What I mean is: | |
| 14:52:50 | EmilienM | cdent: what gives the catalog is timestamps of when service run | |
| 14:52:56 | cdent | at time X something is registered in the catalog, at time Y something starts, at time Z n-cpu tries to talk to placement for the first time, at time A the placement service records its first hit, at time B n-cpu asks for a service catalog | |
| 14:52:59 | EmilienM | cdent: for ex, you can see when we configure cells, etc | |
| 14:53:01 | cdent | when the services _run_ is not enough | |
| 14:53:27 | cdent | it's when the actions happen that we're trying to detangle, isn't it? I'm not entirely sure. | |
| 14:53:36 | EmilienM | right | |
| 14:57:37 | sdague | the service catalog is embedded in the keystone token | |
| 14:58:03 | mordred | keypairs are per-AZ? | |
| 14:58:39 | sdague | johnthetubaguy / cdent perhaps there is a placement service call before it's registered? | |
| 14:58:45 | EmilienM | cdent: damn, now we promoted tripleo CI, I also see jobs flapping with placement service | |
| 14:58:53 | EmilienM | cdent: it's working sometimes but failing very often | |
| 14:59:25 | nicolasbock | Hi, I am trying to setup the placement-api | |
| 14:59:35 | cdent | sdague: would it not "try again"? | |
| 14:59:53 | nicolasbock | according to http://docs.openstack.org/developer/nova/placement.html there should be a wsgi script somewhere in the package | |
| 14:59:57 | sdague | cdent: while the token is valid? | |
| 15:00:15 | sdague | I thought typically for service users that token is only rotated on expiration | |
| 15:00:16 | nicolasbock | but I can't find that script | |
| 15:00:28 | nicolasbock | could someone give me a pointer where I can find it? | |
| 15:00:29 | cdent | sdague: i don't know, see above about not knowing enough | |
| 15:00:59 | sdague | mordred: there is nothing I know about in the keypair data model that makes it per AZ | |
| 15:01:28 | johnthetubaguy | mordred: +1 what sdague said, its per region, AFAIK | |
| 15:01:43 | mordred | cool. that's what I thought | |
| 15:01:58 | mordred | I got this bug report https://github.com/ansible/ansible/issues/21191 and it seemed crazy to me - I'll dig more, thanks | |
| 15:02:04 | dansmith | yep, and stored with the instance now, so not per cell either | |
| 15:02:12 | EmilienM | nicolasbock: /usr/bin/nova-placement-api | |
| 15:02:23 | nicolasbock | EmilienM, ah, cool! | |
| 15:02:25 | cdent | nicolasbock: a file called nova-placement-api will be automatically created as part of the install process. that is the wsgi script. where it ends up depends on your python setup | |
| 15:02:33 | cdent | aw, EmilienM beat me to it | |
| 15:02:39 | nicolasbock | cdent, thanks :) | |
| 15:02:53 | sdague | mordred: http://developer.openstack.org/api-ref/compute/?expanded=create-or-import-keypair-detail#create-or-import-keypair | |
| 15:03:03 | nicolasbock | I was expecting something like app.wsgi as it is done in some other services | |
| 15:03:23 | EmilienM | dansmith: hey good morning - we promoted TripleO Ci with nova/trunk because jobs passed but it's racy, I see some failure now. I have debug on http://logs.openstack.org/15/424915/13/check/gate-puppet-openstack-integration-4-scenario001-tempest-centos-7/42fa9b9/logs/nova/nova-placement-api.txt.gz | |
| 15:03:23 | sdague | mordred: I have no idea why they believe that setting per AZ is a thing | |
| 15:03:35 | cdent | nicolasbock: what's being done with placement is an artifact of pbr | |
| 15:03:59 | dansmith | EmilienM: yeah, I've been watching. I have no ideas at the moment | |
| 15:04:13 | EmilienM | dansmith: ok, I keep digging | |
| 15:04:17 | nicolasbock | cdent, I see. I | |
| 15:04:36 | nicolasbock | is there a timetable for when placement will be split off into a separate entity? | |
| 15:04:45 | cdent | dansmith, EmilienM: if sdague is right that a valid token will not try again to get a service catalog and that service catalog doesn't yet have a placement entry, then that would explain things? | |
| 15:05:03 | cdent | nicolasbock: no, not yet. It's on the agenda for ptg, but nothing solid or detailed yet | |
| 15:05:10 | mordred | sdague: what confuses me more is that they seem to be getting availability_zone back in their response dict | |
| 15:05:11 | sdague | mordred could probably answer that keystoneauth1 assumption best | |
| 15:05:13 | nicolasbock | cdent, ok | |
| 15:05:24 | sdague | mordred: from what provider? | |
| 15:05:39 | sdague | mordred: that's definitely no where near upstream code | |
| 15:06:00 | mordred | sdague: unknown. I'll be asking- and telling them I will not support az for that resource | |
| 15:06:08 | mordred | sdague: what keystoneauth assumption? | |
| 15:06:09 | EmilienM | cdent, sdague: http://logs.openstack.org/15/424915/13/check/gate-puppet-openstack-integration-4-scenario001-tempest-centos-7/42fa9b9/logs/keystone-resources.txt.gz | |
| 15:06:39 | cdent | EmilienM: but at what time is that list made? | |
| 15:06:43 | mordred | yes. keystoneauth1 will not get a new token if the current token is valid | |
| 15:06:48 | EmilienM | cdent: after CI run | |
| 15:06:54 | EmilienM | cdent: it's in post-ci thing, when we gather logs | |
| 15:06:57 | cdent | if it is late in the process then it doesn't tell us anything | |
| 15:06:59 | EmilienM | cdent: so wayyyyy after tempest ran | |
| 15:07:11 | cdent | if the catalog doesn't have placement in it when n-cpu starts up, then boom? | |
| 15:07:24 | EmilienM | that's a good question ^ | |
| 15:07:38 | EmilienM | because nova-cpu starts BEFORE httpd, which means before endpoints/services are created | |
| 15:08:00 | mordred | that seems like ordering you'll need to change | |
| 15:08:07 | EmilienM | so if nova-cpu is not able to retry by itself, indeed this is a problem | |
| 15:08:11 | mordred | as more and more things decide to use the catalog to get references to each other | |
| 15:08:12 | EmilienM | mordred: I disagree | |
| 15:08:23 | EmilienM | mordred: nova compute should be able to retry by itself | |
| 15:09:04 | sdague | EmilienM: you really need to have layed out your service catalog before any services start | |
| 15:09:05 | EmilienM | and placement registration is already retrying in nova-compute (dan wrote it iirc) | |
| 15:09:07 | mordred | EmilienM: but if the catalog is required to find out where other apis are, then one should register services with keystone before starting dependent services no? | |
| 15:09:14 | sdague | mordred: ++ | |
| 15:09:38 | dansmith | agree, catalog should be available first | |
| 15:09:54 | EmilienM | ok I disagree with your approach but I'll follow your suggestion and tell nova compute to start after | |
| 15:10:08 | EmilienM | I'll let you know how it works very shortly | |
| 15:10:08 | sdague | this is a pretty basic flow assumption | |
| 15:10:26 | sdague | bring up keystone, create all service users, create all endpoint definitions, bring up rest of services | |
| 15:10:48 | dansmith | just like start apache after you have network up so it can bind to addresses, etc | |
| 15:11:07 | EmilienM | sdague: ok, let me change that now. | |