| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-02-09 | |||
| 14:50:20 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Enable cold migration with target host(2/2) https://review.openstack.org/408964 | |
| 14:50:34 | EmilienM | cdent: I reduced the catalog: http://paste.fedoraproject.org/551757/86651825/ | |
| 14:50:39 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Add functional tests for cold migration to same host https://review.openstack.org/414926 | |
| 14:50:57 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: api-ref: Add parameters in cold migrate action https://review.openstack.org/410042 | |
| 14:51:26 | cdent | EmilienM: I'm not sure that providing a log of the puppet activity (that is what that is isn't it?0 really tells us what we need to know. We need to understand the interactions of the various services, in a timeline, which means extracting signal from the logs of the various services, unforutnately. What I mean is: | |
| 14:52:50 | EmilienM | cdent: what gives the catalog is timestamps of when service run | |
| 14:52:56 | cdent | at time X something is registered in the catalog, at time Y something starts, at time Z n-cpu tries to talk to placement for the first time, at time A the placement service records its first hit, at time B n-cpu asks for a service catalog | |
| 14:52:59 | EmilienM | cdent: for ex, you can see when we configure cells, etc | |
| 14:53:01 | cdent | when the services _run_ is not enough | |
| 14:53:27 | cdent | it's when the actions happen that we're trying to detangle, isn't it? I'm not entirely sure. | |
| 14:53:36 | EmilienM | right | |
| 14:57:37 | sdague | the service catalog is embedded in the keystone token | |
| 14:58:03 | mordred | keypairs are per-AZ? | |
| 14:58:39 | sdague | johnthetubaguy / cdent perhaps there is a placement service call before it's registered? | |
| 14:58:45 | EmilienM | cdent: damn, now we promoted tripleo CI, I also see jobs flapping with placement service | |
| 14:58:53 | EmilienM | cdent: it's working sometimes but failing very often | |
| 14:59:25 | nicolasbock | Hi, I am trying to setup the placement-api | |
| 14:59:35 | cdent | sdague: would it not "try again"? | |
| 14:59:53 | nicolasbock | according to http://docs.openstack.org/developer/nova/placement.html there should be a wsgi script somewhere in the package | |
| 14:59:57 | sdague | cdent: while the token is valid? | |
| 15:00:15 | sdague | I thought typically for service users that token is only rotated on expiration | |
| 15:00:16 | nicolasbock | but I can't find that script | |
| 15:00:28 | nicolasbock | could someone give me a pointer where I can find it? | |
| 15:00:29 | cdent | sdague: i don't know, see above about not knowing enough | |
| 15:00:59 | sdague | mordred: there is nothing I know about in the keypair data model that makes it per AZ | |
| 15:01:28 | johnthetubaguy | mordred: +1 what sdague said, its per region, AFAIK | |
| 15:01:43 | mordred | cool. that's what I thought | |
| 15:01:58 | mordred | I got this bug report https://github.com/ansible/ansible/issues/21191 and it seemed crazy to me - I'll dig more, thanks | |
| 15:02:04 | dansmith | yep, and stored with the instance now, so not per cell either | |
| 15:02:12 | EmilienM | nicolasbock: /usr/bin/nova-placement-api | |
| 15:02:23 | nicolasbock | EmilienM, ah, cool! | |
| 15:02:25 | cdent | nicolasbock: a file called nova-placement-api will be automatically created as part of the install process. that is the wsgi script. where it ends up depends on your python setup | |
| 15:02:33 | cdent | aw, EmilienM beat me to it | |
| 15:02:39 | nicolasbock | cdent, thanks :) | |
| 15:02:53 | sdague | mordred: http://developer.openstack.org/api-ref/compute/?expanded=create-or-import-keypair-detail#create-or-import-keypair | |
| 15:03:03 | nicolasbock | I was expecting something like app.wsgi as it is done in some other services | |
| 15:03:23 | sdague | mordred: I have no idea why they believe that setting per AZ is a thing | |
| 15:03:23 | EmilienM | dansmith: hey good morning - we promoted TripleO Ci with nova/trunk because jobs passed but it's racy, I see some failure now. I have debug on http://logs.openstack.org/15/424915/13/check/gate-puppet-openstack-integration-4-scenario001-tempest-centos-7/42fa9b9/logs/nova/nova-placement-api.txt.gz | |
| 15:03:35 | cdent | nicolasbock: what's being done with placement is an artifact of pbr | |
| 15:03:59 | dansmith | EmilienM: yeah, I've been watching. I have no ideas at the moment | |
| 15:04:13 | EmilienM | dansmith: ok, I keep digging | |
| 15:04:17 | nicolasbock | cdent, I see. I | |
| 15:04:36 | nicolasbock | is there a timetable for when placement will be split off into a separate entity? | |
| 15:04:45 | cdent | dansmith, EmilienM: if sdague is right that a valid token will not try again to get a service catalog and that service catalog doesn't yet have a placement entry, then that would explain things? | |
| 15:05:03 | cdent | nicolasbock: no, not yet. It's on the agenda for ptg, but nothing solid or detailed yet | |
| 15:05:10 | mordred | sdague: what confuses me more is that they seem to be getting availability_zone back in their response dict | |
| 15:05:11 | sdague | mordred could probably answer that keystoneauth1 assumption best | |
| 15:05:13 | nicolasbock | cdent, ok | |
| 15:05:24 | sdague | mordred: from what provider? | |
| 15:05:39 | sdague | mordred: that's definitely no where near upstream code | |
| 15:06:00 | mordred | sdague: unknown. I'll be asking- and telling them I will not support az for that resource | |
| 15:06:08 | mordred | sdague: what keystoneauth assumption? | |
| 15:06:09 | EmilienM | cdent, sdague: http://logs.openstack.org/15/424915/13/check/gate-puppet-openstack-integration-4-scenario001-tempest-centos-7/42fa9b9/logs/keystone-resources.txt.gz | |
| 15:06:39 | cdent | EmilienM: but at what time is that list made? | |
| 15:06:43 | mordred | yes. keystoneauth1 will not get a new token if the current token is valid | |
| 15:06:48 | EmilienM | cdent: after CI run | |
| 15:06:54 | EmilienM | cdent: it's in post-ci thing, when we gather logs | |
| 15:06:57 | cdent | if it is late in the process then it doesn't tell us anything | |
| 15:06:59 | EmilienM | cdent: so wayyyyy after tempest ran | |
| 15:07:11 | cdent | if the catalog doesn't have placement in it when n-cpu starts up, then boom? | |
| 15:07:24 | EmilienM | that's a good question ^ | |
| 15:07:38 | EmilienM | because nova-cpu starts BEFORE httpd, which means before endpoints/services are created | |
| 15:08:00 | mordred | that seems like ordering you'll need to change | |
| 15:08:07 | EmilienM | so if nova-cpu is not able to retry by itself, indeed this is a problem | |
| 15:08:11 | mordred | as more and more things decide to use the catalog to get references to each other | |
| 15:08:12 | EmilienM | mordred: I disagree | |
| 15:08:23 | EmilienM | mordred: nova compute should be able to retry by itself | |
| 15:09:04 | sdague | EmilienM: you really need to have layed out your service catalog before any services start | |
| 15:09:05 | EmilienM | and placement registration is already retrying in nova-compute (dan wrote it iirc) | |
| 15:09:07 | mordred | EmilienM: but if the catalog is required to find out where other apis are, then one should register services with keystone before starting dependent services no? | |
| 15:09:14 | sdague | mordred: ++ | |
| 15:09:38 | dansmith | agree, catalog should be available first | |
| 15:09:54 | EmilienM | ok I disagree with your approach but I'll follow your suggestion and tell nova compute to start after | |
| 15:10:08 | sdague | this is a pretty basic flow assumption | |
| 15:10:08 | EmilienM | I'll let you know how it works very shortly | |
| 15:10:26 | sdague | bring up keystone, create all service users, create all endpoint definitions, bring up rest of services | |
| 15:10:48 | dansmith | just like start apache after you have network up so it can bind to addresses, etc | |
| 15:11:07 | EmilienM | sdague: ok, let me change that now. | |
| 15:11:11 | mordred | so - that brings to mind a question though... | |
| 15:11:12 | mriedem | johnthetubaguy: thanks for running the meeting, looks like it was a short one | |
| 15:11:14 | openstackgerrit | Jeremy Liu proposed openstack/nova master: [3/3]Replace six.iteritems() with .items() https://review.openstack.org/415080 | |
| 15:11:32 | mordred | what happens when all the service apis run in apache? | |
| 15:11:55 | EmilienM | mordred: nova-api in WSGI / apache is not supported | |
| 15:12:02 | sdague | mordred: what's the question? | |
| 15:12:07 | EmilienM | mordred: they suggested me to revert what we did in TripleO and use eventlet | |
| 15:12:09 | mordred | I know - but what about when we get to the point where it is? | |
| 15:12:21 | sdague | mordred: I don't think that changes the flow | |
| 15:12:26 | mordred | so that there isn't really a possibility of starting api endpoints in a particular order | |
| 15:12:29 | mordred | sdague: no? | |
| 15:12:30 | EmilienM | in the case of nova compute, no | |
| 15:12:35 | mordred | oh. duh. nevermind | |
| 15:12:39 | sdague | mordred: you can totally enable just keystone | |
| 15:12:44 | cdent | keystone becomes a...keystone | |
| 15:12:46 | sdague | do all it's stuff | |
| 15:12:51 | sdague | then bring up everything else | |
| 15:13:03 | mordred | yah - nevermind - only on second coffee | |
| 15:13:04 | mordred | :) | |
| 15:13:05 | sdague | heh | |
| 15:13:55 | sdague | mordred: yeh, even in those cases, you'll still need a keystone bootstrapping phase. I can't imagine that ever goes away | |
| 15:15:49 | cdent | TIL | |