| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-02-09 | |||
| 14:52:56 | cdent | at time X something is registered in the catalog, at time Y something starts, at time Z n-cpu tries to talk to placement for the first time, at time A the placement service records its first hit, at time B n-cpu asks for a service catalog | |
| 14:52:59 | EmilienM | cdent: for ex, you can see when we configure cells, etc | |
| 14:53:01 | cdent | when the services _run_ is not enough | |
| 14:53:27 | cdent | it's when the actions happen that we're trying to detangle, isn't it? I'm not entirely sure. | |
| 14:53:36 | EmilienM | right | |
| 14:57:37 | sdague | the service catalog is embedded in the keystone token | |
| 14:58:03 | mordred | keypairs are per-AZ? | |
| 14:58:39 | sdague | johnthetubaguy / cdent perhaps there is a placement service call before it's registered? | |
| 14:58:45 | EmilienM | cdent: damn, now we promoted tripleo CI, I also see jobs flapping with placement service | |
| 14:58:53 | EmilienM | cdent: it's working sometimes but failing very often | |
| 14:59:25 | nicolasbock | Hi, I am trying to setup the placement-api | |
| 14:59:35 | cdent | sdague: would it not "try again"? | |
| 14:59:53 | nicolasbock | according to http://docs.openstack.org/developer/nova/placement.html there should be a wsgi script somewhere in the package | |
| 14:59:57 | sdague | cdent: while the token is valid? | |
| 15:00:15 | sdague | I thought typically for service users that token is only rotated on expiration | |
| 15:00:16 | nicolasbock | but I can't find that script | |
| 15:00:28 | nicolasbock | could someone give me a pointer where I can find it? | |
| 15:00:29 | cdent | sdague: i don't know, see above about not knowing enough | |
| 15:00:59 | sdague | mordred: there is nothing I know about in the keypair data model that makes it per AZ | |
| 15:01:28 | johnthetubaguy | mordred: +1 what sdague said, its per region, AFAIK | |
| 15:01:43 | mordred | cool. that's what I thought | |
| 15:01:58 | mordred | I got this bug report https://github.com/ansible/ansible/issues/21191 and it seemed crazy to me - I'll dig more, thanks | |
| 15:02:04 | dansmith | yep, and stored with the instance now, so not per cell either | |
| 15:02:12 | EmilienM | nicolasbock: /usr/bin/nova-placement-api | |
| 15:02:23 | nicolasbock | EmilienM, ah, cool! | |
| 15:02:25 | cdent | nicolasbock: a file called nova-placement-api will be automatically created as part of the install process. that is the wsgi script. where it ends up depends on your python setup | |
| 15:02:33 | cdent | aw, EmilienM beat me to it | |
| 15:02:39 | nicolasbock | cdent, thanks :) | |
| 15:02:53 | sdague | mordred: http://developer.openstack.org/api-ref/compute/?expanded=create-or-import-keypair-detail#create-or-import-keypair | |
| 15:03:03 | nicolasbock | I was expecting something like app.wsgi as it is done in some other services | |
| 15:03:23 | EmilienM | dansmith: hey good morning - we promoted TripleO Ci with nova/trunk because jobs passed but it's racy, I see some failure now. I have debug on http://logs.openstack.org/15/424915/13/check/gate-puppet-openstack-integration-4-scenario001-tempest-centos-7/42fa9b9/logs/nova/nova-placement-api.txt.gz | |
| 15:03:23 | sdague | mordred: I have no idea why they believe that setting per AZ is a thing | |
| 15:03:35 | cdent | nicolasbock: what's being done with placement is an artifact of pbr | |
| 15:03:59 | dansmith | EmilienM: yeah, I've been watching. I have no ideas at the moment | |
| 15:04:13 | EmilienM | dansmith: ok, I keep digging | |
| 15:04:17 | nicolasbock | cdent, I see. I | |
| 15:04:36 | nicolasbock | is there a timetable for when placement will be split off into a separate entity? | |
| 15:04:45 | cdent | dansmith, EmilienM: if sdague is right that a valid token will not try again to get a service catalog and that service catalog doesn't yet have a placement entry, then that would explain things? | |
| 15:05:03 | cdent | nicolasbock: no, not yet. It's on the agenda for ptg, but nothing solid or detailed yet | |
| 15:05:10 | mordred | sdague: what confuses me more is that they seem to be getting availability_zone back in their response dict | |
| 15:05:11 | sdague | mordred could probably answer that keystoneauth1 assumption best | |
| 15:05:13 | nicolasbock | cdent, ok | |
| 15:05:24 | sdague | mordred: from what provider? | |
| 15:05:39 | sdague | mordred: that's definitely no where near upstream code | |
| 15:06:00 | mordred | sdague: unknown. I'll be asking- and telling them I will not support az for that resource | |
| 15:06:08 | mordred | sdague: what keystoneauth assumption? | |
| 15:06:09 | EmilienM | cdent, sdague: http://logs.openstack.org/15/424915/13/check/gate-puppet-openstack-integration-4-scenario001-tempest-centos-7/42fa9b9/logs/keystone-resources.txt.gz | |
| 15:06:39 | cdent | EmilienM: but at what time is that list made? | |
| 15:06:43 | mordred | yes. keystoneauth1 will not get a new token if the current token is valid | |
| 15:06:48 | EmilienM | cdent: after CI run | |
| 15:06:54 | EmilienM | cdent: it's in post-ci thing, when we gather logs | |
| 15:06:57 | cdent | if it is late in the process then it doesn't tell us anything | |
| 15:06:59 | EmilienM | cdent: so wayyyyy after tempest ran | |
| 15:07:11 | cdent | if the catalog doesn't have placement in it when n-cpu starts up, then boom? | |
| 15:07:24 | EmilienM | that's a good question ^ | |
| 15:07:38 | EmilienM | because nova-cpu starts BEFORE httpd, which means before endpoints/services are created | |
| 15:08:00 | mordred | that seems like ordering you'll need to change | |
| 15:08:07 | EmilienM | so if nova-cpu is not able to retry by itself, indeed this is a problem | |
| 15:08:11 | mordred | as more and more things decide to use the catalog to get references to each other | |
| 15:08:12 | EmilienM | mordred: I disagree | |
| 15:08:23 | EmilienM | mordred: nova compute should be able to retry by itself | |
| 15:09:04 | sdague | EmilienM: you really need to have layed out your service catalog before any services start | |
| 15:09:05 | EmilienM | and placement registration is already retrying in nova-compute (dan wrote it iirc) | |
| 15:09:07 | mordred | EmilienM: but if the catalog is required to find out where other apis are, then one should register services with keystone before starting dependent services no? | |
| 15:09:14 | sdague | mordred: ++ | |
| 15:09:38 | dansmith | agree, catalog should be available first | |
| 15:09:54 | EmilienM | ok I disagree with your approach but I'll follow your suggestion and tell nova compute to start after | |
| 15:10:08 | EmilienM | I'll let you know how it works very shortly | |
| 15:10:08 | sdague | this is a pretty basic flow assumption | |
| 15:10:26 | sdague | bring up keystone, create all service users, create all endpoint definitions, bring up rest of services | |
| 15:10:48 | dansmith | just like start apache after you have network up so it can bind to addresses, etc | |
| 15:11:07 | EmilienM | sdague: ok, let me change that now. | |
| 15:11:11 | mordred | so - that brings to mind a question though... | |
| 15:11:12 | mriedem | johnthetubaguy: thanks for running the meeting, looks like it was a short one | |
| 15:11:14 | openstackgerrit | Jeremy Liu proposed openstack/nova master: [3/3]Replace six.iteritems() with .items() https://review.openstack.org/415080 | |
| 15:11:32 | mordred | what happens when all the service apis run in apache? | |
| 15:11:55 | EmilienM | mordred: nova-api in WSGI / apache is not supported | |
| 15:12:02 | sdague | mordred: what's the question? | |
| 15:12:07 | EmilienM | mordred: they suggested me to revert what we did in TripleO and use eventlet | |
| 15:12:09 | mordred | I know - but what about when we get to the point where it is? | |
| 15:12:21 | sdague | mordred: I don't think that changes the flow | |
| 15:12:26 | mordred | so that there isn't really a possibility of starting api endpoints in a particular order | |
| 15:12:29 | mordred | sdague: no? | |
| 15:12:30 | EmilienM | in the case of nova compute, no | |
| 15:12:35 | mordred | oh. duh. nevermind | |
| 15:12:39 | sdague | mordred: you can totally enable just keystone | |
| 15:12:44 | cdent | keystone becomes a...keystone | |
| 15:12:46 | sdague | do all it's stuff | |
| 15:12:51 | sdague | then bring up everything else | |
| 15:13:03 | mordred | yah - nevermind - only on second coffee | |
| 15:13:04 | mordred | :) | |
| 15:13:05 | sdague | heh | |
| 15:13:55 | sdague | mordred: yeh, even in those cases, you'll still need a keystone bootstrapping phase. I can't imagine that ever goes away | |
| 15:15:49 | cdent | TIL | |
| 15:17:30 | openstackgerrit | John Garbutt proposed openstack/nova master: Assume neutron port_binding extensions enabled https://review.openstack.org/357726 | |
| 15:17:35 | johnthetubaguy | mriedem: no worries | |
| 15:18:16 | jroll | johnthetubaguy: we have a "Virt and Metal in the same region" topic in our ptg list, feels like that should be a nova topic, wdyt? | |
| 15:18:50 | johnthetubaguy | jroll: feels like a joint topic | |
| 15:19:05 | jroll | johnthetubaguy: maybe, I know it's mostly nova issues though | |
| 15:19:12 | johnthetubaguy | jroll: oh, well more nova than ironic though | |