| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-02-08 | |||
| 14:54:58 | EmilienM | it can be in credentials, I'm checking now. | |
| 14:55:10 | dansmith | actually, placement access is not zero bytes, it just looks empty | |
| 14:55:16 | EmilienM | johnthetubaguy: you said you checked in keystone logs, did you see something? | |
| 14:55:32 | johnthetubaguy | EmilienM: I haven't looked there, was looking at placement | |
| 14:55:52 | cdent | EmilienM: I'm pretty sure when apache starts it opens the log file, but only when the wsgi application is requested to, will it write and when it does the error log should get a bunch of oslo config noise | |
| 14:56:23 | EmilienM | cdent: right, and it's used, see http://logs.openstack.org/71/430571/1/check/gate-tripleo-ci-centos-7-nonha-multinode/bae4746/logs/subnode-2/var/log/nova/nova-placement-api.txt.gz#_2017-02-08_06_40_29_306 | |
| 14:56:58 | cdent | too many different log files! ;) | |
| 14:57:07 | mriedem | i second that | |
| 14:57:09 | dansmith | oh jeez | |
| 14:57:12 | dansmith | why are those there? | |
| 14:57:13 | mriedem | i get totally lost in the tripleo logs | |
| 14:57:27 | dansmith | mriedem: and no filtering :/ | |
| 14:57:36 | mriedem | yes no filtering is my other top complaint | |
| 14:57:59 | EmilienM | that's useful feedback, I'll take some actions | |
| 14:58:02 | dansmith | and it's nearly a full system dump, so lots of noise, and some of the files don't have a mimetype that lets them be displayed in a browser | |
| 14:58:06 | johnthetubaguy | dansmith: thats what I meant by it looks like the scheduler can fetch, even though compute can't :S | |
| 14:58:15 | cdent | so, that log starts at 06:40:28, but the nova-compute is delcaring the endpoing not found at 06:40:24 | |
| 14:58:28 | cdent | s/that log/the placement api log/ | |
| 14:58:45 | dansmith | johnthetubaguy: gotcha, so confusing | |
| 14:58:52 | EmilienM | let me check when endpoint is created | |
| 14:58:55 | johnthetubaguy | 2017-02-08 06:40:28.684 is when placement starts | |
| 14:58:56 | EmilienM | would it recheck later? | |
| 14:59:10 | mriedem | the compute will keep trying | |
| 14:59:18 | johnthetubaguy | 2017-02-08 06:40:29.306 is first request | |
| 14:59:38 | mriedem | the scheduler will keep trying to, but not per request | |
| 15:00:05 | EmilienM | endpoint is created at 06:38:25 | |
| 15:00:26 | EmilienM | oh wait no, it's for regionOne/nova::compute | |
| 15:00:27 | EmilienM | not placement | |
| 15:00:29 | dansmith | surely the credentials are created by that time right? | |
| 15:00:46 | dansmith | I figure they must be created before they're written to the configfile yes? | |
| 15:00:59 | EmilienM | 06:38:25 centos-7-2-node-osic-cloud1-s3500-7172104-416160 os-collect-config: #033[mNotice: /Stage[main]/Nova::Keystone::Auth_placement/Keystone::Resource::Service_identity[placement]/Keystone_endpoint[regionOne/placement::placement]/ensure: created | |
| 15:01:53 | EmilienM | and it's highly possible nova-compute starts afterward | |
| 15:02:35 | EmilienM | err, before I mean. I checked and nova-compute started at 06:32:06 | |
| 15:02:49 | EmilienM | 06:32:06 centos-7-2-node-osic-cloud1-s3500-7172104-416160 os-collect-config: #033[mNotice: /Stage[main]/Nova::Compute/Nova::Generic_service[compute]/Service[nova-compute]/ensure: ensure changed 'stopped' to 'running' | |
| 15:03:01 | cdent | EmilienM: right, which explains the change from endpoint not found to "Placement service credentials do not work." | |
| 15:03:20 | EmilienM | right, but IIUC it should retry | |
| 15:03:21 | cdent | in the nova-compute logs | |
| 15:03:25 | cdent | it does | |
| 15:03:31 | cdent | that's why we get the change | |
| 15:03:43 | cdent | can't find endpoint, wait, retry, found endpoint credentials don't work | |
| 15:03:46 | EmilienM | and if you look at logs, it retries only 2 times after the first fialure | |
| 15:04:03 | cdent | (or at least ks.Unauthorized, which means many things) | |
| 15:04:24 | dansmith | cdent: did you ever elaborate on what the multiple things are? | |
| 15:04:36 | dansmith | cdent: can't get a token and a real 403 I assume? | |
| 15:04:45 | EmilienM | what worries me is "Unable to refresh my resource provider record" | |
| 15:05:06 | dansmith | EmilienM: that's part of the same thing | |
| 15:05:08 | mriedem | EmilienM: "and if you look at logs, it retries only 2 times after the first fialure" - are you sure it didn't just stop giving errors after 2 retries because it started working? | |
| 15:05:12 | dansmith | EmilienM: can't GET it before it can't POST it | |
| 15:05:18 | cdent | dansmith: no, I'm trying to find it again, but from a few days ago it was something much weirder than that | |
| 15:05:23 | mriedem | stepping back, wasn't this working like a week ago? | |
| 15:05:27 | dansmith | cdent: okay, knowing might be relevant | |
| 15:05:29 | mriedem | if so, what changed on the tripleo side? | |
| 15:05:32 | cdent | dansmith: yes :) | |
| 15:05:54 | EmilienM | mriedem: it works fine today when I deploy OpenStack from trunk except nova which is pin to ocata-3 | |
| 15:05:56 | dansmith | cdent: just call me captain obvious | |
| 15:06:17 | EmilienM | mriedem: I'm having a CI test patch that deploy nova from trunk and it fails | |
| 15:06:30 | EmilienM | mriedem: but like johnthetubaguy and dansmith said, it was probably silently failing? | |
| 15:06:58 | EmilienM | mriedem: if you want I can show you how it works with ocata-3 and you can tell if nova placement/ novacompute are actually working together | |
| 15:06:59 | owalsh | EmilienM: don't we have POST requests in the ocata-3 placement-api log? | |
| 15:07:21 | EmilienM | owalsh: let me check a sec | |
| 15:07:28 | mriedem | EmilienM: so...last week tripleo ci was testing from nova trunk and things worked, but now it's nova ocata-3 (which is older than rc1) and things aren't working? | |
| 15:07:58 | owalsh | mriedem: ocata-3 works, trunk does not | |
| 15:08:19 | mriedem | owalsh: ok | |
| 15:08:22 | EmilienM | cdent, owalsh: that is an ocata-3 CI job: http://logs.openstack.org/50/330050/17/check/gate-tripleo-ci-centos-7-nonha-multinode/95f4ed9/logs/subnode-2/var/log/nova/nova-placement-api.txt.gz#_2017-02-08_13_22_00_143 | |
| 15:08:42 | EmilienM | mriedem: no, we don't test nova-trunk in tripleo, we manually pin to ocata-3 because trunk doesn't work for us | |
| 15:08:47 | EmilienM | mriedem: we do that since days | |
| 15:08:52 | EmilienM | since ocata-3 actually | |
| 15:09:07 | EmilienM | cdent, owalsh : is the POST means we're good? | |
| 15:09:40 | cdent | EmilienM: yes, that POST means "a resource provider was created" | |
| 15:10:01 | cdent | and in the context of this stuff, the only resource provider that ought to be create is a compute node | |
| 15:10:08 | EmilienM | cdent: cool so it works fine with ocata-3 | |
| 15:10:19 | EmilienM | and btw, credentials and config are same | |
| 15:10:29 | EmilienM | the only thing that change is an rpm that contains nova code | |
| 15:10:31 | cdent | EmilienM: is this using eventlet on nova-api? | |
| 15:10:38 | EmilienM | cdent: let me check | |
| 15:10:53 | EmilienM | cdent: http://logs.openstack.org/50/330050/17/check/gate-tripleo-ci-centos-7-nonha-multinode/95f4ed9/logs/subnode-2/var/log/httpd/ | |
| 15:10:55 | EmilienM | eventlet | |
| 15:11:06 | EmilienM | otherwise we would have httpd logs in ^ | |
| 15:12:56 | siva_krishnan | mdbooth: can you please review this spec whenever you find time https://review.openstack.org/#/c/429913/ | |
| 15:13:01 | EmilienM | cdent: and now, if you look at our logs, we don't have the POST anymore | |
| 15:13:16 | EmilienM | could it be related to cells v2? | |
| 15:14:13 | cdent | Is ocata-3 the stable/ocata branch or is there some other hash I can use directly? | |
| 15:14:30 | dansmith | where are the package build logs so we can see the nova hash being tested? | |
| 15:15:09 | mriedem | EmilienM: if the tripleo jobs pin nova at certain points, how do the tripleo jobs in nova's experimental queue actually test the nova change? | |
| 15:15:21 | EmilienM | cdent: rc1 is stable/ocata for nova. In tripleo we don't have stable/ocata yet | |
| 15:15:39 | EmilienM | mriedem: they build it, the pin doesn't work | |
| 15:15:45 | EmilienM | mriedem: so we build nova from master in that case | |
| 15:16:15 | dansmith | EmilienM: so a change against tripleo tests against the pin and a change against nova tests against master+change ? | |
| 15:16:31 | EmilienM | dansmith: exactly | |
| 15:16:46 | EmilienM | we use "delorean" to build rpms from git commits | |
| 15:16:55 | EmilienM | please admit the name is cool | |
| 15:17:11 | mriedem | didn't that guy rip everyone off and go bankrupt? | |
| 15:17:19 | dansmith | yes | |
| 15:17:33 | EmilienM | lol | |
| 15:18:13 | owalsh | ...and get arrested with a suitcase full of coke IIRC | |
| 15:18:22 | mdbooth | siva_krishnan: Something lyarwood was trying to do with rescue was simply add an alternate boot image and set it as the boot volume. For unrescue we do the reverse operation. IIRC this also makes live-migration pretty simple to fix. | |
| 15:18:55 | mdbooth | And it makes rescue better, because the rescue environment has all ephemeral disks and cinder volumes. | |
| 15:19:00 | dansmith | cdent: looking at things that have changed since o3, I see your generation patch | |
| 15:19:02 | mdbooth | As well as the original root disk. | |
| 15:19:06 | dansmith | and the scheduler calling placement | |