Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-02-08
15:03:21 cdent in the nova-compute logs
15:03:25 cdent it does
15:03:31 cdent that's why we get the change
15:03:43 cdent can't find endpoint, wait, retry, found endpoint credentials don't work
15:03:46 EmilienM and if you look at logs, it retries only 2 times after the first fialure
15:04:03 cdent (or at least ks.Unauthorized, which means many things)
15:04:24 dansmith cdent: did you ever elaborate on what the multiple things are?
15:04:36 dansmith cdent: can't get a token and a real 403 I assume?
15:04:45 EmilienM what worries me is "Unable to refresh my resource provider record"
15:05:06 dansmith EmilienM: that's part of the same thing
15:05:08 mriedem EmilienM: "and if you look at logs, it retries only 2 times after the first fialure" - are you sure it didn't just stop giving errors after 2 retries because it started working?
15:05:12 dansmith EmilienM: can't GET it before it can't POST it
15:05:18 cdent dansmith: no, I'm trying to find it again, but from a few days ago it was something much weirder than that
15:05:23 mriedem stepping back, wasn't this working like a week ago?
15:05:27 dansmith cdent: okay, knowing might be relevant
15:05:29 mriedem if so, what changed on the tripleo side?
15:05:32 cdent dansmith: yes :)
15:05:54 EmilienM mriedem: it works fine today when I deploy OpenStack from trunk except nova which is pin to ocata-3
15:05:56 dansmith cdent: just call me captain obvious
15:06:17 EmilienM mriedem: I'm having a CI test patch that deploy nova from trunk and it fails
15:06:30 EmilienM mriedem: but like johnthetubaguy and dansmith said, it was probably silently failing?
15:06:58 EmilienM mriedem: if you want I can show you how it works with ocata-3 and you can tell if nova placement/ novacompute are actually working together
15:06:59 owalsh EmilienM: don't we have POST requests in the ocata-3 placement-api log?
15:07:21 EmilienM owalsh: let me check a sec
15:07:28 mriedem EmilienM: so...last week tripleo ci was testing from nova trunk and things worked, but now it's nova ocata-3 (which is older than rc1) and things aren't working?
15:07:58 owalsh mriedem: ocata-3 works, trunk does not
15:08:19 mriedem owalsh: ok
15:08:22 EmilienM cdent, owalsh: that is an ocata-3 CI job: http://logs.openstack.org/50/330050/17/check/gate-tripleo-ci-centos-7-nonha-multinode/95f4ed9/logs/subnode-2/var/log/nova/nova-placement-api.txt.gz#_2017-02-08_13_22_00_143
15:08:42 EmilienM mriedem: no, we don't test nova-trunk in tripleo, we manually pin to ocata-3 because trunk doesn't work for us
15:08:47 EmilienM mriedem: we do that since days
15:08:52 EmilienM since ocata-3 actually
15:09:07 EmilienM cdent, owalsh : is the POST means we're good?
15:09:40 cdent EmilienM: yes, that POST means "a resource provider was created"
15:10:01 cdent and in the context of this stuff, the only resource provider that ought to be create is a compute node
15:10:08 EmilienM cdent: cool so it works fine with ocata-3
15:10:19 EmilienM and btw, credentials and config are same
15:10:29 EmilienM the only thing that change is an rpm that contains nova code
15:10:31 cdent EmilienM: is this using eventlet on nova-api?
15:10:38 EmilienM cdent: let me check
15:10:53 EmilienM cdent: http://logs.openstack.org/50/330050/17/check/gate-tripleo-ci-centos-7-nonha-multinode/95f4ed9/logs/subnode-2/var/log/httpd/
15:10:55 EmilienM eventlet
15:11:06 EmilienM otherwise we would have httpd logs in ^
15:12:56 siva_krishnan mdbooth: can you please review this spec whenever you find time https://review.openstack.org/#/c/429913/
15:13:01 EmilienM cdent: and now, if you look at our logs, we don't have the POST anymore
15:13:16 EmilienM could it be related to cells v2?
15:14:13 cdent Is ocata-3 the stable/ocata branch or is there some other hash I can use directly?
15:14:30 dansmith where are the package build logs so we can see the nova hash being tested?
15:15:09 mriedem EmilienM: if the tripleo jobs pin nova at certain points, how do the tripleo jobs in nova's experimental queue actually test the nova change?
15:15:21 EmilienM cdent: rc1 is stable/ocata for nova. In tripleo we don't have stable/ocata yet
15:15:39 EmilienM mriedem: they build it, the pin doesn't work
15:15:45 EmilienM mriedem: so we build nova from master in that case
15:16:15 dansmith EmilienM: so a change against tripleo tests against the pin and a change against nova tests against master+change ?
15:16:31 EmilienM dansmith: exactly
15:16:46 EmilienM we use "delorean" to build rpms from git commits
15:16:55 EmilienM please admit the name is cool
15:17:11 mriedem didn't that guy rip everyone off and go bankrupt?
15:17:19 dansmith yes
15:17:33 EmilienM lol
15:18:13 owalsh ...and get arrested with a suitcase full of coke IIRC
15:18:22 mdbooth siva_krishnan: Something lyarwood was trying to do with rescue was simply add an alternate boot image and set it as the boot volume. For unrescue we do the reverse operation. IIRC this also makes live-migration pretty simple to fix.
15:18:55 mdbooth And it makes rescue better, because the rescue environment has all ephemeral disks and cinder volumes.
15:19:00 dansmith cdent: looking at things that have changed since o3, I see your generation patch
15:19:02 mdbooth As well as the original root disk.
15:19:06 dansmith and the scheduler calling placement
15:19:16 mdbooth siva_krishnan: Could you resurrect that instead?
15:19:21 dansmith cdent: but not much else
15:19:48 siva_krishnan mdbooth: will look into that :)
15:20:05 mdbooth siva_krishnan: lyarwood had come at some point. Hassle him about it :)
15:20:22 mriedem EmilienM: shouldn't you be testing at the highest rc1?
15:20:26 mriedem rather than master?
15:20:59 mriedem https://github.com/openstack/nova/compare/15.0.0.0b3...15.0.0.0rc1
15:21:08 mdbooth siva_krishnan: Also, does this need a spec? The spec-level stuff would seem to me to be: 'all drivers should be able to live migrate rescued instances if possible'.
15:21:16 mdbooth +1 agreed :)
15:21:21 dansmith mriedem: oh right, I've been thinking of rc1 not o3
15:21:24 mriedem mdbooth: i think there is already a spec for that...
15:21:46 mdbooth mriedem: Yeah, I was just wondering if the implementation detail of that was suitable for a spec.
15:22:01 EmilienM openstack-nova-common-15.0.0-0.20170208055035.fefac70.el7.centos.noarch
15:22:02 mriedem mdbooth: https://review.openstack.org/#/c/347161/
15:22:05 EmilienM that's our version in CI
15:22:16 EmilienM where fefac70 is the last commit in Nova
15:22:17 mriedem mdbooth: that was accepted in newton, didn't make it, then when reviewing the spec again for ocata i raised some issues
15:22:26 mriedem which weren't resolved and now paul-carlton is gone i think
15:22:30 dansmith EmilienM: that's not a nova commit afaict
15:22:32 mdbooth mriedem: siva_krishnan has a new one
15:22:37 EmilienM which is https://review.openstack.org/#/c/429008/
15:22:40 EmilienM (based on master)
15:22:44 mdbooth https://review.openstack.org/#/c/429913/
15:22:45 dansmith ah
15:22:46 EmilienM dansmith: it is ^
15:23:51 EmilienM what is weird is that https://review.openstack.org/#/c/429008/ is passing tripleo multinode job
15:24:27 EmilienM even gate-tripleo-ci-centos-7-ovb-nonha ...
15:24:28 mdbooth Ah, the new one is actually the old one resurrected
15:24:31 raj_singh mdbooth: mriedem: It is the same spec. We didn't have enough time in Ocata to get it merge and not proposing for Pike
15:24:43 EmilienM so here's my plan. I'm going to unpin Nova in RDO, build images and retry TripleO Ci
15:24:57 raj_singh Hopefully we can resolve the conflict and push it forward
15:24:58 EmilienM maybe there is something the packaging building tool is missing
15:25:02 EmilienM (a dependency or something)
15:25:18 dansmith EmilienM: isn't that what your 429008 patch is doing?
15:25:43 EmilienM dansmith: what surprises me is that 429008 is green and it test master
15:25:48 mdbooth raj_singh: Yeah, that's cool. I still wonder if this needs a spec, though. Doesn't seem very 'architectural', just a bunch of implementation detail. That's not what mriedem is saying, though, so I suggest you don't listen to me :)

Earlier   Later