| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-02-08 | |||
| 14:13:05 | EmilienM | at 06:38:25 we run nova-cell_v2-discover_hosts | |
| 14:13:21 | EmilienM | but we start nova compute at 06:32:06 | |
| 14:13:31 | johnthetubaguy | EmilienM: I think the three is being registered in the cell, registered in placement and registered in the API | |
| 14:13:46 | johnthetubaguy | let me find the compute log, one sec... | |
| 14:13:49 | EmilienM | we think nova compute is started at http://logs.openstack.org/71/430571/1/check/gate-tripleo-ci-centos-7-nonha-multinode/bae4746/logs/subnode-2/var/log/nova/nova-compute.txt.gz#_2017-02-08_06_34_19_476 | |
| 14:14:28 | johnthetubaguy | http://logs.openstack.org/71/430571/1/check/gate-tripleo-ci-centos-7-nonha-multinode/bae4746/logs/subnode-2/var/log/nova/nova-compute.txt.gz#_2017-02-08_06_30_59_875 | |
| 14:14:56 | johnthetubaguy | thats where you see the compute node being created, i.e. registered in the child cell | |
| 14:15:01 | johnthetubaguy | you need to discover hosts after that | |
| 14:15:06 | johnthetubaguy | I think | |
| 14:16:14 | johnthetubaguy | hmm, I guess I don't understand the state of the system | |
| 14:16:28 | johnthetubaguy | it looks more like placement is not returning any hosts, I could be wrong | |
| 14:16:48 | owalsh | johnthetubaguy: I don't see any POST requests in the placement api log | |
| 14:17:07 | cdent | EmilienM, johnthetubaguy: I'm back, catching up. which is the best log to start with? | |
| 14:17:36 | johnthetubaguy | owalsh: yeah, compute logs say its not configured in the compute log, at least in the one I am looking at | |
| 14:17:53 | johnthetubaguy | owalsh: http://logs.openstack.org/71/430571/1/check/gate-tripleo-ci-centos-7-nonha-multinode/bae4746/logs/subnode-2/var/log/nova/nova-compute.txt.gz#_2017-02-08_06_34_01_297 | |
| 14:18:20 | owalsh | johnthetubaguy: service isn't up yet | |
| 14:18:45 | johnthetubaguy | but thats a different log | |
| 14:19:06 | johnthetubaguy | I still see error here, looks like its configured by not able to reach it maybe | |
| 14:19:07 | johnthetubaguy | http://logs.openstack.org/71/430571/1/check/gate-tripleo-ci-centos-7-nonha-multinode/bae4746/logs/subnode-2/var/log/nova/nova-compute.txt.gz#_2017-02-08_06_43_28_610 | |
| 14:19:25 | johnthetubaguy | owalsh: we look at the public endpoint in the service catalog for placement, in case that helps | |
| 14:19:35 | cdent | this is an error we were seeing a while back: http://logs.openstack.org/71/430571/1/check/gate-tripleo-ci-centos-7-nonha-multinode/bae4746/logs/subnode-2/var/log/nova/nova-compute.txt.gz#_2017-02-08_06_34_19_311 | |
| 14:19:58 | johnthetubaguy | been looking at stuff with openstack-ansible folks last week, so wondering if its similar | |
| 14:23:22 | owalsh | johnthetubaguy: looks like it's created at 06:37:11 http://logs.openstack.org/71/430571/1/check/gate-tripleo-ci-centos-7-nonha-multinode/bae4746/logs/subnode-2/var/log/keystone/keystone.txt.gz#_2017-02-08_06_37_11_641 | |
| 14:25:29 | owalsh | johnthetubaguy, cdent: and no more errors about the endpoint after this | |
| 14:25:37 | owalsh | http://logs.openstack.org/71/430571/1/check/gate-tripleo-ci-centos-7-nonha-multinode/bae4746/logs/subnode-2/var/log/nova/nova-compute.txt.gz#_2017-02-08_06_38_22_609 | |
| 14:27:22 | sfinucan | sdague: nova-dhcpbridge only makes sense with nova-network, correct? https://linux.die.net/man/1/nova-dhcpbridge | |
| 14:31:12 | EmilienM | cdent, johnthetubaguy, owalsh: could it be something in nova.conf [placement] ? | |
| 14:31:35 | cdent | EmilienM: maybe | |
| 14:32:15 | EmilienM | cdent: the thing it, when we deploy nova from ocata-3, it works fine. Let me show you tripleo logs from our CI where nova trunk is not used | |
| 14:32:37 | johnthetubaguy | are those placement credentials passing the placement is_admin check, for example? | |
| 14:33:12 | EmilienM | cdent: http://logs.openstack.org/47/430447/1/check/gate-tripleo-ci-centos-7-nonha-multinode/dd43509/logs/subnode-2/var/log/nova/nova-placement-api.txt.gz#_2017-02-07_23_06_08_825 | |
| 14:33:15 | johnthetubaguy | EmilienM: thats probably just after we starting asking placement which host to use, potentially | |
| 14:33:24 | EmilienM | cdent: that is a CI job that doesn't run nova from trunk, but nova from ocata-3 | |
| 14:33:59 | EmilienM | johnthetubaguy: what is is_admin check ? | |
| 14:34:07 | cdent | EmilienM: can you enumerate what's different on the tripleo side, besides the switch to using nova-api with eventel (if anything)? | |
| 14:34:19 | mordred | sdague: something like this: https://review.openstack.org/430967 | |
| 14:34:19 | EmilienM | oh is_admin in keystone | |
| 14:34:30 | johnthetubaguy | EmilienM: placement logs didn't show any 403s so I am wrong, I was thinking is it an admin user | |
| 14:34:41 | Guest94046 | johnthetubaguy: yep, o3 won't break without placement | |
| 14:38:08 | EmilienM | cdent: what do you want to know exactly?? the diff between the 2 jobs? or the history of what we changed recently? | |
| 14:40:30 | cdent | EmilienM: I'm not sure. It would just be useful to be able to figure out what if anything may have impacted what's going on. | |
| 14:40:45 | EmilienM | the major thing is that we switched back to eventlet | |
| 14:42:58 | dansmith | EmilienM: not for n-cpu which is where this problem is, right? | |
| 14:44:03 | EmilienM | dansmith: I'm afraid I don't understand the question. We switched Nova API to not run in WSGI with apache, but run in classic eventlet. What are you asking about nova compute? | |
| 14:44:35 | dansmith | EmilienM: the problem is that nova-compute is not reporting into placement for the subnode right? | |
| 14:44:44 | johnthetubaguy | FWIW, this error log on the first compute node seems odd to me, not sure if thats normal: http://logs.openstack.org/71/430571/1/check/gate-tripleo-ci-centos-7-nonha-multinode/bae4746/logs/var/log/nova/nova-compute.txt.gz#_2017-02-08_06_43_11_488 | |
| 14:44:48 | EmilienM | cdent: in the logs, I see "Placement service credentials do not work." 3 times but not more so I guess it could register at some point? | |
| 14:45:16 | mriedem | johnthetubaguy: http://logs.openstack.org/71/430571/1/check/gate-tripleo-ci-centos-7-nonha-multinode/bae4746/logs/var/log/nova/nova-compute.txt.gz#_2017-02-08_06_43_11_488 is 'normal' | |
| 14:45:23 | mriedem | we should probably drop that down from an error | |
| 14:45:26 | EmilienM | dansmith: the subnode is a single node overcloud, consider it as a all-in-one node | |
| 14:45:40 | cdent | EmilienM: there's no evidence that the placement application has ever been contacted (in the web logs), unless I'm looking at the wrong ones | |
| 14:45:49 | mriedem | johnthetubaguy: https://github.com/openstack/nova/blob/d48aeb5783ea9bd9495eb2e7c67bc587dfd7d85d/nova/compute/manager.py#L6597 | |
| 14:45:56 | EmilienM | dansmith: node1 == undercloud, which is completely another story (and by the way, undercloud can deploy the overcloud fine in OVB jobs, that run Nova / Ironic) | |
| 14:45:57 | dansmith | the passwords in the two nodes for placement look very different | |
| 14:46:19 | EmilienM | dansmith: yeah don't look at subnode1 please | |
| 14:46:27 | dansmith | alright | |
| 14:46:29 | EmilienM | dansmith: that's another instance of nova, that we don't use in this case | |
| 14:46:42 | EmilienM | subnode2 is the only logs we should look today | |
| 14:46:57 | EmilienM | http://logs.openstack.org/71/430571/1/check/gate-tripleo-ci-centos-7-nonha-multinode/bae4746/logs/subnode-2/var/log/nova/ | |
| 14:47:15 | johnthetubaguy | mriedem: OK, I clearly have less understanding of how this works than I thought | |
| 14:47:22 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Improve assertJsonEqual error reporting https://review.openstack.org/428199 | |
| 14:47:23 | dansmith | oh, I'm not looking at subnode1 | |
| 14:47:28 | dansmith | I'm looking at -2 | |
| 14:47:33 | EmilienM | ok good | |
| 14:47:43 | dansmith | or are you calling the primary node "subnode1" ? | |
| 14:48:21 | EmilienM | so: http://logs.openstack.org/71/430571/1/check/gate-tripleo-ci-centos-7-nonha-multinode/bae4746/logs/var/log/nova/ is the undercloud and please don't look at it we don't care now | |
| 14:48:36 | EmilienM | http://logs.openstack.org/71/430571/1/check/gate-tripleo-ci-centos-7-nonha-multinode/bae4746/logs/subnode-2/var/log/nova/ is overcloud, and we currently can't spawn a VM on it | |
| 14:48:49 | johnthetubaguy | according to placement, its only ever been reached by the scheduler I guess | |
| 14:48:53 | EmilienM | FYI I'm using https://review.openstack.org/#/c/430571/ for testing | |
| 14:49:26 | cdent | johnthetubaguy: yes (I was looking at the main logs when I saw no activity above, not the subnode-2, but subnode-2 is just hearing from the scheduler) | |
| 14:49:41 | johnthetubaguy | I just wish we had more details on why compute isn't able to talk to placement in here: http://logs.openstack.org/71/430571/1/check/gate-tripleo-ci-centos-7-nonha-multinode/bae4746/logs/subnode-2/var/log/nova/nova-compute.txt.gz#_2017-02-08_06_43_28_610 | |
| 14:50:20 | cdent | johnthetubaguy: shortly before that; http://logs.openstack.org/71/430571/1/check/gate-tripleo-ci-centos-7-nonha-multinode/bae4746/logs/subnode-2/var/log/nova/nova-compute.txt.gz#_2017-02-08_06_40_24_634 | |
| 14:50:33 | dansmith | ot | |
| 14:50:42 | dansmith | it is clearly hitting placement because it's getting Unauthorized right? | |
| 14:51:02 | dansmith | which means that the service catalog is working, and that the credentials just aren't right I think | |
| 14:51:16 | dansmith | unless it's hitting placement for the undercloud somehow, but I doubt it | |
| 14:51:34 | cdent | dansmith: where is unauthorized happening? | |
| 14:51:41 | EmilienM | dansmith: our current CI jobs (pin to nova-ocata-3) use the same credentials, and it seems to work. I can give you a link so you can confirm | |
| 14:51:55 | dansmith | cdent: https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L70-L76 | |
| 14:52:02 | dansmith | cdent: that's the message in the logs, yes? | |
| 14:52:03 | johnthetubaguy | EmilienM: it is probably just silently failing | |
| 14:52:25 | johnthetubaguy | EmilienM: would wouldn't notice anything wrong if compute couldn't ever talk to placement until after ocata-3 | |
| 14:52:39 | EmilienM | ok now i'm checking our haproxy config, etc to make sure placement api is reachable | |
| 14:52:50 | dansmith | cdent: placement access log is completely empty on the undercloud? | |
| 14:53:01 | EmilienM | yeah | |
| 14:53:06 | EmilienM | please don't look undercloud for multinode jobs | |
| 14:53:11 | EmilienM | nova is not used to spawn the overcloud | |
| 14:53:14 | johnthetubaguy | dansmith: its odd we don't see 401 in the placement logs, or is that normal? | |
| 14:53:16 | EmilienM | we use nodepool to spawn the subnode2 | |
| 14:53:20 | dansmith | johnthetubaguy: I dunno | |
| 14:53:39 | cdent | ks_exc.Unauthorized means two things | |
| 14:53:43 | EmilienM | if you want to read logs from a real undercloud that spawns an undercloud, please look OVB jobs: https://review.openstack.org/#/c/430571/ | |
| 14:53:46 | mordred | cdent: of course it does | |
| 14:53:48 | johnthetubaguy | I wonder if its not actually got the correct url from the service catalog, or maybe it just can't talk to keystone to get it? | |
| 14:53:55 | cdent | one of them is entirely driven by code, not a response from anything | |
| 14:53:55 | dansmith | johnthetubaguy: placement access log on the undercloud is zero bytes, but we do have a few gets in the overcloud log | |
| 14:53:59 | cdent | mordred: ikr | |
| 14:54:17 | johnthetubaguy | dansmith: looks like they are the scheduler calls that work, yeah | |
| 14:54:31 | johnthetubaguy | dansmith: well the empty list would give us the no-valid host error I think | |
| 14:54:44 | EmilienM | I don't think it can comes from HAproxy because nova-placement-api log file exists, so API was reached | |