| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-02-23 | |||
| 21:00:52 | mriedem | and site https://launchpad.net/~ubuntu-cloud-archive/+archive/ubuntu/ocata-staging/+sourcepub/8774271/+listing-archive-extra | |
| 21:01:10 | mnaser | mriedem: i can do that too, probably easier and quicker | |
| 21:01:22 | mriedem | *cite ? | |
| 21:01:31 | mriedem | mnaser: it would be easier and quicker for now yeah, | |
| 21:01:40 | mriedem | could get mtreinish to take a gander | |
| 21:01:59 | mriedem | alternatively we could run the pike UCA on stable/pike... | |
| 21:02:13 | andreaf | mriedem mnaser there's a patch up to make that value configurable already | |
| 21:02:20 | mriedem | https://review.openstack.org/#/c/536798/ | |
| 21:02:32 | andreaf | https://review.openstack.org/#/c/547431/ | |
| 21:02:38 | andreaf | mriedem mnaser ^^^ | |
| 21:02:39 | mnaser | mriedem: grenade uses stable/ocata though | |
| 21:02:59 | mriedem | mnaser: yeah the old side does | |
| 21:03:00 | mriedem | true | |
| 21:03:30 | mnaser | andreaf: thats a nice patch, what does the gate use for SERVICE_TIMEOUT ? | |
| 21:03:39 | mriedem | it defaults to 60 | |
| 21:04:18 | mnaser | so i guess we use that patch+backport+bump up SERVICE_TIMEOUT in stable/ocata + stable/pike ? | |
| 21:04:55 | mriedem | i guess...i'm not sure how much people are going to want to bump the default timeout | |
| 21:04:58 | mriedem | globally i mean | |
| 21:05:16 | mriedem | that seems worse than just hard-coding the wait_for_compute timeout to 120 | |
| 21:05:51 | mnaser | yeah i think i'd want to bump up wait_for_compute because then everything will wait 120s which might be hiding regressions | |
| 21:07:21 | mriedem | so i suggest a new stackrc variable that defaults to $SERVICE_TIMEOUT, but that we can bump to be higher in stable/ocata/pike if we want | |
| 21:07:30 | mriedem | noted that in frickler's patch | |
| 21:14:41 | andreaf | mriedem this is the log line in devstack where there is some sign of the issue going on I think http://logs.openstack.org/24/545724/9/check/tempest-multinode-full/1bbec81/compute1/logs/devstacklog.txt.gz#_2018-02-23_17_22_21_709 | |
| 21:15:16 | andreaf | mriedem but I few lines later it seems to be fine | |
| 21:15:19 | mnaser | mriedem, andreaf, mtreinish: https://review.openstack.org/547431 Create NOVA_COMPUTE_SERVICE_TIMEOUT in is_nova_ready function | |
| 21:15:20 | mriedem | that's normal | |
| 21:15:38 | mriedem | andreaf: n-cpu barfs that the compute node that represents the local host doesn't exist, and then it creates it | |
| 21:15:42 | mriedem | and stops the barfing | |
| 21:15:42 | mnaser | if that can be approved, i can backport it and then bump it in stable/ocata and stable/pike | |
| 21:16:33 | andreaf | mriedem so I don't know why I get http://logs.openstack.org/24/545724/9/check/tempest-multinode-full/1bbec81/job-output.txt.gz#_2018-02-23_17_22_40_367118 on every single VM created in that job | |
| 21:18:37 | mriedem | is it every one or just the subnode? | |
| 21:18:43 | mriedem | ubuntu-xenial-inap-mtl01-0002692907 is the subnode | |
| 21:19:20 | andreaf | mriedem actually every single VM on the 907 node | |
| 21:19:44 | mriedem | andreaf: so this is the warning you get on first startup | |
| 21:19:45 | mriedem | http://logs.openstack.org/24/545724/9/check/tempest-multinode-full/1bbec81/compute1/logs/screen-n-cpu.txt.gz?level=INFO#_Feb_23_17_22_20_954537 | |
| 21:19:50 | mriedem | No compute node record found for host ubuntu-xenial-inap-mtl01-0002692907. If this is the first time this service is starting on this host, then you can ignore this warning.: ComputeHostNotFound_Remote: Compute host ubuntu-xenial-inap-mtl01-0002692907 could not be found. | |
| 21:20:03 | mriedem | then the compute node record is created | |
| 21:20:04 | mriedem | http://logs.openstack.org/24/545724/9/check/tempest-multinode-full/1bbec81/compute1/logs/screen-n-cpu.txt.gz?level=INFO#_Feb_23_17_22_21_021412 | |
| 21:20:09 | mriedem | Feb 23 17:22:21.021412 ubuntu-xenial-inap-mtl01-0002692907 nova-compute[21947]: INFO nova.compute.resource_tracker [None req-5ceeacfd-f9f6-4b19-9044-6fd13c9adadc None None] Compute node record created for ubuntu-xenial-inap-mtl01-0002692907:ubuntu-xenial-inap-mtl01-0002692907 with uuid: 85868727-6173-4345-bd2e-a92bfaf1de8a | |
| 21:20:29 | openstackgerrit | Matthew Edmonds proposed openstack/nova master: Fix N358 hacking check https://review.openstack.org/547670 | |
| 21:21:06 | mriedem | http://logs.openstack.org/24/545724/9/check/tempest-multinode-full/1bbec81/compute1/logs/devstacklog.txt.gz#_2018-02-23_17_22_18_880 | |
| 21:21:06 | mriedem | on that subnode, wait_for_compute starts here: | |
| 21:21:11 | mriedem | 2018-02-23 17:22:18.880 | + lib/nova:is_nova_ready:1003 : wait_for_compute 60 | |
| 21:21:21 | mriedem | and ends here | |
| 21:21:22 | mriedem | 2018-02-23 17:22:21.607 | + functions:wait_for_compute:463 : return 0 | |
| 21:21:56 | andreaf | so it's successful | |
| 21:21:58 | mriedem | so in that case, it looks like the compute node record was created before devstack started waiting | |
| 21:22:58 | mnaser | mriedem: round 2 https://review.openstack.org/#/c/547431/ | |
| 21:23:07 | andreaf | right... so I'm missing what's going wrong... | |
| 21:23:49 | mnaser | its friday and the servers are lazy | |
| 21:23:51 | mriedem | http://logs.openstack.org/24/545724/9/check/tempest-multinode-full/1bbec81/controller/logs/devstacklog.txt.gz#_2018-02-23_17_13_29_978 | |
| 21:23:51 | mriedem | the host mapping for the n-cpu on the controller node is created here: | |
| 21:23:57 | mriedem | 2018-02-23 17:13:29.978 | Creating host mapping for compute host 'ubuntu-xenial-inap-mtl01-0002692906': adbcb697-55fc-4750-9242-b13aa75cab07 | |
| 21:24:28 | mriedem | discover_hosts doesn't run on the subnode | |
| 21:24:39 | mriedem | and i don't see discover_hosts running after the subnode compute is created to discover it | |
| 21:25:22 | mnaser | oh that's why its different | |
| 21:25:26 | mriedem | andreaf: you'd likely need to compare to an existing multinode job and see where discover_hosts runs, | |
| 21:25:30 | mnaser | grenade uses simple_cell_setup which does discover_hosts | |
| 21:25:37 | mriedem | because discover_hosts *has* to be run every time you startup a new n-cpu | |
| 21:25:45 | andreaf | mriedem yeah that's what I was about to do :) | |
| 21:26:08 | andreaf | should it run on the controller? | |
| 21:26:32 | andreaf | discover_host is a devstack function right? | |
| 21:26:52 | mnaser | andreaf: discover_hosts is a nova-manage cli func | |
| 21:26:59 | mnaser | nova-manage cell_v2 discover_hosts | |
| 21:27:03 | mnaser | it runs on the controller after you add new computes | |
| 21:27:05 | mriedem | andreaf: OH I KNOW THE PROBLEM!!! | |
| 21:27:15 | mriedem | your problem is zuulv3 | |
| 21:27:16 | mriedem | :) | |
| 21:27:22 | andreaf | heh | |
| 21:27:26 | mnaser | i thought that was the solution for everything | |
| 21:27:27 | mriedem | devstack-gate would run discover_hosts on the primary after the subnode was setup | |
| 21:27:42 | mriedem | the devstack multinode guide says this too | |
| 21:27:56 | mriedem | i'll look for the d-g code | |
| 21:28:01 | andreaf | mriedem oh, right | |
| 21:28:07 | andreaf | mriedem I missed that in d-g | |
| 21:28:24 | mriedem | https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate.sh#L783 | |
| 21:28:28 | mriedem | andreaf: ^ | |
| 21:28:34 | mriedem | d-g would run discover_hosts after the subnode was setup | |
| 21:28:48 | mriedem | so you'll need an ansible task to run discover_hosts on the primary node once the subnode is setup | |
| 21:28:50 | mriedem | that should do the trick | |
| 21:29:24 | andreaf | mriedem heh the one line I did not port to ansible! | |
| 21:29:48 | andreaf | mriedem I guess I kind of assumed cells did not matter for the base job :( but it's cells v2 | |
| 21:29:50 | mriedem | mnaser: need to fix the commit message | |
| 21:29:58 | mriedem | cells v2 is king | |
| 21:30:11 | andreaf | mriedem cool thanks a lot, much appreciated!! | |
| 21:30:11 | mnaser | oops | |
| 21:30:15 | mriedem | yw | |
| 21:31:02 | andreaf | mriedem # NOTE(mriedem): We want to remove this if/when nova supports auto-registration of computes with cells, but that's not happening in Ocata. | |
| 21:31:12 | mnaser | ok fixed the commit msg, andreaf i'd borrow your +2 back please https://review.openstack.org/#/c/547431/ :) | |
| 21:31:18 | andreaf | mriedem I guess it's not happening in pike or queens either :D | |
| 21:31:25 | mnaser | s/Ocata/a while/ | |
| 21:31:59 | andreaf | mnaser +2 | |
| 21:32:08 | mnaser | thank you | |
| 21:33:21 | mriedem | andreaf: right | |
| 21:34:01 | mriedem | andreaf: also we don't run discover_hosts from the subnode because it's not configured to have access to the nova_api db | |
| 21:34:03 | mriedem | on purpose | |
| 21:34:48 | mriedem | we want to make sure that when people like leakypipes try to add code to nova-compute that hits the API it blows up :P | |
| 21:47:16 | leakypipes | mriedem: ha. ha. | |
| 21:52:43 | cfriesen | if we tell nova to boot an instance (specifying that we want interfaces on two networks), then the instance fails to start, so we reschedule to another node, would we expect to re-use the ports that were created for the first compute node? | |
| 21:58:59 | cfriesen | answering my own question, based on a comment in ComputeManager._build_networks_for_instance() the answer seems to be "yes" | |