| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-03-02 | |||
| 15:16:42 | mriedem | sileht: looks like both of those are before you're listing instances | |
| 15:17:25 | mriedem | markus_z: thanks | |
| 15:18:00 | mriedem | markus_z: note that logstash only goes back 10 days | |
| 15:18:04 | mriedem | so it might have been an issue before 2/21 | |
| 15:18:39 | markus_z | mriedem: ohh, ok, I didn't know that this is a hard limit. I'll check earlier changes too. | |
| 15:19:00 | markus_z | mriedem: I'm writing the e-r entry, takes a minute, haven't done it in a while. | |
| 15:19:48 | mriedem | markus_z: actually we probably don't want an e-r query for this, | |
| 15:19:53 | mriedem | it's mostly in the neutron multinode nv job | |
| 15:19:57 | mriedem | we don't have e-r queries for nv jobs | |
| 15:20:07 | mriedem | so it's probably something that just fails in multinode | |
| 15:20:56 | markus_z | mriedem: OK, got it. | |
| 15:23:01 | mriedem | sileht: so something is weird, looking at http://logs.openstack.org/56/439156/2/check/gate-telemetry-dsvm-integration-gnocchi-ubuntu-xenial/d4a6c69/logs/screen-n-api.txt.gz#_2017-03-02_09_16_43_150 where we don't get anything back | |
| 15:23:16 | mriedem | i see "Skipping already-collected cell0" just before that | |
| 15:23:27 | mriedem | which is this https://github.com/openstack/nova/blob/68e808623e42442482e92f0d61a6667876f29e19/nova/compute/api.py#L2538 | |
| 15:23:36 | mriedem | but i don't see this message after it | |
| 15:23:37 | mriedem | https://github.com/openstack/nova/blob/68e808623e42442482e92f0d61a6667876f29e19/nova/compute/api.py#L2540 | |
| 15:23:47 | mriedem | which is where it should be listing instances from the main cell | |
| 15:24:23 | mriedem | sileht: do you have a bug tracking the issue? i wanted to push some debug patches | |
| 15:24:32 | sileht | mriedem, I open it | |
| 15:25:35 | mriedem | so at this point we should have 2 cells | |
| 15:25:35 | mriedem | http://logs.openstack.org/56/439156/2/check/gate-telemetry-dsvm-integration-gnocchi-ubuntu-xenial/d4a6c69/logs/screen-n-api.txt.gz#_2017-03-02_09_13_54_642 | |
| 15:25:38 | mriedem | cell0 and cell1 | |
| 15:26:16 | mriedem | but 3 minutes later, we don't process cell1 http://logs.openstack.org/56/439156/2/check/gate-telemetry-dsvm-integration-gnocchi-ubuntu-xenial/d4a6c69/logs/screen-n-api.txt.gz#_2017-03-02_09_16_43_150 | |
| 15:26:26 | mriedem | which is where the instance would be | |
| 15:27:09 | mriedem | dansmith: melwitt: just thought of something, but on the CELLS cache in the api https://github.com/openstack/nova/blob/68e808623e42442482e92f0d61a6667876f29e19/nova/compute/api.py#L2524 | |
| 15:27:22 | mriedem | once we set that, we don't refresh it w/o restarting nova-api, | |
| 15:27:44 | mriedem | so you could map_cell0, start nova-api, and then create (or add) another cell | |
| 15:27:49 | mriedem | i think that's a known issue | |
| 15:27:52 | dansmith | yeah true | |
| 15:28:03 | dansmith | I was literally just thinking about updating those things | |
| 15:28:24 | dansmith | however, like the service catalog.. that's once per setup | |
| 15:28:28 | mriedem | seems we should re-fetch if len(CELLS) == 1 and CELLS[0].uuid == cell0.uuid | |
| 15:29:31 | mriedem | this is really odd actually | |
| 15:29:35 | mriedem | at this point we have cell0 http://logs.openstack.org/56/439156/2/check/gate-telemetry-dsvm-integration-gnocchi-ubuntu-xenial/d4a6c69/logs/screen-n-api.txt.gz#_2017-03-02_09_07_05_528 | |
| 15:29:52 | mriedem | at this point we have 2 cells http://logs.openstack.org/56/439156/2/check/gate-telemetry-dsvm-integration-gnocchi-ubuntu-xenial/d4a6c69/logs/screen-n-api.txt.gz#_2017-03-02_09_13_54_642 | |
| 15:30:01 | mriedem | that could only happen if nova-api was restarted in between | |
| 15:31:18 | mriedem | i don't see that happening though | |
| 15:31:51 | sileht | mriedem, https://bugs.launchpad.net/nova/+bug/1669473 | |
| 15:31:51 | openstack | Launchpad bug 1669473 in OpenStack Compute (nova) "openstack server list returns an empty list" [Undecided,New] | |
| 15:41:00 | mdbooth | BobBall: Are you the citrix xenserver guy? | |
| 15:41:10 | mriedem | sileht: ok dumped some debug details in the bug, i'm going to work on adding some debug to nova | |
| 15:41:14 | mriedem | mdbooth: he is | |
| 15:41:22 | sileht | mriedem, cool thx | |
| 15:41:24 | mdbooth | Thanks. | |
| 15:41:34 | mdbooth | BobBall: CI looks sad. Are you aware? | |
| 15:43:05 | mdbooth | mriedem: After reviewing your patch this morning, btw, I decided to 2 a cleanup of check_instance_state and found 2 bugs: https://review.openstack.org/#/c/440536/ | |
| 15:43:28 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Log with cell.uuid if cell.name is not set https://review.openstack.org/440644 | |
| 15:43:34 | mriedem | mdbooth: which patch? | |
| 15:43:40 | mriedem | dansmith: ^ easy debug fix patch | |
| 15:43:49 | mdbooth | Nothing earth shattering, but still bugs caused by readability | |
| 15:44:19 | mdbooth | mriedem: https://review.openstack.org/#/c/429476/ | |
| 15:47:11 | mriedem | ah | |
| 15:47:15 | mriedem | yeah check_instance_state is confusing | |
| 15:47:23 | mriedem | i always have to look at that 5 times to figure it out | |
| 15:47:44 | sileht | mriedem, , hum you are setting the global CELLS without lock that quiet dangerous | |
| 15:47:55 | sileht | quiet/quite | |
| 15:48:13 | sileht | mriedem, especially if get_all() does IO | |
| 15:48:19 | mdbooth | mriedem: I looked at it and assumed you were wrong, btw. Then looked 4 more times and worked out you were right :) | |
| 15:48:32 | mriedem | mdbooth: yeah that's par for the course with that one | |
| 15:51:48 | mriedem | sileht: yeah i thought the same - i don't think that's the issue here though since i'm seeing the "found 1 cells" and "found 2 cells" at different points | |
| 15:51:53 | mriedem | that aren't close together | |
| 15:52:06 | mriedem | and i don't see anything that will remove anything from that cache or overwrite it again | |
| 15:52:16 | sileht | mriedem, do you have multiple api workers ? | |
| 15:52:20 | mriedem | if it went from 'found 2 cells' to 'found 1 cells' that would be a definite issue | |
| 15:52:23 | mriedem | yeah | |
| 15:52:24 | mriedem | 2 workers | |
| 15:52:33 | dansmith | it's not super efficient, but it's fine | |
| 15:52:33 | mriedem | oh hmm | |
| 15:52:33 | sileht | mriedem, perhaps one worker have 1 cells, the other 2 | |
| 15:52:37 | mriedem | yeah... | |
| 15:52:43 | mriedem | i bet that's it | |
| 15:53:01 | dansmith | only if we're starting api while we're setting up cells | |
| 15:53:06 | dansmith | which shouldn't be legit | |
| 15:53:54 | mriedem | yeah that's the other thing, cell0 and cell1 should exist before all of this | |
| 15:54:02 | mriedem | we start nova-api here http://logs.openstack.org/56/439156/2/check/gate-telemetry-dsvm-integration-gnocchi-ubuntu-xenial/d4a6c69/logs/devstacklog.txt.gz#_2017-03-02_08_59_34_428 | |
| 15:54:06 | mriedem | 2017-03-02 08:59:34.428 | + functions-common:run_process:1491 : _run_process n-api /usr/local/bin/nova-api '' | |
| 15:54:22 | mriedem | we map cell0 here: http://logs.openstack.org/56/439156/2/check/gate-telemetry-dsvm-integration-gnocchi-ubuntu-xenial/d4a6c69/logs/devstacklog.txt.gz#_2017-03-02_08_54_16_288 | |
| 15:54:28 | mriedem | 2017-03-02 08:54:16.288 | + lib/nova:init_nova:711 : nova-manage cell_v2 map_cell0 --database_connection 'mysql+pymysql://root:secretmysql@127.0.0.1/nova_cell0?charset=utf8' | |
| 15:54:40 | mriedem | hold the phone | |
| 15:54:41 | mriedem | http://logs.openstack.org/56/439156/2/check/gate-telemetry-dsvm-integration-gnocchi-ubuntu-xenial/d4a6c69/logs/devstacklog.txt.gz#_2017-03-02_09_08_19_738 | |
| 15:54:57 | mriedem | 2017-03-02 09:08:19.738 | + lib/nova:create_cell:974 : nova-manage cell_v2 simple_cell_setup --transport-url rabbit://stackrabbit:secretrabbit@104.130.199.12:5672/ | |
| 15:55:05 | mriedem | n-api is running before cell1 is created | |
| 15:55:30 | mriedem | so any request before that is going to only see cell0 and cache it | |
| 15:55:42 | mriedem | any request after that doesn't have the cache on that worker is going to get the 2 cells | |
| 15:55:55 | dansmith | is that just with their plugin thing or upstream? I don't see how that's possible with base devstack | |
| 15:55:58 | BobBall | mdbooth: Thanks - looking. | |
| 15:56:21 | sileht | anyways, assuming nobody must call nova-api to not fall into that issue, looks like a serious bug | |
| 15:56:56 | BobBall | mdbooth: ci-watch.tintri.com shows a small blip earlier today with 7 failures, but it's OK now. Got some specific sadness that you're worried about? | |
| 15:57:24 | mdbooth | BobBall: This one looked pretty sad: https://review.openstack.org/#/c/440536/ | |
| 15:57:37 | mriedem | dansmith: i assume with their plugin | |
| 15:57:54 | mdbooth | If it was just a blip I'll just resubmit. Didn't want to add to a pile if it was an ongoing problem, though. | |
| 15:58:09 | sileht | mriedem, it seems ceilometer poll nova just before nova-manage cell_v2 simple_cell_setup | |
| 15:58:20 | BobBall | mdbooth: That is quite an unhappy test :( Checking for other issues | |
| 15:58:27 | sileht | mriedem, here: http://logs.openstack.org/56/439156/2/check/gate-telemetry-dsvm-integration-gnocchi-ubuntu-xenial/d4a6c69/logs/screen-n-api.txt.gz#_2017-03-02_09_07_05_559 | |
| 15:58:30 | BobBall | I mean other chnages with the same issue | |
| 15:59:58 | mriedem | sileht: yeah | |
| 16:00:01 | mriedem | ok working on something here we can test with | |
| 16:00:33 | BobBall | mdbooth: The blip earlier today was ansible related - but that's now resolved. I don't see any other recent failures with your problem | |
| 16:00:50 | mdbooth | BobBall: Ok, then. Once more into the breach :) | |
| 16:01:05 | mdbooth | BobBall: Thanks for looking. | |