Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-03-02
15:43:34 mriedem mdbooth: which patch?
15:43:40 mriedem dansmith: ^ easy debug fix patch
15:43:49 mdbooth Nothing earth shattering, but still bugs caused by readability
15:44:19 mdbooth mriedem: https://review.openstack.org/#/c/429476/
15:47:11 mriedem ah
15:47:15 mriedem yeah check_instance_state is confusing
15:47:23 mriedem i always have to look at that 5 times to figure it out
15:47:44 sileht mriedem, , hum you are setting the global CELLS without lock that quiet dangerous
15:47:55 sileht quiet/quite
15:48:13 sileht mriedem, especially if get_all() does IO
15:48:19 mdbooth mriedem: I looked at it and assumed you were wrong, btw. Then looked 4 more times and worked out you were right :)
15:48:32 mriedem mdbooth: yeah that's par for the course with that one
15:51:48 mriedem sileht: yeah i thought the same - i don't think that's the issue here though since i'm seeing the "found 1 cells" and "found 2 cells" at different points
15:51:53 mriedem that aren't close together
15:52:06 mriedem and i don't see anything that will remove anything from that cache or overwrite it again
15:52:16 sileht mriedem, do you have multiple api workers ?
15:52:20 mriedem if it went from 'found 2 cells' to 'found 1 cells' that would be a definite issue
15:52:23 mriedem yeah
15:52:24 mriedem 2 workers
15:52:33 dansmith it's not super efficient, but it's fine
15:52:33 mriedem oh hmm
15:52:33 sileht mriedem, perhaps one worker have 1 cells, the other 2
15:52:37 mriedem yeah...
15:52:43 mriedem i bet that's it
15:53:01 dansmith only if we're starting api while we're setting up cells
15:53:06 dansmith which shouldn't be legit
15:53:54 mriedem yeah that's the other thing, cell0 and cell1 should exist before all of this
15:54:02 mriedem we start nova-api here http://logs.openstack.org/56/439156/2/check/gate-telemetry-dsvm-integration-gnocchi-ubuntu-xenial/d4a6c69/logs/devstacklog.txt.gz#_2017-03-02_08_59_34_428
15:54:06 mriedem 2017-03-02 08:59:34.428 | + functions-common:run_process:1491 : _run_process n-api /usr/local/bin/nova-api ''
15:54:22 mriedem we map cell0 here: http://logs.openstack.org/56/439156/2/check/gate-telemetry-dsvm-integration-gnocchi-ubuntu-xenial/d4a6c69/logs/devstacklog.txt.gz#_2017-03-02_08_54_16_288
15:54:28 mriedem 2017-03-02 08:54:16.288 | + lib/nova:init_nova:711 : nova-manage cell_v2 map_cell0 --database_connection 'mysql+pymysql://root:secretmysql@127.0.0.1/nova_cell0?charset=utf8'
15:54:40 mriedem hold the phone
15:54:41 mriedem http://logs.openstack.org/56/439156/2/check/gate-telemetry-dsvm-integration-gnocchi-ubuntu-xenial/d4a6c69/logs/devstacklog.txt.gz#_2017-03-02_09_08_19_738
15:54:57 mriedem 2017-03-02 09:08:19.738 | + lib/nova:create_cell:974 : nova-manage cell_v2 simple_cell_setup --transport-url rabbit://stackrabbit:secretrabbit@104.130.199.12:5672/
15:55:05 mriedem n-api is running before cell1 is created
15:55:30 mriedem so any request before that is going to only see cell0 and cache it
15:55:42 mriedem any request after that doesn't have the cache on that worker is going to get the 2 cells
15:55:55 dansmith is that just with their plugin thing or upstream? I don't see how that's possible with base devstack
15:55:58 BobBall mdbooth: Thanks - looking.
15:56:21 sileht anyways, assuming nobody must call nova-api to not fall into that issue, looks like a serious bug
15:56:56 BobBall mdbooth: ci-watch.tintri.com shows a small blip earlier today with 7 failures, but it's OK now. Got some specific sadness that you're worried about?
15:57:24 mdbooth BobBall: This one looked pretty sad: https://review.openstack.org/#/c/440536/
15:57:37 mriedem dansmith: i assume with their plugin
15:57:54 mdbooth If it was just a blip I'll just resubmit. Didn't want to add to a pile if it was an ongoing problem, though.
15:58:09 sileht mriedem, it seems ceilometer poll nova just before nova-manage cell_v2 simple_cell_setup
15:58:20 BobBall mdbooth: That is quite an unhappy test :( Checking for other issues
15:58:27 sileht mriedem, here: http://logs.openstack.org/56/439156/2/check/gate-telemetry-dsvm-integration-gnocchi-ubuntu-xenial/d4a6c69/logs/screen-n-api.txt.gz#_2017-03-02_09_07_05_559
15:58:30 BobBall I mean other chnages with the same issue
15:59:58 mriedem sileht: yeah
16:00:01 mriedem ok working on something here we can test with
16:00:33 BobBall mdbooth: The blip earlier today was ansible related - but that's now resolved. I don't see any other recent failures with your problem
16:00:50 mdbooth BobBall: Ok, then. Once more into the breach :)
16:01:05 mdbooth BobBall: Thanks for looking.
16:03:15 sileht mdbooth, in devstack, why create_cell is ran after 'run_phase stack extra' ?
16:03:18 sileht mdbooth, sorry
16:03:20 sileht mriedem, ^
16:04:40 mriedem sileht: i think that's just legacy from that being where the cells v1 setup happened
16:04:51 mriedem which probably no other devstack plugin based job was using cells v1
16:06:37 sileht mriedem, I sent a devstack patch as workaround
16:07:00 sileht mriedem, but using a global that way looks very dangerous from my point of view
16:08:29 sileht mriedem, https://review.openstack.org/440651
16:08:50 sileht mriedem, I don't close the bug, since the root cause is more serious
16:09:53 openstackgerrit Matt Riedemann proposed openstack/nova master: Log with cell.uuid if cell.name is not set https://review.openstack.org/440644
16:09:53 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Update CELLS cache if we only have cell0 https://review.openstack.org/440652
16:09:54 mriedem sileht: can you post a gnocchi patch that depends on ^ to see if it helps?
16:12:07 sileht mriedem, https://review.openstack.org/440657
16:12:38 openstackgerrit Sujitha proposed openstack/nova master: Remove mox from nova.tests.unit.api.openstack.compute.test_create_backup https://review.openstack.org/435610
16:12:41 jaypipes edleafe, alex_xu, bauzas, mriedem, dansmith: https://review.openstack.org/440654, https://review.openstack.org/440655, https://review.openstack.org/440656
16:13:15 bauzas jaypipes: oh cools
16:13:26 edleafe jaypipes: will review
16:13:52 jaypipes danke
16:14:40 mriedem sileht: you know, https://review.openstack.org/#/c/427392/ is probably the thing that triggered this bug
16:14:53 mriedem sileht: because before that, cell0 wouldn't be mapped and nova-api wouldn't list any instances
16:16:21 sileht mriedem, I think we having another behavior before, and the issue was not so often
16:17:01 zioproto dansmith: are you online ?
16:17:10 sileht mriedem, thanks for your help
16:18:24 mriedem sileht: yeah i mean it's still a bug, but also, if we didn't have cell1 created yet and you tried creating a server, it would fail because we have nowhere to place it
16:18:36 mriedem but you could list servers and get nothing back, and that would be fine if you never tried creating any servers
16:18:53 mriedem which i think that's what ceilometer was doing
16:18:55 mriedem just listing servers
16:18:58 mriedem on startup or something, not sure
16:19:17 mriedem but that GET call was caching cell0 whereas before https://review.openstack.org/#/c/427392/ we didn't have cell0 mapped so nothing was cached
16:23:08 bauzas jaypipes: you're hitting the pbr issue for os-traits lib
16:23:45 mriedem jaypipes: yeah you need to update the hacking version here https://github.com/jaypipes/os-traits/blob/master/test-requirements.txt#L5
16:23:58 mriedem http://git.openstack.org/cgit/openstack/requirements/tree/global-requirements.txt#n424
16:24:16 openstackgerrit Balazs Gibizer proposed openstack/nova master: handle uninited fields in notification payload https://review.openstack.org/415857
16:24:22 openstackgerrit Arnaud Morin proposed openstack/nova master: Set default value to HTTP_X_FORWARDED_PROTO https://review.openstack.org/423375
16:28:12 jaypipes mriedem: thx for the heads up, will fix up now.
16:29:15 mriedem dansmith: before i write tests, how much do you hate this? https://review.openstack.org/#/c/440652/
16:29:35 mriedem trying to think of ways to auto-invalidate or update that CELLS cache when a new cell mapping is created
16:30:09 dansmith mriedem: one cell is not valid regardless of why so I think you need only check the length
16:30:31 dansmith I still assert this is a silly workaround, but if you want it that's fine
16:31:31 mriedem i don't know that i want it
16:31:40 mriedem the devstack change would also unblock gnocchi
16:31:41 mriedem i think
16:35:44 dansmith mriedem: what was the change in the last day or so that caused this?
16:35:57 mriedem dansmith: https://review.openstack.org/#/c/427392/
16:36:16 dansmith ah
16:36:30 dansmith that caused me trouble too
16:38:20 jaypipes edleafe, alex_xu, bauzas, mriedem, dansmith: fixed up pep8/hacking thing. https://review.openstack.org/440678 now new root of series.

Earlier   Later