Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-03-02
16:04:51 mriedem which probably no other devstack plugin based job was using cells v1
16:06:37 sileht mriedem, I sent a devstack patch as workaround
16:07:00 sileht mriedem, but using a global that way looks very dangerous from my point of view
16:08:29 sileht mriedem, https://review.openstack.org/440651
16:08:50 sileht mriedem, I don't close the bug, since the root cause is more serious
16:09:53 openstackgerrit Matt Riedemann proposed openstack/nova master: Log with cell.uuid if cell.name is not set https://review.openstack.org/440644
16:09:53 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Update CELLS cache if we only have cell0 https://review.openstack.org/440652
16:09:54 mriedem sileht: can you post a gnocchi patch that depends on ^ to see if it helps?
16:12:07 sileht mriedem, https://review.openstack.org/440657
16:12:38 openstackgerrit Sujitha proposed openstack/nova master: Remove mox from nova.tests.unit.api.openstack.compute.test_create_backup https://review.openstack.org/435610
16:12:41 jaypipes edleafe, alex_xu, bauzas, mriedem, dansmith: https://review.openstack.org/440654, https://review.openstack.org/440655, https://review.openstack.org/440656
16:13:15 bauzas jaypipes: oh cools
16:13:26 edleafe jaypipes: will review
16:13:52 jaypipes danke
16:14:40 mriedem sileht: you know, https://review.openstack.org/#/c/427392/ is probably the thing that triggered this bug
16:14:53 mriedem sileht: because before that, cell0 wouldn't be mapped and nova-api wouldn't list any instances
16:16:21 sileht mriedem, I think we having another behavior before, and the issue was not so often
16:17:01 zioproto dansmith: are you online ?
16:17:10 sileht mriedem, thanks for your help
16:18:24 mriedem sileht: yeah i mean it's still a bug, but also, if we didn't have cell1 created yet and you tried creating a server, it would fail because we have nowhere to place it
16:18:36 mriedem but you could list servers and get nothing back, and that would be fine if you never tried creating any servers
16:18:53 mriedem which i think that's what ceilometer was doing
16:18:55 mriedem just listing servers
16:18:58 mriedem on startup or something, not sure
16:19:17 mriedem but that GET call was caching cell0 whereas before https://review.openstack.org/#/c/427392/ we didn't have cell0 mapped so nothing was cached
16:23:08 bauzas jaypipes: you're hitting the pbr issue for os-traits lib
16:23:45 mriedem jaypipes: yeah you need to update the hacking version here https://github.com/jaypipes/os-traits/blob/master/test-requirements.txt#L5
16:23:58 mriedem http://git.openstack.org/cgit/openstack/requirements/tree/global-requirements.txt#n424
16:24:16 openstackgerrit Balazs Gibizer proposed openstack/nova master: handle uninited fields in notification payload https://review.openstack.org/415857
16:24:22 openstackgerrit Arnaud Morin proposed openstack/nova master: Set default value to HTTP_X_FORWARDED_PROTO https://review.openstack.org/423375
16:28:12 jaypipes mriedem: thx for the heads up, will fix up now.
16:29:15 mriedem dansmith: before i write tests, how much do you hate this? https://review.openstack.org/#/c/440652/
16:29:35 mriedem trying to think of ways to auto-invalidate or update that CELLS cache when a new cell mapping is created
16:30:09 dansmith mriedem: one cell is not valid regardless of why so I think you need only check the length
16:30:31 dansmith I still assert this is a silly workaround, but if you want it that's fine
16:31:31 mriedem i don't know that i want it
16:31:40 mriedem the devstack change would also unblock gnocchi
16:31:41 mriedem i think
16:35:44 dansmith mriedem: what was the change in the last day or so that caused this?
16:35:57 mriedem dansmith: https://review.openstack.org/#/c/427392/
16:36:16 dansmith ah
16:36:30 dansmith that caused me trouble too
16:38:20 jaypipes edleafe, alex_xu, bauzas, mriedem, dansmith: fixed up pep8/hacking thing. https://review.openstack.org/440678 now new root of series.
16:38:31 mriedem it does follow https://docs.openstack.org/developer/nova/cells.html#fresh-install more closely now, but simple_cell_setup and when it's called is kind of our issue now i think
16:38:54 mriedem dansmith: if we created cell1 around the same time as cell0, before starting nova-api, then we wouldn't have the cache issue
16:39:17 mriedem we'd just have to run discover_hosts after starting n-cpu
16:39:34 dansmith I still don't understand why we have the problem at all, since devstack is sequential
16:39:37 bauzas jaypipes: I thought you were hit by the fact we were pulling pbr-2.0.0
16:40:41 mriedem dansmith: devstack is mapping cell0, starting n-api, then loading extras (ceilometer plugin) which starts querying n-api, then devstack eventually does create_cell which calls simple_cell_setup
16:40:46 mriedem which creates the cell1 mapping
16:40:57 mriedem for the 'normal' setup there are no extras so we don't see it
16:41:52 dansmith mriedem: ah so we really are starting it in the wrong order we just don't hit the cache?
16:42:24 mriedem dansmith: yeah
16:42:34 dansmith mriedem: okay yeah then we should change that
16:42:36 mriedem ceilometer hit the cache and stored only cell0
16:42:44 dansmith mriedem: you're going to break my fleetify patch, but..
16:43:10 markus_z mriedem: I don't find a clue to solve the novnc issue. Tomorrow I have no time left, I'll give it another try on Monday.
16:43:22 mriedem markus_z: ok thanks for looking
16:43:35 mriedem dansmith: sileht has this https://review.openstack.org/#/c/440651/
16:43:39 mriedem but it is a workaround yeah
16:43:58 mriedem so the proper fix is probably what i described above, with creating cell1 at the same time as cell0
16:44:08 dansmith mriedem: yeah
16:44:10 mriedem and then change simple_cell_setup in the post-setup to just discover_hosts
16:44:17 mriedem or map_cell_and_hosts or whatever
16:44:51 dansmith not map_cell_and_hosts
16:44:52 mriedem i don't know if/how that messes up upgrade because we'd already have cell1
16:44:53 dansmith that creates a cell
16:44:56 dansmith just discover at the end
16:46:05 mriedem we need to not create cell1 if it already exists, which we could maybe figure out with the list_cells command
16:46:40 mriedem jaypipes: +2 on https://review.openstack.org/#/c/440678/
16:47:54 dansmith mriedem: that point where we create cell0 we're on an empty db regardless of upgrade because we're on the old side right?
16:49:13 mdbooth Is this going to be a problem in general when adding a cell? If you can add a cell and have instances scheduled to it instantly, but those instances won't be visible, isn't this something you're always going to hit when adding a cell?
16:49:14 jaypipes mriedem: cheers. the end goal here is to release os-traits lib with a git tag today so we can begin working on tests for that "sync os-traits" thing in placement-manage per bauzas and alex_xu's discussion on the traits spec.
16:50:01 mdbooth iow is it worth fixing this with something a bit more general purpose than a workaround for cell1?
16:50:32 mriedem dansmith: i think https://github.com/openstack-dev/devstack/blob/master/lib/nova#L700 is the dirty secret
16:50:40 mriedem we drop and recreate the nova_api db
16:50:51 mriedem dansmith: same for cell0
16:50:58 dansmith mriedem: eh?
16:51:03 mriedem i think from newton->ocata that wasn't a problem for cell0 because in newton we didn't put anything into cell0
16:51:15 mriedem but ocata cell0 could have stuff in it
16:51:22 dansmith init_nova only runs on the old side right?
16:51:31 mriedem idk
16:51:57 mriedem i thought it ran on both but i don't have grenade memorized
16:52:02 dansmith we blow away all the dbs there.. nothing would work if we ran it in both places
16:53:34 mriedem ok. i was thinking of this https://review.openstack.org/#/c/393441/2/projects/60_nova/from-newton/upgrade-nova@8
16:53:42 mriedem unrelated and confusing myself i suppose
16:54:15 mriedem anyway, i can work a change to do create_cell cell1 after map_cell0 and change simple_cell_setup to be discover_hosts in devstack
16:54:24 mriedem and if grenade hates it i guess we'll find out
16:55:46 openstackgerrit Lee Yarwood proposed openstack/nova master: WIP compute: Move detach logic from manager into driver BDM https://review.openstack.org/439520
16:55:47 openstackgerrit Lee Yarwood proposed openstack/nova master: compute: Only destroy BDMs after successful detach call https://review.openstack.org/440693
16:58:31 aunnam johnthetubaguy, Will you be updating the poc patch https://review.openstack.org/#/c/434842/ by removing scope and access as they are not needed right now?
16:59:12 johnthetubaguy aunnam: I was hoping you or sneti might fancy doing that :)
16:59:18 aunnam johnthetubaguy, Or you want us to update the patch?
16:59:35 aunnam johnthetubaguy, sure will do that, thanks
17:00:32 mriedem sileht: looks like my nova workaround patch fixed the gnocchi issue
17:02:20 mdbooth mriedem: re the CELLs cache, would it be feasible to manage that with memcached via oslo_cache? Could a newly created cell poke itself into the cache before allowing anything to be scheduled there?
17:02:55 mdbooth Should mean you'd have no weirdness period after creating a new cell.
17:03:13 mriedem mdbooth: at a minimum yes on oslo.cache to add a timer to it
17:03:19 mriedem mdbooth: feel free to start that
17:03:36 mriedem as for how to auto-register a new cell to invalidate the cache or put itself into it, i'm not sure how we do that right now

Earlier   Later