Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-01-13
15:14:13 mandy bye
15:16:35 EmilienM mwhahaha: out of your mind, would it be doable to run this cell_v2 simple_cell_setup *after* starting nova-compute in tripleo?
15:16:44 EmilienM mwhahaha: I'm wondering about the steps
15:17:02 bauzas EmilienM: http://logs.openstack.org/90/409890/1/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/447ab55/logs/devstacklog.txt.gz#_2016-12-12_21_54_28_042
15:17:13 mriedem mdbooth: how are you feeling about https://review.openstack.org/#/c/312488/ ?
15:17:16 bauzas (sorry, in internal meeting now)
15:17:26 mriedem the vz ci is running test_verify_created_server_ephemeral_disk now
15:17:36 openstackgerrit Vasyl Saienko proposed openstack/nova: Implement interface attach/detach in ironic virt driver https://review.openstack.org/419975
15:17:55 openstackgerrit Vasyl Saienko proposed openstack/nova: Implement interface attach/detach in ironic virt driver https://review.openstack.org/419975
15:18:02 mwhahaha why is this cell v2 setup stuff so complex
15:18:13 mdbooth mriedem: In a meeting. Get back in an hour or so.
15:18:37 mwhahaha it's so not 'simple' if you require cross node coordination prior to running it
15:19:00 amoralej EmilienM, i can start by removing the command as part of nova::db::sync and adding it at the end in p-o-i
15:19:12 amoralej that would be the first step, am i right?
15:19:26 amoralej mwhahaha ^
15:19:37 mwhahaha no
15:19:44 mwhahaha it's lined up in the nova::deps
15:19:52 EmilienM amoralej: change needs to happen in puppet-nova, not poi (if we have to change something)
15:19:56 EmilienM right
15:20:07 mwhahaha https://github.com/openstack/puppet-nova/blob/master/manifests/deps.pp#L23-L26
15:20:55 mwhahaha also we would need to remove https://github.com/openstack/puppet-nova/blob/master/manifests/db/sync_api.pp#L45-L47
15:21:24 amoralej this one is the one i have seen, not the deps stuff
15:21:50 mwhahaha we would want to touch both because we want to run sync after compute service starts
15:22:08 mwhahaha i still don't understand why these things keep getting added as hard requirements after the fact
15:22:29 mwhahaha first it was the db upgrade that broke everything and now it's you have to run it after computes, is any of this stuff documented?
15:22:48 bauzas EmilienM: amoralej: so, about a very short-term fix, here are my thoughts
15:22:48 EmilienM I agree it's not easy for deployers :(
15:22:57 EmilienM mriedem: ^ fyi
15:23:00 amoralej not really mwhahaha, i've spent the whole morning trying to understand it
15:23:26 mwhahaha i've probably wasted over a month on this stuff and operators are expected to use it?
15:23:30 bauzas EmilienM: amoralej: while I understand the point that it could be fancy for operators to have a way to automatically add the cell mapping when starting the compute, that's not written yet
15:24:07 mwhahaha why is cell v2 being a hard requirement if docs and the code aren't written yet?
15:24:14 bauzas EmilienM: amoralej: here, instead, in order to punt your problem, I'd rather recommend you to modify https://github.com/openstack/puppet-nova/blob/master/manifests/db/sync_cell_v2.pp#L32 and accept return codes to be 0 and 1
15:24:16 amoralej bauzas, and requiring to run discover_hosts after every new compute nodes should be automated too, imho
15:24:39 EmilienM bauzas: but would tables be created in DB?
15:24:51 mwhahaha bauzas: that still doesn't handle the upgrade case where we need to have it created before we run the api sync for n ->o
15:24:59 mwhahaha bauzas: that's not acceptable
15:25:09 bauzas EmilienM: amoralej: it would put your CI back green, but would still require the cell command to be run *after* starting the compute
15:25:41 bauzas mwhahaha: upgrade is a very different case
15:25:48 bauzas mwhahaha: because you already have computes
15:26:09 mwhahaha bauzas: please point me to the documentation for upgrades vs install of cell v2
15:26:46 bauzas mwhahaha: here, the discussion is about the critical bug that tramples both puppet users and the TripleO CI
15:27:00 bauzas I'm just trying to tackle the problem asp
15:27:02 bauzas asap
15:27:15 mwhahaha this problem has existed in one form or another for more than a month
15:27:34 mwhahaha and every time we make progress, nova checks a change that completely destroys what we thought the process was previously
15:27:46 mwhahaha so i'd like not to have to repeat this again in a week
15:28:09 mwhahaha so we all understand how this thing is supposed to work, is there any documentation around how this is expected to function
15:28:43 bauzas mwhahaha: http://docs.openstack.org/releasenotes/nova/unreleased.html#id2 can help you
15:28:47 EmilienM I agree this thing is complex to deploy outside devstack
15:28:52 EmilienM specially in production
15:28:53 bauzas "Ocata requires that your deployment have created the cell and host mappings in Newton. If you have not done this, Ocata’s db sync command will fail. Small deployments will want to run nova-manage cell_v2 simple_cell_setup on Newton before upgrading. Operators must create a new database for cell0 before running cell_v2 simple_cell_setup. The simple cell setup command expects the name of the cell0 database to be <API database
15:29:02 mwhahaha bauzas: that's what we're doing
15:29:10 mwhahaha bauzas: but no mention of discover hosts or anything else
15:29:17 mwhahaha also release notes != documentation on the overall process
15:29:49 EmilienM it doesn't mention ordering with nova-compute process, does it?
15:33:20 bauzas EmilienM: well, that's because we added the simple cell command in Newton :(
15:33:27 bauzas EmilienM: to answer your question, http://docs.openstack.org/releasenotes/nova/unreleased.html#id10
15:35:01 openstackgerrit Davanum Srinivas (dims) proposed openstack/nova: [py35] Fixes to get rally scenarios working https://review.openstack.org/419476
15:35:26 mriedem so docs is a known gap, i.e. docs that aren't release notes
15:35:34 mriedem i think dansmith was working on a blog post to start that,
15:35:41 mriedem but it would eventually be in the devref
15:35:57 mriedem laski had started some docs at one point but it's probably diverged quite a bit from how things actualy work now
15:36:17 mriedem having said all that, dan is out on vacation for a week
15:37:18 mriedem i don't know how far along dan was on a blog post, but i might start a doc if i can get some time
15:37:38 melwitt yeah, there isn't yet detail about how compute nodes are needed before running simple_cell_setup
15:37:58 mriedem melwitt: you mean otherwise you have to run discover_hosts?
15:38:17 mriedem i haven't read all of the scollback,
15:38:25 melwitt mriedem: well, discover_hosts actually won't work either because that requires you have at least one cell_mapping already
15:38:37 melwitt so really it would be you have to run map_cell_and_hosts I think
15:38:48 melwitt after compute hosts show up later
15:38:49 mriedem but if there are major things that people have run into in practice trying to get this setup, it would be helpful to post those in the mailing list for posterity rather than a dump in IRC which hits maybe 1% of people right now on a friday
15:39:01 mriedem EmilienM: mwhahaha: amoralej: ^
15:39:13 mriedem because those are the things i want to make sure are addressed in a doc
15:39:14 melwitt what I'm wondering is whether the cells v2 upgrade check runs on a fresh install or not?
15:39:22 mwhahaha it doesn't
15:39:32 mwhahaha the upgrade problem is something else
15:39:42 mwhahaha the problem is that you're adding requirements that no one know sabout
15:39:50 mriedem melwitt: which cells v2 upgrade check? the db schema migration?
15:39:51 openstack Launchpad bug 1656276 in tripleo "Error running nova-manage cell_v2 simple_cell_setup when configuring nova with puppet-nova" [Critical,Triaged]
15:39:51 bauzas melwitt: https://bugs.launchpad.net/nova/+bug/1656276 refers to a fresh install
15:40:06 mwhahaha oh yea,that's new as of today
15:40:08 melwitt mriedem: the thing that raises an error if there aren't two cell mappings
15:40:16 bauzas melwitt: and the proposed fix to amend https://github.com/openstack/puppet-nova/blob/master/manifests/db/sync_cell_v2.pp#L32 with return code 1 as valid would fix $
15:40:22 bauzas would fix ^ (I mean)
15:40:40 bauzas what mwhahaha is discussing is another issue, related to the upgrade path
15:40:40 mriedem melwitt: mwhahaha: so the db schema migration runs on a fresh install by first checking to see if there are any flavors, if there aren't then it noops
15:40:43 mwhahaha https://bugs.launchpad.net/tripleo/+bug/1649341 was teh upgrade bug
15:40:44 openstack Launchpad bug 1649341 in tripleo "Undercloud upgrade fails with "Cell mappings are not created, but required for Ocata"" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz)
15:40:57 bauzas so another bug
15:40:58 melwitt mriedem: the (_('Cells v2'), _check_cellsv2), in nova/cmd/status.py
15:41:05 mriedem melwitt: so that's different
15:41:07 melwitt I see
15:41:18 mriedem and we haven't integrated the nova-status upgrade check stuff into our CI pipeline yet
15:41:21 mriedem another TODO of mine
15:41:41 openstack Launchpad bug 1656276 in tripleo "Error running nova-manage cell_v2 simple_cell_setup when configuring nova with puppet-nova" [Critical,Triaged]
15:41:41 mwhahaha which the problem is to solve https://bugs.launchpad.net/tripleo/+bug/1649341 i need to know how to properly tackle https://bugs.launchpad.net/nova/+bug/1656276
15:41:49 mwhahaha where i didn't until today
15:41:59 mwhahaha so now we're just adding complexity on complexity
15:42:31 mwhahaha and adding accept 1 doesn't solve the upgrade case

Earlier   Later