Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-01-13
15:21:50 mwhahaha we would want to touch both because we want to run sync after compute service starts
15:22:08 mwhahaha i still don't understand why these things keep getting added as hard requirements after the fact
15:22:29 mwhahaha first it was the db upgrade that broke everything and now it's you have to run it after computes, is any of this stuff documented?
15:22:48 bauzas EmilienM: amoralej: so, about a very short-term fix, here are my thoughts
15:22:48 EmilienM I agree it's not easy for deployers :(
15:22:57 EmilienM mriedem: ^ fyi
15:23:00 amoralej not really mwhahaha, i've spent the whole morning trying to understand it
15:23:26 mwhahaha i've probably wasted over a month on this stuff and operators are expected to use it?
15:23:30 bauzas EmilienM: amoralej: while I understand the point that it could be fancy for operators to have a way to automatically add the cell mapping when starting the compute, that's not written yet
15:24:07 mwhahaha why is cell v2 being a hard requirement if docs and the code aren't written yet?
15:24:14 bauzas EmilienM: amoralej: here, instead, in order to punt your problem, I'd rather recommend you to modify https://github.com/openstack/puppet-nova/blob/master/manifests/db/sync_cell_v2.pp#L32 and accept return codes to be 0 and 1
15:24:16 amoralej bauzas, and requiring to run discover_hosts after every new compute nodes should be automated too, imho
15:24:39 EmilienM bauzas: but would tables be created in DB?
15:24:51 mwhahaha bauzas: that still doesn't handle the upgrade case where we need to have it created before we run the api sync for n ->o
15:24:59 mwhahaha bauzas: that's not acceptable
15:25:09 bauzas EmilienM: amoralej: it would put your CI back green, but would still require the cell command to be run *after* starting the compute
15:25:41 bauzas mwhahaha: upgrade is a very different case
15:25:48 bauzas mwhahaha: because you already have computes
15:26:09 mwhahaha bauzas: please point me to the documentation for upgrades vs install of cell v2
15:26:46 bauzas mwhahaha: here, the discussion is about the critical bug that tramples both puppet users and the TripleO CI
15:27:00 bauzas I'm just trying to tackle the problem asp
15:27:02 bauzas asap
15:27:15 mwhahaha this problem has existed in one form or another for more than a month
15:27:34 mwhahaha and every time we make progress, nova checks a change that completely destroys what we thought the process was previously
15:27:46 mwhahaha so i'd like not to have to repeat this again in a week
15:28:09 mwhahaha so we all understand how this thing is supposed to work, is there any documentation around how this is expected to function
15:28:43 bauzas mwhahaha: http://docs.openstack.org/releasenotes/nova/unreleased.html#id2 can help you
15:28:47 EmilienM I agree this thing is complex to deploy outside devstack
15:28:52 EmilienM specially in production
15:28:53 bauzas "Ocata requires that your deployment have created the cell and host mappings in Newton. If you have not done this, Ocata’s db sync command will fail. Small deployments will want to run nova-manage cell_v2 simple_cell_setup on Newton before upgrading. Operators must create a new database for cell0 before running cell_v2 simple_cell_setup. The simple cell setup command expects the name of the cell0 database to be <API database
15:29:02 mwhahaha bauzas: that's what we're doing
15:29:10 mwhahaha bauzas: but no mention of discover hosts or anything else
15:29:17 mwhahaha also release notes != documentation on the overall process
15:29:49 EmilienM it doesn't mention ordering with nova-compute process, does it?
15:33:20 bauzas EmilienM: well, that's because we added the simple cell command in Newton :(
15:33:27 bauzas EmilienM: to answer your question, http://docs.openstack.org/releasenotes/nova/unreleased.html#id10
15:35:01 openstackgerrit Davanum Srinivas (dims) proposed openstack/nova: [py35] Fixes to get rally scenarios working https://review.openstack.org/419476
15:35:26 mriedem so docs is a known gap, i.e. docs that aren't release notes
15:35:34 mriedem i think dansmith was working on a blog post to start that,
15:35:41 mriedem but it would eventually be in the devref
15:35:57 mriedem laski had started some docs at one point but it's probably diverged quite a bit from how things actualy work now
15:36:17 mriedem having said all that, dan is out on vacation for a week
15:37:18 mriedem i don't know how far along dan was on a blog post, but i might start a doc if i can get some time
15:37:38 melwitt yeah, there isn't yet detail about how compute nodes are needed before running simple_cell_setup
15:37:58 mriedem melwitt: you mean otherwise you have to run discover_hosts?
15:38:17 mriedem i haven't read all of the scollback,
15:38:25 melwitt mriedem: well, discover_hosts actually won't work either because that requires you have at least one cell_mapping already
15:38:37 melwitt so really it would be you have to run map_cell_and_hosts I think
15:38:48 melwitt after compute hosts show up later
15:38:49 mriedem but if there are major things that people have run into in practice trying to get this setup, it would be helpful to post those in the mailing list for posterity rather than a dump in IRC which hits maybe 1% of people right now on a friday
15:39:01 mriedem EmilienM: mwhahaha: amoralej: ^
15:39:13 mriedem because those are the things i want to make sure are addressed in a doc
15:39:14 melwitt what I'm wondering is whether the cells v2 upgrade check runs on a fresh install or not?
15:39:22 mwhahaha it doesn't
15:39:32 mwhahaha the upgrade problem is something else
15:39:42 mwhahaha the problem is that you're adding requirements that no one know sabout
15:39:50 mriedem melwitt: which cells v2 upgrade check? the db schema migration?
15:39:51 openstack Launchpad bug 1656276 in tripleo "Error running nova-manage cell_v2 simple_cell_setup when configuring nova with puppet-nova" [Critical,Triaged]
15:39:51 bauzas melwitt: https://bugs.launchpad.net/nova/+bug/1656276 refers to a fresh install
15:40:06 mwhahaha oh yea,that's new as of today
15:40:08 melwitt mriedem: the thing that raises an error if there aren't two cell mappings
15:40:16 bauzas melwitt: and the proposed fix to amend https://github.com/openstack/puppet-nova/blob/master/manifests/db/sync_cell_v2.pp#L32 with return code 1 as valid would fix $
15:40:22 bauzas would fix ^ (I mean)
15:40:40 bauzas what mwhahaha is discussing is another issue, related to the upgrade path
15:40:40 mriedem melwitt: mwhahaha: so the db schema migration runs on a fresh install by first checking to see if there are any flavors, if there aren't then it noops
15:40:43 mwhahaha https://bugs.launchpad.net/tripleo/+bug/1649341 was teh upgrade bug
15:40:44 openstack Launchpad bug 1649341 in tripleo "Undercloud upgrade fails with "Cell mappings are not created, but required for Ocata"" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz)
15:40:57 bauzas so another bug
15:40:58 melwitt mriedem: the (_('Cells v2'), _check_cellsv2), in nova/cmd/status.py
15:41:05 mriedem melwitt: so that's different
15:41:07 melwitt I see
15:41:18 mriedem and we haven't integrated the nova-status upgrade check stuff into our CI pipeline yet
15:41:21 mriedem another TODO of mine
15:41:41 openstack Launchpad bug 1656276 in tripleo "Error running nova-manage cell_v2 simple_cell_setup when configuring nova with puppet-nova" [Critical,Triaged]
15:41:41 mwhahaha which the problem is to solve https://bugs.launchpad.net/tripleo/+bug/1649341 i need to know how to properly tackle https://bugs.launchpad.net/nova/+bug/1656276
15:41:49 mwhahaha where i didn't until today
15:41:59 mwhahaha so now we're just adding complexity on complexity
15:42:31 mwhahaha and adding accept 1 doesn't solve the upgrade case
15:42:45 melwitt yeah, I don't think accepting 1 helps anything really
15:42:52 mwhahaha i had a solution for the upgrade until the requirement of compute nodes was added
15:43:01 mriedem mwhahaha: EmilienM: so can we hold on a minute,
15:43:08 mriedem i understand you're frustrated,
15:43:31 bauzas melwitt: it only helps their CI to be green
15:43:51 mriedem i need help with posting the list of issues you have in a form that doesn't require me to be chasing 3 conversations at once
15:43:54 melwitt I'm not sure it'll run correctly after that if there are no cell mappings created though (nova)
15:43:56 EmilienM it's not about a green CI
15:44:05 openstackgerrit Davanum Srinivas (dims) proposed openstack/nova: [py35] Fixes to get rally scenarios working https://review.openstack.org/419476
15:44:05 EmilienM it's about how people outside devstack deploy your software
15:44:17 mriedem mwhahaha: EmilienM: so can you guys help me out by starting a ML thread with the current known issues you have so we can work through it there?
15:44:23 EmilienM mriedem: of course
15:44:58 mriedem b/c right now i'm not keeping up with this fire hose of stuff
15:45:03 mriedem and not everyone is around right now
15:45:18 mriedem so we need to start a list of items in the ML and tackle those
15:45:31 mwhahaha yea we can move to the ML
15:45:46 bauzas EmilienM: I don't disagree, I'm just trying to reduce the criticity so we can sit down and discuss without having the tense of urgency
15:46:26 mriedem we could start an etherpad too if that is helpful
15:46:41 mriedem hell i'm pretty sure we already have an etherpad for cells v2
15:46:54 bauzas EmilienM: I'm really sorry that you folks are experiencing bugs against our recent improvements and I think your insights are definitely valid
15:47:54 mriedem i can't find the etherpad, only the wiki page
15:48:04 mriedem melwitt: do you know if we had a generic cells v2 etherpad for work items?

Earlier   Later