| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-01-17 | |||
| 17:11:51 | mriedem | can someone please reset the 'days since tripleo disaster' counter? | |
| 17:12:48 | owalsh | mriedem: :-) | |
| 17:12:52 | mriedem | so what are your steps again? | |
| 17:13:07 | mriedem | install newton, sync api db, sync nova db, create flavors,... | |
| 17:13:16 | sdague | mriedem: https://github.com/openstack-dev/devstack/blob/3c8ccecb1f0dff855e8043ba681d1749645de738/lib/nova#L468-L469 - probably that should be conditional behind n-api / n-cond / n-sched enabled | |
| 17:17:03 | owalsh | mriedem: install newton, create flavors (implicitly by installing TripleO undercloud), upgrade to ocata, create cell0 db, map cell0, create cell1, sync nova db, discover_hosts (none yet), sync api db ... deploy computes, discover_hosts | |
| 17:17:26 | owalsh | mriedem: I err think... | |
| 17:17:47 | openstackgerrit | Dan Peschman proposed openstack/nova: nova-manage cell_v2 map_cell0 exit 0 https://review.openstack.org/420132 | |
| 17:18:28 | mriedem | owalsh: so between "discover_hosts (none yet), sync api db" you blow up | |
| 17:18:34 | owalsh | mriedem: yes | |
| 17:19:28 | mriedem | fudg | |
| 17:19:32 | mriedem | *e | |
| 17:20:38 | mriedem | see how nova-status handles this exact same thing https://github.com/openstack/nova/blob/master/nova/cmd/status.py#L153-L171 | |
| 17:20:42 | mriedem | it doesn't consider it a failure | |
| 17:22:15 | owalsh | mriedem: yea, I'm thinking the db migraion could be relaxed. It will fail on the first check if the user hasn't run simple_cell_setup | |
| 17:23:24 | owalsh | mriedem: if it gets as far as the host mapping check it implies that map_cell0 and create_cell were used, or simple_cell_setup failed... | |
| 17:28:27 | mriedem | i need to reset my brain on how we can even add cell mappings before running the api db sync | |
| 17:29:53 | mriedem | in devstack we create the nova_api db and then sync it here https://github.com/openstack-dev/devstack/blob/3c8ccecb1f0dff855e8043ba681d1749645de738/lib/nova#L694 | |
| 17:30:11 | mriedem | then later we run simple_cell_setup here https://github.com/openstack-dev/devstack/blob/3c8ccecb1f0dff855e8043ba681d1749645de738/lib/nova#L950 | |
| 17:30:28 | openstackgerrit | Davanum Srinivas (dims) proposed openstack/nova: [WIP] ignore me - trying experimental jobs https://review.openstack.org/421209 | |
| 17:30:28 | mriedem | oh devstack works because when we sync the api_db we don't have any flavors yet | |
| 17:30:32 | mriedem | so it noops | |
| 17:31:38 | mriedem | ok let me wrap up this thing i'm fixing first, then i can come back to this | |
| 17:34:10 | gibi | mriedem: hi! I saw you patch chasing the race in the notification sample test https://review.openstack.org/#/c/421221 | |
| 17:34:25 | mriedem | yeah | |
| 17:34:27 | gibi | mriedem: do we want to merge that to see more logs or just use to locally reproduce the problem? | |
| 17:34:35 | mriedem | gibi: more logs | |
| 17:34:42 | mriedem | and so i can fingerprint the failure in elastic-recheck | |
| 17:35:04 | gibi | mriedem: OK, then I'm totally +1 on that | |
| 17:40:10 | openstackgerrit | Diana Clarke proposed openstack/nova: nova-manage docs: cell_v2 map_instances https://review.openstack.org/421404 | |
| 17:50:56 | melwitt | mriedem: sweet! glad I could help by not being around | |
| 17:51:25 | mriedem | melwitt: i think this is the culmination of 2 days of frustration https://review.openstack.org/#/c/421399/1 | |
| 17:51:37 | mriedem | and about 4 hours of sleep | |
| 17:52:00 | dims | mriedem : wow one liner! | |
| 17:52:07 | melwitt | oof | |
| 17:53:42 | mriedem | so in devstack runs we create flavors basically last | |
| 17:53:54 | mriedem | we create the dbs and sync them, | |
| 17:53:58 | mriedem | we run simple_cell_setup | |
| 17:54:01 | mriedem | then we create the flavors | |
| 17:54:34 | mriedem | and that's how devstack avoids failing the api db migration for there being no host mappings (well we'd have host mappings after running simple_cell_setup) | |
| 17:55:18 | owalsh | ... and the api db doesn't know about hosts etc... so the flavor is the trigger yea | |
| 17:55:22 | mriedem | in any other fresh install, you can't create flavors in the api db until you sync the api db | |
| 17:55:57 | owalsh | sequence above is an upgrade, so we have the tables from newton I expect | |
| 17:56:11 | mriedem | so, we have a particularly terrible scenario where you're actively going out of your way to create the flavors in the api db in newton, then upgrade to ocata and sync the api db before having any hosts | |
| 17:56:24 | mriedem | owalsh: wtf does tripleo do this? | |
| 17:56:40 | owalsh | mriedem: it doesn't, but it's a possible sequence of event | |
| 17:56:48 | owalsh | mriedem: user would need to do this | |
| 17:57:05 | mriedem | i thought the point of tripleo was to keep the user from hurting themselves? | |
| 17:57:20 | mriedem | s/user/operator/ | |
| 17:58:21 | mriedem | i mean i guess it's valid to create the newton control plane + flavors, and then decide, i might as well upgrade this to ocata even though i don't have any computes yet | |
| 17:58:25 | mriedem | it just seems really weird | |
| 17:59:20 | owalsh | it wouldn't be typical, but if an operator did this they wouldn't be able to recover easily I expect | |
| 18:01:09 | mriedem | well, they'd have to delete the flavors | |
| 18:01:43 | mriedem | there would be 2 outs: | |
| 18:02:06 | mriedem | 1. delete the flavors, sync the api db, do the cell setup stuff, then add the flavors | |
| 18:02:32 | mriedem | 2. create the computes, run discover_hosts or simple_cell_setup, then sync the api db | |
| 18:04:35 | owalsh | should clarify, I'm talking about the all-in-one cloud that deploys the real cloud | |
| 18:06:13 | owalsh | operator could install it, maybe create custom flavors etc... upgrade to ocata, then add hosts to deploy to | |
| 18:06:26 | owalsh | far less likely to have a production cloud with no hosts | |
| 18:06:47 | owalsh | but with flavors | |
| 18:16:25 | openstackgerrit | Diana Clarke proposed openstack/nova: nova-manage docs: cell_v2 map_cell_and_hosts https://review.openstack.org/421420 | |
| 18:17:52 | openstackgerrit | Matt Riedemann proposed openstack/nova: Fix and add some notes to the cells v2 first time setup doc https://review.openstack.org/421421 | |
| 18:17:57 | mriedem | melwitt: found a typo in the cells v2 setup doc ^ | |
| 18:20:14 | owalsh | mriedem: so for fresh installs it would be: install ocata, create nova/api/cell0 dbs, sync api db, map_cell0, create_cell cell1, sync nova db ... deploy computes, discover_hosts | |
| 18:23:51 | mriedem | owalsh: well, create_cell cell1 implies you have a cell1 db | |
| 18:25:30 | mriedem | you don't have to create a cell1, | |
| 18:25:43 | mriedem | simple_cell_setup and map_cell_and_hosts will default to use the nova db as cell1 | |
| 18:26:00 | owalsh | can't use simple_cell_setup as we have no hosts yet | |
| 18:26:18 | owalsh | create_cell cell1 defaults to using the nova db too AFAICT | |
| 18:26:43 | mriedem | yeah which fails w/o https://review.openstack.org/#/c/420079/ | |
| 18:27:03 | mriedem | owalsh: yeah you're right, | |
| 18:27:18 | mriedem | create_cell will use CONF.database.connection which is by default the main (non-api) db | |
| 18:28:45 | mriedem | owalsh: so yeah i think that's all correct for first time initial ocata setup | |
| 18:28:48 | mriedem | also in http://docs.openstack.org/developer/nova/cells.html#setup-of-cells-v2 | |
| 18:29:12 | mriedem | i think we plan on eventually getting rid of simple_cell_setup as it's not that simple for fresh install | |
| 18:31:15 | comstud | mriedem: what's this cells thing you speak of? | |
| 18:31:24 | comstud | never heard of it | |
| 18:31:36 | comstud | i only know zones | |
| 18:36:55 | mriedem | comstud: cells v2.0 | |
| 18:37:00 | mriedem | cells: the reckoning | |
| 18:37:44 | comstud | mriedem: Cells 2: Electric Boogaloo | |
| 18:38:30 | comstud | i've been meaning to look @ what laski et al came up with, but no time | |
| 18:39:06 | openstackgerrit | Diana Clarke proposed openstack/nova: nova-manage docs: cell_v2 verify_instance https://review.openstack.org/421436 | |
| 18:39:43 | mriedem | cells IV: the revenge of behrens | |
| 18:39:49 | comstud | lol | |
| 18:40:10 | mriedem | that's for when you come back | |
| 18:41:40 | openstackgerrit | Diana Clarke proposed openstack/nova: nova-manage docs: cell_v2 verify_instance https://review.openstack.org/421436 | |
| 18:42:09 | owalsh | mriedem: I think the difficulty with all of this is the sync api db | |
| 18:42:41 | owalsh | ideall upgrades would be similar: install ocata, create nova/api/cell0 dbs, sync api db, map_cell0, create_cell cell1, sync nova db, discover_hosts | |
| 18:42:55 | owalsh | s/ideall/ideally/ | |
| 18:46:04 | openstackgerrit | Diana Clarke proposed openstack/nova: nova-manage docs: cell_v2 discover_hosts https://review.openstack.org/421442 | |
| 18:46:26 | mriedem | sdague: melwitt: got past that db host issue on the subnode \o/ | |
| 18:47:01 | mriedem | owalsh: is there an upgrade step in there? | |
| 18:47:19 | mriedem | wait, upgrade step in where? the doc? | |
| 18:48:17 | owalsh | mriedem: install ocata, from newton | |
| 18:48:54 | mriedem | ok. | |
| 18:48:55 | melwitt | mriedem: sweet, nice work | |
| 18:49:02 | mriedem | when coming from newton you'd have the nova and nova_api dbs | |
| 18:49:08 | mriedem | you still have to create cell0 | |
| 18:49:14 | owalsh | ah, yes, typo | |