| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-01-31 | |||
| 19:33:00 | artom | dansmith, nope | |
| 19:33:03 | dansmith | ah, okay | |
| 19:33:07 | dansmith | then yeah | |
| 19:33:30 | jaypipes | mriedem, cdent: yes, I'm still working on functional tests for the Ironic patch. Didn't want you to think I forgot or am slacking. | |
| 19:34:08 | mriedem | i'm pretty far removed from that patch by now, | |
| 19:34:12 | mriedem | once john manhandled it | |
| 19:34:44 | mriedem | like, before i start reviewing it again, someone is probably going to have to give me a recap of what happened | |
| 19:35:57 | mriedem | browne: probably... | |
| 19:36:26 | mriedem | i feel like russellb used to have a cheat sheet for this | |
| 19:42:08 | openstackgerrit | Matt Riedemann proposed openstack/nova: doc: add upgrade notes to the placement devref https://review.openstack.org/427367 | |
| 19:45:37 | mordred | mriedem: I've got this error coming up on nodepool patches: http://logs.openstack.org/06/425706/7/check/gate-dsvm-nodepool-src/7004c5b/logs/screen-nodepool.txt.gz#_2017-01-31_19_02_00_177 | |
| 19:45:57 | mordred | mriedem: which is reporting a 500 error from nova and <class 'sqlalchemy.exc.ProgrammingError'> as the error | |
| 19:46:33 | mriedem | nice http://logs.openstack.org/06/425706/7/check/gate-dsvm-nodepool-src/7004c5b/logs/screen-n-api.txt.gz?level=TRACE#_2017-01-31_19_02_00_169 | |
| 19:46:41 | mordred | yah | |
| 19:46:51 | mordred | isn't that pretty? (I was just about to paste that link next :) ) | |
| 19:46:53 | mriedem | Table 'nova_cell0.instance_extra' doesn't exist | |
| 19:47:00 | mriedem | mordred: you don't have your cells v2 stuff setup | |
| 19:47:02 | mriedem | in whatever that thing is | |
| 19:47:18 | mordred | well, it's nodepool's functional tests that run on top of a devstack node | |
| 19:47:36 | mordred | does devstack-gate need cellsv2 now but not set it up by default? :( | |
| 19:47:47 | mriedem | devstack sets up cells v2 | |
| 19:48:05 | mriedem | https://github.com/openstack-dev/devstack/blob/master/lib/nova#L684 | |
| 19:48:09 | mordred | neat. well, apparently something I'm doing there is not letting it I guess - Ill look fruther | |
| 19:48:10 | mriedem | ^ creates the cell0 db | |
| 19:48:28 | mordred | wow. so it should just do it without any flags it looks like | |
| 19:48:43 | mriedem | mordred: and this stuff at the end https://github.com/openstack-dev/devstack/blob/master/lib/nova#L947 | |
| 19:48:53 | mriedem | mordred: yes, cells v2 is required now in ocata | |
| 19:49:12 | mriedem | all ocata nova deployments will be at least a cell of one | |
| 19:49:24 | mordred | mriedem: is there a cell service I need to enable? | |
| 19:49:41 | mriedem | no | |
| 19:49:49 | mriedem | that's cells v1 | |
| 19:49:55 | mriedem | cells v2 is baked in | |
| 19:50:05 | mriedem | here is the nova_cell0 db create in your run http://logs.openstack.org/06/425706/7/check/gate-dsvm-nodepool-src/7004c5b/logs/devstacklog.txt.gz#_2017-01-31_18_58_29_045 | |
| 19:50:20 | mriedem | map_cell0 http://logs.openstack.org/06/425706/7/check/gate-dsvm-nodepool-src/7004c5b/logs/devstacklog.txt.gz#_2017-01-31_19_01_45_588 | |
| 19:50:31 | mriedem | and simple_cell_setup http://logs.openstack.org/06/425706/7/check/gate-dsvm-nodepool-src/7004c5b/logs/devstacklog.txt.gz#_2017-01-31_19_01_48_335 | |
| 19:50:48 | mordred | WEIRD | |
| 19:51:03 | mriedem | that's an appropriate reaction | |
| 19:51:25 | mriedem | http://docs.openstack.org/developer/nova/cells.html | |
| 19:52:37 | mriedem | so when your thing blew up in n-api, cell0 would be mapped | |
| 19:52:42 | mriedem | so i'm not sure why it's not finding the table | |
| 19:52:43 | mriedem | *db | |
| 19:52:53 | mriedem | or, maybe it found the db but the schema wasn't migrated? | |
| 19:53:05 | mriedem | simple_cell_setup will migrate the db schema on the cell0 db | |
| 19:53:16 | mordred | does it do the migration asynchronously? | |
| 19:53:18 | mriedem | you can see it doing that here http://logs.openstack.org/06/425706/7/check/gate-dsvm-nodepool-src/7004c5b/logs/devstacklog.txt.gz#_2017-01-31_19_02_00_128 | |
| 19:53:20 | mriedem | no | |
| 19:53:29 | diana_clarke | perhaps this? "WARNING: cell0 mapping not found - not syncing cell0." | |
| 19:53:31 | mriedem | Warning: (1831, u"Duplicate index 'block_device_mapping_instance_uuid_virtual_name_device_name_idx' defined on the table 'nova_cell0.block_device_mapping' | |
| 19:53:34 | diana_clarke | http://logs.openstack.org/06/425706/7/check/gate-dsvm-nodepool-src/7004c5b/logs/devstacklog.txt.gz#_2017-01-31_18_58_31_449 | |
| 19:54:06 | mriedem | diana_clarke: that's expected | |
| 19:54:17 | mriedem | we've created the dbs, but haven't mapped cell0 yet, that comeslater | |
| 19:54:19 | mriedem | *comes later | |
| 19:54:46 | mriedem | https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L638 | |
| 19:54:52 | mriedem | db sync attempts to fan out by default | |
| 19:55:03 | mriedem | i'm not sure that should be a WARNING?!?!??! | |
| 19:55:25 | mriedem | https://review.openstack.org/#/c/420973/ fixes the fanout thing | |
| 19:56:31 | mriedem | we see this https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L1100 | |
| 19:56:45 | mriedem | and then it migrates the schema for the cell0 db here https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L1107 | |
| 19:57:53 | diana_clarke | mriedem: but only in the non-cells v1 case, it looks to me like cells v1 is enabled here. (ignore me though, I'm probably wrong) | |
| 19:58:16 | openstackgerrit | Eric Fried proposed openstack/nova: PowerVM Driver: power_on/off and reboot https://review.openstack.org/427380 | |
| 19:58:51 | diana_clarke | (b/c is_service_enabled n-cell returns 1, which I assume means true) | |
| 19:58:52 | mriedem | diana_clarke: in the cells v1 case, cell0 db schema would be migrated here https://github.com/openstack-dev/devstack/blob/master/lib/nova#L956 | |
| 19:59:18 | mordred | mriedem: I just pushed this up: remote: https://review.openstack.org/427381 Update nodepool dsvm test to disable services | |
| 19:59:18 | mriedem | but no, cells v1 isn't enabled here | |
| 19:59:30 | mordred | mriedem: just in case something is wonk with us whitelisting a set of services | |
| 19:59:42 | mordred | mriedem: I don't see any other real differences between job configs | |
| 20:00:07 | mriedem | mordred: there is no extra service for cells so it wouldn't be a problem - but you probably need ^ for placement-api anyway | |
| 20:00:15 | mriedem | since nova requires placement-api in ocata to start computes | |
| 20:00:30 | mordred | shouldn't that default to on in devstack then? | |
| 20:00:31 | mriedem | or it will once this merges https://review.openstack.org/#/c/425806/ | |
| 20:00:34 | mriedem | mordred: it does | |
| 20:00:37 | mordred | ok. cool | |
| 20:00:43 | mordred | but I need to not whitelist things | |
| 20:00:47 | mordred | so that I get the placement-api things | |
| 20:00:59 | mriedem | https://github.com/openstack-dev/devstack/blob/master/stackrc#L58 | |
| 20:01:00 | mriedem | indeed | |
| 20:01:14 | mriedem | which makes me think, i should probably sound the alarm in the dev list quick | |
| 20:01:29 | mriedem | before the hordes show up saying their big tent CIs are now all failing because nova-compute won't start | |
| 20:02:06 | openstackgerrit | Eric Fried proposed openstack/nova: PowerVM Driver: power_on/off and reboot https://review.openstack.org/427380 | |
| 20:02:36 | mordred | ++ | |
| 20:03:40 | mriedem | done | |
| 20:04:45 | mriedem | so when this blows up, we're getting here https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2440 | |
| 20:04:51 | mriedem | which means map_cell0 worked, we have the cell0 mapping | |
| 20:05:57 | mriedem | and from what i could tell in the devstack logs, the db schema was migrated for nova_cell0 | |
| 20:07:00 | mriedem | should be right here http://logs.openstack.org/06/425706/7/check/gate-dsvm-nodepool-src/7004c5b/logs/devstacklog.txt.gz#_2017-01-31_19_01_50_743 | |
| 20:07:08 | mriedem | 2017-01-31 19:01:50.743 | 2017-01-31 19:01:50.743 4887 DEBUG migrate.versioning.script.base [req-067d40dc-316e-42b5-82e8-feaec4129b2b - -] Loading script /opt/stack/new/nova/nova/db/sqlalchemy/migrate_repo/versions/252_add_instance_extra_table.py... __init__ /usr/local/lib/python2.7/dist-packages/migrate/versioning/script/base.py:27 | |
| 20:08:30 | mriedem | actually more like http://logs.openstack.org/06/425706/7/check/gate-dsvm-nodepool-src/7004c5b/logs/devstacklog.txt.gz#_2017-01-31_19_02_22_726 | |
| 20:08:52 | mriedem | mordred: which is 22 seconds after your failure | |
| 20:09:02 | mriedem | fails here 2017-01-31_19_02_00_177 | |
| 20:09:03 | mordred | mriedem: oh. I mean ... | |
| 20:09:14 | mriedem | nova_cell0 db schema is being migrated here 2017-01-31_19_02_22_726 | |
| 20:09:40 | mordred | mriedem: it seems that devstack is still setting itself up when nodepool is running :( | |
| 20:10:55 | mordred | jeblair: ^^ you may (or may not) find this scrollback interesting | |
| 20:11:13 | mordred | jeblair: tl;dr - it's possible openstack is not yet ready when nodepool tries to start talking to it | |
| 20:13:18 | mordred | sdague, ianw: ^^ you two may or may not also find it interesting | |
| 20:13:32 | jeblair | mordred: that's weirdly parallel with our zookeeper conversation... but anyway, wouldn't any devstack test (eg tempest) have this issue? | |
| 20:13:33 | mriedem | so... | |
| 20:13:47 | mriedem | jeblair: d-g runs tempest after stack.sh is done | |
| 20:13:48 | mriedem | so no | |
| 20:14:14 | mriedem | i think we could move that map_cell0 call way further up in the stack run though, | |