| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-01-31 | |||
| 20:01:00 | mriedem | indeed | |
| 20:01:14 | mriedem | which makes me think, i should probably sound the alarm in the dev list quick | |
| 20:01:29 | mriedem | before the hordes show up saying their big tent CIs are now all failing because nova-compute won't start | |
| 20:02:06 | openstackgerrit | Eric Fried proposed openstack/nova: PowerVM Driver: power_on/off and reboot https://review.openstack.org/427380 | |
| 20:02:36 | mordred | ++ | |
| 20:03:40 | mriedem | done | |
| 20:04:45 | mriedem | so when this blows up, we're getting here https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2440 | |
| 20:04:51 | mriedem | which means map_cell0 worked, we have the cell0 mapping | |
| 20:05:57 | mriedem | and from what i could tell in the devstack logs, the db schema was migrated for nova_cell0 | |
| 20:07:00 | mriedem | should be right here http://logs.openstack.org/06/425706/7/check/gate-dsvm-nodepool-src/7004c5b/logs/devstacklog.txt.gz#_2017-01-31_19_01_50_743 | |
| 20:07:08 | mriedem | 2017-01-31 19:01:50.743 | 2017-01-31 19:01:50.743 4887 DEBUG migrate.versioning.script.base [req-067d40dc-316e-42b5-82e8-feaec4129b2b - -] Loading script /opt/stack/new/nova/nova/db/sqlalchemy/migrate_repo/versions/252_add_instance_extra_table.py... __init__ /usr/local/lib/python2.7/dist-packages/migrate/versioning/script/base.py:27 | |
| 20:08:30 | mriedem | actually more like http://logs.openstack.org/06/425706/7/check/gate-dsvm-nodepool-src/7004c5b/logs/devstacklog.txt.gz#_2017-01-31_19_02_22_726 | |
| 20:08:52 | mriedem | mordred: which is 22 seconds after your failure | |
| 20:09:02 | mriedem | fails here 2017-01-31_19_02_00_177 | |
| 20:09:03 | mordred | mriedem: oh. I mean ... | |
| 20:09:14 | mriedem | nova_cell0 db schema is being migrated here 2017-01-31_19_02_22_726 | |
| 20:09:40 | mordred | mriedem: it seems that devstack is still setting itself up when nodepool is running :( | |
| 20:10:55 | mordred | jeblair: ^^ you may (or may not) find this scrollback interesting | |
| 20:11:13 | mordred | jeblair: tl;dr - it's possible openstack is not yet ready when nodepool tries to start talking to it | |
| 20:13:18 | mordred | sdague, ianw: ^^ you two may or may not also find it interesting | |
| 20:13:32 | jeblair | mordred: that's weirdly parallel with our zookeeper conversation... but anyway, wouldn't any devstack test (eg tempest) have this issue? | |
| 20:13:33 | mriedem | so... | |
| 20:13:47 | mriedem | jeblair: d-g runs tempest after stack.sh is done | |
| 20:13:48 | mriedem | so no | |
| 20:14:14 | mriedem | i think we could move that map_cell0 call way further up in the stack run though, | |
| 20:14:27 | sdague | mordred: is there a log somewhere? | |
| 20:14:28 | mriedem | so that the cell0 schema is migrated way early when setting up the dbs, not at the tail end | |
| 20:14:34 | mriedem | dansmith: melwitt: ^ | |
| 20:14:35 | mordred | mriedem: ++ | |
| 20:14:50 | mordred | jeblair: OOOOHHHHHH - we install nodepool as a devstack plugin - we do not install it during the post_test_hook | |
| 20:14:52 | mriedem | i'll push something up to test that out | |
| 20:15:11 | jeblair | mordred: aha, of course. :) | |
| 20:15:26 | mriedem | mordred: it's always good when it's the users fault :) | |
| 20:15:50 | mordred | mriedem: :) | |
| 20:16:01 | mordred | mriedem: although I vote for moving the cells call way further up too | |
| 20:16:09 | mriedem | yeah i can still tinker on that | |
| 20:16:20 | sdague | mordred: yeh, the post_test_hook should give you a better guaruntee of where you want to be | |
| 20:16:36 | sdague | you only need to move your start there, right? | |
| 20:16:37 | mordred | sdague: ++ | |
| 20:16:57 | sdague | all the install / configure can happen earlier I would think | |
| 20:17:31 | sdague | the post_test_hook is kind of recent, because ironic needed to ensure they could do a thing after tempest was installed and configured | |
| 20:18:01 | mordred | well, we're already using post_test_hook too | |
| 20:18:07 | sdague | mordred: oh, interesting | |
| 20:18:08 | mordred | so I think we just need to consolidate a bit | |
| 20:18:24 | mtreinish | sdague: I thought post_test_hook was the d-g for running things after testing is done? | |
| 20:18:51 | sdague | mtreinish: oh, sorry, there is a new devstack plugin phase | |
| 20:18:53 | sdague | one sec | |
| 20:19:04 | mtreinish | I thought I called the plugin phase test-config or something like that | |
| 20:19:12 | sdague | yeh, that | |
| 20:19:13 | mordred | I may need/want that - because we're using devstack functions like crazy | |
| 20:19:29 | mordred | so I make a test_config_nodepool function? | |
| 20:19:33 | mordred | to use that phase? | |
| 20:19:46 | sdague | mordred: yeh, test-config | |
| 20:20:00 | sdague | mordred: so change https://github.com/openstack-infra/nodepool/blob/12d3caf560aa827ed47f320259cdb66a20f1e24b/devstack/plugin.sh#L465-L469 | |
| 20:20:05 | sdague | s/extra/test-config/ | |
| 20:20:14 | mordred | cool. thank you! that'll be super helpful | |
| 20:20:39 | sdague | that executes here in stack.sh - https://github.com/openstack-dev/devstack/blob/e6f8dbb55fc8fccb18f4ccb5ed5723a2efc3b025/stack.sh#L1401 | |
| 20:22:32 | mriedem | mordred: sdague: here is the devstack change https://review.openstack.org/427392 | |
| 20:23:17 | mordred | mriedem: I tell you what - we can get some code written aroud here | |
| 20:23:28 | mriedem | just not reviewed or merged :) | |
| 20:23:30 | mriedem | hi-o! | |
| 20:24:30 | mordred | :) | |
| 20:24:44 | mordred | I've been trying to land a shade patch since last tuesday | |
| 20:24:59 | mordred | I'm going to get it landed today if I have to write and land a patch to every repo in openstack | |
| 20:27:14 | openstackgerrit | melanie witt proposed openstack/nova: Read instances from API cell for cells v1 https://review.openstack.org/427394 | |
| 20:27:27 | mtreinish | mordred: don't you have ninja merge powers... | |
| 20:28:17 | mordred | mtreinish: don't tempt me | |
| 20:29:51 | melwitt | dansmith: ^ see if that's what you expect for the cells v1 get_all | |
| 20:30:38 | openstackgerrit | Eric Fried proposed openstack/nova: Introduce base PowerVM driver (disk drv) https://review.openstack.org/409401 | |
| 20:46:03 | mriedem | melwitt: lgtm for the most part - would be nice to have a comment in there about the conditional | |
| 20:46:06 | mriedem | like we have in _get_instance | |
| 20:46:19 | cdent | we collided a bit on that | |
| 20:46:21 | melwitt | mriedem: yeah, adding now | |
| 20:46:30 | melwitt | I saw cdent's comment | |
| 20:46:58 | mriedem | i had to pull it down to make sure the else condition was being hit in tests | |
| 20:47:12 | mriedem | since the compute api unit tests are confusing with the cells v1 and non-cells v1 stuff | |
| 20:49:43 | dansmith | melwitt: effectively, yeah | |
| 20:51:16 | openstackgerrit | melanie witt proposed openstack/nova: Read instances from API cell for cells v1 https://review.openstack.org/427394 | |
| 20:53:02 | mriedem | +2 | |
| 20:53:31 | openstackgerrit | Anusha Unnam proposed openstack/nova: Add post hook script to run tempest with old n-api https://review.openstack.org/401395 | |
| 20:54:19 | openstackgerrit | Eric Fried proposed openstack/nova: Introduce base PowerVM driver https://review.openstack.org/391288 | |
| 20:55:55 | openstackgerrit | Eric Fried proposed openstack/nova: PowerVM Driver: power_on/off and reboot https://review.openstack.org/427380 | |
| 20:56:31 | mriedem | oomichi: you were +2 before https://review.openstack.org/#/c/422171/ want to get that one now? | |
| 20:56:53 | openstackgerrit | Eric Fried proposed openstack/nova: Introduce base PowerVM driver (disk drv) https://review.openstack.org/409401 | |
| 21:02:54 | openstackgerrit | Matt Riedemann proposed openstack/nova: doc: add some detail to the map_cell0 man page https://review.openstack.org/427450 | |
| 21:02:58 | mriedem | diana_clarke: ^ addresses something i noted in your map_cell0 man page change after it was approved | |
| 21:04:06 | openstackgerrit | Eric Fried proposed openstack/nova: PowerVM Driver: spawn/destroy https://review.openstack.org/391288 | |
| 21:06:27 | openstackgerrit | Eric Fried proposed openstack/nova: PowerVM Driver: power_on/off and reboot https://review.openstack.org/427380 | |
| 21:07:18 | openstackgerrit | Eric Fried proposed openstack/nova: Introduce base PowerVM driver (disk drv) https://review.openstack.org/409401 | |
| 21:16:20 | openstackgerrit | Scott DAngelo proposed openstack/nova: WIP Support cinder v3 API https://review.openstack.org/420201 | |
| 21:17:45 | mriedem | browne: so looking at when the newton version aliases were made, that happened in ocata https://review.openstack.org/#/c/386625/ | |
| 21:18:03 | mriedem | browne: which leads me to believe we should wait to do https://review.openstack.org/#/c/425958/ in pike | |
| 21:18:10 | mriedem | dansmith: is that right? or does it not matter? ^ | |
| 21:19:46 | browne | mriedem: ok, so maybe it can wait | |
| 21:20:22 | mriedem | ooo i found the checklist https://wiki.openstack.org/wiki/Nova/ReleaseChecklist | |
| 21:21:35 | openstackgerrit | Chris Dent proposed openstack/nova: Optional separate database for placement API https://review.openstack.org/362766 | |
| 21:21:58 | cdent | omg, "scheduler calling the placement api" just merged. | |
| 21:22:07 | bauzas | indeed | |
| 21:22:15 | mriedem | i thought it was getting bumped again | |
| 21:22:17 | bauzas | back from the cave | |