| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-09-04 | |||
| 18:25:48 | prometheanfire | I'm fine re-creating the cell, but I think my instances would be dead at that point | |
| 18:25:50 | mriedem | looks like it used to happen down in the db api code https://review.openstack.org/#/c/541435/ | |
| 18:25:52 | mriedem | before ^ | |
| 18:26:08 | mriedem | prometheanfire: this doesn't have anything to do with the cell db | |
| 18:27:16 | cdent | What was the original command that started this investingation? | |
| 18:27:17 | prometheanfire | ok | |
| 18:27:22 | jaypipes | mriedem: are we able to get all the instance info from the nova API db for create_incomplete_consumers() then? | |
| 18:27:27 | mriedem | cdent: nova-manage db online_data_migrations | |
| 18:27:28 | prometheanfire | cdent: top of paste | |
| 18:27:37 | cdent | thanks | |
| 18:27:37 | mriedem | jaypipes: no | |
| 18:27:52 | mriedem | jaypipes: you need the instance.user_id right? | |
| 18:28:01 | jaypipes | mriedem: and instance.project_id. | |
| 18:28:10 | mriedem | we have the project_id in the instance_mappings table in the API DB | |
| 18:28:10 | jaypipes | mriedem: that's why I presumed we needed to hit the cell DB. | |
| 18:28:13 | mriedem | but not the user_id | |
| 18:28:42 | mriedem | the online data migration, which you wrote i might add :) - relies on the consumer information in the allocations table, which is populated via running nova | |
| 18:29:21 | mriedem | my guess is https://review.openstack.org/#/c/541435/ regressed something but i have no idea how prometheanfire would hit this but we don't in the gate | |
| 18:29:33 | jaypipes | mriedem: we're talking about populating placement with missing *nova* information. | |
| 18:29:37 | prometheanfire | mriedem: old and jankey install | |
| 18:29:48 | prometheanfire | is there a way to re-init the cell db? | |
| 18:30:20 | prometheanfire | I think my cell db for cell1 is the same as the general nova db | |
| 18:30:25 | prometheanfire | not sure that's a good thing :| | |
| 18:30:32 | mriedem | jaypipes: i don't know what you're talking about. looking at the online data migration, clearly it doesn't care about nova instance information from the cell dbs | |
| 18:30:42 | mriedem | prometheanfire: yes that's expected | |
| 18:30:55 | prometheanfire | ok, good | |
| 18:31:03 | mriedem | jaypipes: this https://github.com/openstack/nova/blob/master/nova/api/openstack/placement/objects/consumer.py#L44 | |
| 18:31:04 | prometheanfire | cell0 has it's own db | |
| 18:31:09 | mriedem | prometheanfire: yes | |
| 18:31:10 | prometheanfire | and nova_api | |
| 18:31:17 | cdent | mriedem: that seems like a good guess. is there a chance we don't have tests that exercise the online migrations? | |
| 18:31:26 | mriedem | cdent: devstack runs them | |
| 18:31:44 | mriedem | https://github.com/openstack-dev/devstack/blob/5da7e4a22ede5f3049e7607a54a0f5ca2b413a29/lib/nova#L787 | |
| 18:31:54 | cdent | is there a paste of prometheanfire's nova.conf somewhere? | |
| 18:32:20 | prometheanfire | no | |
| 18:32:22 | jaypipes | mriedem: sorry, I was referring to https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L1783 | |
| 18:32:33 | mriedem | jaypipes: yeah totally different thing | |
| 18:32:38 | jaypipes | mriedem: sorry | |
| 18:32:43 | mriedem | np | |
| 18:32:45 | prometheanfire | gimme a sec | |
| 18:33:16 | cdent | mriedem: but those migrations aren't checked for count and if a migration method fails all it does is print? | |
| 18:34:22 | cdent | I might be reading this wrong: https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L677-L682 | |
| 18:36:03 | prometheanfire | cdent: http://paste.openstack.org/show/729439/ | |
| 18:37:42 | cdent | thanks | |
| 18:38:00 | mriedem | well if it failed like it is in prometheanfire's paste, we'd notice | |
| 18:38:25 | mriedem | but lemme check the logs | |
| 18:38:29 | mriedem | maybe that Exception is masking it | |
| 18:38:38 | prometheanfire | mriedem: one reason why I think it's an artifact of the old install | |
| 18:39:15 | mriedem | F ME http://logs.openstack.org/08/599208/2/check/tempest-full/608d60a/controller/logs/devstacklog.txt.gz#_2018-09-02_15_04_31_949 | |
| 18:39:21 | mriedem | yup, broken since rocky | |
| 18:39:28 | mriedem | melwitt: time for RC10 | |
| 18:39:32 | cdent | ugh | |
| 18:39:36 | mriedem | well, added to and broken in rocky | |
| 18:39:59 | cdent | didn't papa python always teach us never to catch exception :( | |
| 18:40:31 | mriedem | prometheanfire: you want to create the bug this time? | |
| 18:40:38 | mriedem | it's nearly 2pm and i haven't had lunh yet | |
| 18:40:40 | mriedem | *lunch | |
| 18:40:58 | prometheanfire | mriedem: ya, I'm about to go get lunch I think | |
| 18:41:10 | prometheanfire | mriedem: just the output of my migration in the bug? | |
| 18:41:32 | cdent | if nobody else is aching to fix this tonight, I can do it tomorrow morning | |
| 18:43:19 | openstack | Launchpad bug 1790701 in OpenStack Compute (nova) "online_data_migrations fail in rocky+" [Undecided,New] | |
| 18:43:19 | prometheanfire | https://bugs.launchpad.net/nova/+bug/1790701 | |
| 18:43:20 | prometheanfire | for now | |
| 18:44:40 | prometheanfire | 0 | |
| 18:44:40 | prometheanfire | echo $? | |
| 18:44:44 | prometheanfire | LOL, exits 0 | |
| 18:44:48 | prometheanfire | I'll add that to the bug | |
| 18:45:17 | cdent | should we: a) make a devstack bug to make it be more unhappy when doing the migrations, or b) simply fix the exist code on the script? | |
| 18:46:13 | prometheanfire | imo the exit code being fixed would should make devstack unhappy when it should be unhappy | |
| 18:46:40 | cdent | yes | |
| 18:47:06 | cdent | Until you mentioned the exit code my b was going to be something else that how now passed out of my mind... | |
| 18:47:31 | prometheanfire | :D | |
| 18:47:39 | prometheanfire | fixing the error is still nice though | |
| 18:47:39 | cdent | so two fixes, in nova: exit code handling on that command, intialize the placement db properly | |
| 18:47:49 | cdent | that's backportable then | |
| 18:48:17 | prometheanfire | is that what's happening (api_db sync not init'ing the placement stuff)? | |
| 18:48:20 | mriedem | i'll hack some stuff up | |
| 18:48:28 | mriedem | this isn't api_db sync | |
| 18:48:48 | mriedem | but yes the online_data_migraitons command isn't configuring a global properly | |
| 18:48:48 | cdent | then on top of that we need address that an online placement db migration probably shouldn't be in nova's db migrations (even if placement wasn't being extracted) | |
| 18:49:06 | prometheanfire | ok | |
| 18:49:34 | prometheanfire | I'll be around if you want me to test, I should be able to take a snapshot of my master for rollback type testing too if you want | |
| 18:49:42 | prometheanfire | zfs <3 | |
| 18:49:52 | cdent | mriedem: please ping me if you can't get around to it, or if you do add me to the review please | |
| 18:50:24 | mriedem | sure | |
| 18:55:15 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Swallow fewer exceptions in _run_migration https://review.openstack.org/599744 | |
| 18:56:59 | mriedem | fwiw that blanket try/except has been around since the command was added https://review.openstack.org/#/c/278078/ | |
| 18:58:01 | prometheanfire | mriedem: want a paste with my error with that patch | |
| 18:58:04 | prometheanfire | it's longer | |
| 18:58:14 | mriedem | sure | |
| 18:58:43 | prometheanfire | mriedem: seems to fix the exit code problem too | |
| 18:58:56 | mriedem | yeah b/c you're not getting a nova exception | |
| 18:59:04 | mriedem | so it kills the command | |
| 18:59:07 | mriedem | i expect my patch to make devstack fail | |
| 18:59:16 | prometheanfire | I don't think it's that helpful (my paste) | |
| 18:59:17 | prometheanfire | http://paste.openstack.org/show/729440/ | |
| 18:59:27 | prometheanfire | but it's there anyway | |
| 18:59:42 | mriedem | yeah it's what i'd expect | |
| 18:59:52 | mriedem | i'll push the real fix in a follow up and then squash them | |
| 19:00:00 | mriedem | so we can see the failure in devstack and the fix in the follow up | |