| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-09-04 | |||
| 18:17:33 | cdent | mriedem: what's up? | |
| 18:17:35 | prometheanfire | they are the same for me | |
| 18:17:39 | jaypipes | right... | |
| 18:17:40 | mriedem | jaypipes: no he hasn't, | |
| 18:17:46 | mriedem | but i don't know that the code is correctly looking at this | |
| 18:17:55 | cdent | which code? | |
| 18:18:03 | mriedem | cdent: http://paste.openstack.org/show/729438/ | |
| 18:18:04 | prometheanfire | that placement_database stuff only went in for rocky | |
| 18:18:04 | openstackgerrit | melanie witt proposed openstack/nova stable/rocky: Add functional test for affinity with multiple cells https://review.openstack.org/599731 | |
| 18:18:05 | openstackgerrit | melanie witt proposed openstack/nova stable/rocky: Make scheduler.utils.setup_instance_group query all cells https://review.openstack.org/599732 | |
| 18:18:09 | cdent | thanks | |
| 18:18:15 | mriedem | online_data_migrations trying to hit the projects table, | |
| 18:18:20 | mriedem | but i don't know which db it's trying to hit | |
| 18:18:30 | mriedem | prometheanfire: you're testing here is on rocky yes? | |
| 18:18:33 | mriedem | *your | |
| 18:18:40 | prometheanfire | yes | |
| 18:19:20 | mriedem | i believe devstack in rocky is configuring the placement_database, but this should still work without placement_database, otherwise grenade wouldn't work | |
| 18:20:13 | cdent | mriedem: devstack will only use placement_database is a flag var is set, which we set in nova-next, but not elsewhere (unless someone else has changed it) | |
| 18:20:29 | cdent | #PLACEMENT_DB_ENABLED=True | |
| 18:20:45 | mriedem | ok | |
| 18:21:49 | prometheanfire | no way I can get more debug code? | |
| 18:22:24 | prometheanfire | wasn't there a sqlalchemy setting for more debug? | |
| 18:22:35 | mriedem | yes connection_debug or something | |
| 18:22:50 | mriedem | https://docs.openstack.org/nova/latest/configuration/config.html#api_database.connection_debug | |
| 18:23:11 | prometheanfire | connection_trace = False (Boolean) Add Python stack traces to SQL as comment strings. | |
| 18:23:14 | prometheanfire | that one?? | |
| 18:23:28 | mriedem | there is also connection_debug | |
| 18:23:31 | mriedem | i don't know which is better | |
| 18:23:34 | prometheanfire | ah, that one | |
| 18:23:40 | mriedem | the banner hides everything | |
| 18:24:05 | mriedem | cdent: i don't see what calls this https://github.com/openstack/nova/blob/master/nova/api/openstack/placement/db_api.py#L27 except for the placement wsgi code | |
| 18:25:33 | cdent | mriedem: there's a similar thing in nova-manage for db_sync https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L861-L866 . If that got missed elsewhere, could be a problem | |
| 18:25:48 | prometheanfire | I'm fine re-creating the cell, but I think my instances would be dead at that point | |
| 18:25:50 | mriedem | looks like it used to happen down in the db api code https://review.openstack.org/#/c/541435/ | |
| 18:25:52 | mriedem | before ^ | |
| 18:26:08 | mriedem | prometheanfire: this doesn't have anything to do with the cell db | |
| 18:27:16 | cdent | What was the original command that started this investingation? | |
| 18:27:17 | prometheanfire | ok | |
| 18:27:22 | jaypipes | mriedem: are we able to get all the instance info from the nova API db for create_incomplete_consumers() then? | |
| 18:27:27 | mriedem | cdent: nova-manage db online_data_migrations | |
| 18:27:28 | prometheanfire | cdent: top of paste | |
| 18:27:37 | cdent | thanks | |
| 18:27:37 | mriedem | jaypipes: no | |
| 18:27:52 | mriedem | jaypipes: you need the instance.user_id right? | |
| 18:28:01 | jaypipes | mriedem: and instance.project_id. | |
| 18:28:10 | mriedem | we have the project_id in the instance_mappings table in the API DB | |
| 18:28:10 | jaypipes | mriedem: that's why I presumed we needed to hit the cell DB. | |
| 18:28:13 | mriedem | but not the user_id | |
| 18:28:42 | mriedem | the online data migration, which you wrote i might add :) - relies on the consumer information in the allocations table, which is populated via running nova | |
| 18:29:21 | mriedem | my guess is https://review.openstack.org/#/c/541435/ regressed something but i have no idea how prometheanfire would hit this but we don't in the gate | |
| 18:29:33 | jaypipes | mriedem: we're talking about populating placement with missing *nova* information. | |
| 18:29:37 | prometheanfire | mriedem: old and jankey install | |
| 18:29:48 | prometheanfire | is there a way to re-init the cell db? | |
| 18:30:20 | prometheanfire | I think my cell db for cell1 is the same as the general nova db | |
| 18:30:25 | prometheanfire | not sure that's a good thing :| | |
| 18:30:32 | mriedem | jaypipes: i don't know what you're talking about. looking at the online data migration, clearly it doesn't care about nova instance information from the cell dbs | |
| 18:30:42 | mriedem | prometheanfire: yes that's expected | |
| 18:30:55 | prometheanfire | ok, good | |
| 18:31:03 | mriedem | jaypipes: this https://github.com/openstack/nova/blob/master/nova/api/openstack/placement/objects/consumer.py#L44 | |
| 18:31:04 | prometheanfire | cell0 has it's own db | |
| 18:31:09 | mriedem | prometheanfire: yes | |
| 18:31:10 | prometheanfire | and nova_api | |
| 18:31:17 | cdent | mriedem: that seems like a good guess. is there a chance we don't have tests that exercise the online migrations? | |
| 18:31:26 | mriedem | cdent: devstack runs them | |
| 18:31:44 | mriedem | https://github.com/openstack-dev/devstack/blob/5da7e4a22ede5f3049e7607a54a0f5ca2b413a29/lib/nova#L787 | |
| 18:31:54 | cdent | is there a paste of prometheanfire's nova.conf somewhere? | |
| 18:32:20 | prometheanfire | no | |
| 18:32:22 | jaypipes | mriedem: sorry, I was referring to https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L1783 | |
| 18:32:33 | mriedem | jaypipes: yeah totally different thing | |
| 18:32:38 | jaypipes | mriedem: sorry | |
| 18:32:43 | mriedem | np | |
| 18:32:45 | prometheanfire | gimme a sec | |
| 18:33:16 | cdent | mriedem: but those migrations aren't checked for count and if a migration method fails all it does is print? | |
| 18:34:22 | cdent | I might be reading this wrong: https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L677-L682 | |
| 18:36:03 | prometheanfire | cdent: http://paste.openstack.org/show/729439/ | |
| 18:37:42 | cdent | thanks | |
| 18:38:00 | mriedem | well if it failed like it is in prometheanfire's paste, we'd notice | |
| 18:38:25 | mriedem | but lemme check the logs | |
| 18:38:29 | mriedem | maybe that Exception is masking it | |
| 18:38:38 | prometheanfire | mriedem: one reason why I think it's an artifact of the old install | |
| 18:39:15 | mriedem | F ME http://logs.openstack.org/08/599208/2/check/tempest-full/608d60a/controller/logs/devstacklog.txt.gz#_2018-09-02_15_04_31_949 | |
| 18:39:21 | mriedem | yup, broken since rocky | |
| 18:39:28 | mriedem | melwitt: time for RC10 | |
| 18:39:32 | cdent | ugh | |
| 18:39:36 | mriedem | well, added to and broken in rocky | |
| 18:39:59 | cdent | didn't papa python always teach us never to catch exception :( | |
| 18:40:31 | mriedem | prometheanfire: you want to create the bug this time? | |
| 18:40:38 | mriedem | it's nearly 2pm and i haven't had lunh yet | |
| 18:40:40 | mriedem | *lunch | |
| 18:40:58 | prometheanfire | mriedem: ya, I'm about to go get lunch I think | |
| 18:41:10 | prometheanfire | mriedem: just the output of my migration in the bug? | |
| 18:41:32 | cdent | if nobody else is aching to fix this tonight, I can do it tomorrow morning | |
| 18:43:19 | openstack | Launchpad bug 1790701 in OpenStack Compute (nova) "online_data_migrations fail in rocky+" [Undecided,New] | |
| 18:43:19 | prometheanfire | https://bugs.launchpad.net/nova/+bug/1790701 | |
| 18:43:20 | prometheanfire | for now | |
| 18:44:40 | prometheanfire | 0 | |
| 18:44:40 | prometheanfire | echo $? | |
| 18:44:44 | prometheanfire | LOL, exits 0 | |
| 18:44:48 | prometheanfire | I'll add that to the bug | |
| 18:45:17 | cdent | should we: a) make a devstack bug to make it be more unhappy when doing the migrations, or b) simply fix the exist code on the script? | |