| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-09-04 | |||
| 18:04:12 | mriedem | as in you moved in ocata? or rocky? | |
| 18:04:16 | mriedem | or... | |
| 18:04:28 | mriedem | and i'm assuming list_cells works now b/c you set a name in the DB? | |
| 18:04:35 | prometheanfire | it does | |
| 18:04:55 | mriedem | i just want to make sure we have a nova bug to track that, i'll open it | |
| 18:05:09 | prometheanfire | online data migrations fail though | |
| 18:06:21 | prometheanfire | it keeps on trying to look up the projects table in the nova database, not the nova_api database | |
| 18:06:40 | mriedem | projects table? | |
| 18:06:51 | mriedem | got a paste? | |
| 18:06:55 | melwitt | must be the placement db | |
| 18:07:10 | melwitt | (projects table) | |
| 18:07:28 | prometheanfire | mriedem: http://paste.openstack.org/show/729438/ | |
| 18:07:34 | mriedem | ok so the create_incomplete_consumers online data migration | |
| 18:07:53 | mriedem | prometheanfire: is the placement db defined in nova.conf separately? in [placement_database]? | |
| 18:08:02 | prometheanfire | mriedem: atm, no | |
| 18:08:16 | mriedem | did you run nova-manage api_db sync before running online_data_migrations? | |
| 18:08:46 | mriedem | you've gotta sync the api db schema before running the online data migrations | |
| 18:08:51 | prometheanfire | yep | |
| 18:08:55 | mriedem | https://bugs.launchpad.net/nova/+bug/1790695 btw | |
| 18:08:55 | openstack | Launchpad bug 1790695 in OpenStack Compute (nova) "TypeError in nova-manage cell_v2 list_cells if a cell does not have a name" [High,Triaged] | |
| 18:09:45 | prometheanfire | where does create_incomplete_consumers get the db that it's connecting to? | |
| 18:11:37 | prometheanfire | also, running nova-manage with --debug doesn't print much debug info :P | |
| 18:13:50 | mriedem | create_incomplete_consumers is getting an admin RequestContext which won't have any db connection set on it | |
| 18:14:02 | mriedem | that context should then be changed by @db_api.placement_context_manager.writer | |
| 18:14:28 | jaypipes | prometheanfire, mriedem: create_incomplete_consumers would need to query both the API database (which == placement DB) as well as the nova cell DBs, right? | |
| 18:14:37 | mriedem | which should use the api_database if placement_database isn't configured | |
| 18:15:53 | mriedem | jaypipes: there is nothing about that online data migration that is iterating cell dbs | |
| 18:16:06 | mriedem | and no it doesn't need to hit the cell dbs | |
| 18:16:16 | mriedem | it creates missing consumer records for existing allocations records | |
| 18:16:44 | mriedem | what i don't know is if this is trying to connect to the placement database or the api db | |
| 18:17:05 | mriedem | cdent wrote the placement db stuff so he'd likely need to look | |
| 18:17:06 | jaypipes | mriedem: those are the same thing, no? | |
| 18:17:10 | mriedem | no | |
| 18:17:14 | mriedem | not if you configure the placement db | |
| 18:17:30 | jaypipes | but isn't that a deliberate thing? has prometheanfire deliberately configured a placement DB? | |
| 18:17:33 | cdent | mriedem: what's up? | |
| 18:17:35 | prometheanfire | they are the same for me | |
| 18:17:39 | jaypipes | right... | |
| 18:17:40 | mriedem | jaypipes: no he hasn't, | |
| 18:17:46 | mriedem | but i don't know that the code is correctly looking at this | |
| 18:17:55 | cdent | which code? | |
| 18:18:03 | mriedem | cdent: http://paste.openstack.org/show/729438/ | |
| 18:18:04 | openstackgerrit | melanie witt proposed openstack/nova stable/rocky: Add functional test for affinity with multiple cells https://review.openstack.org/599731 | |
| 18:18:04 | prometheanfire | that placement_database stuff only went in for rocky | |
| 18:18:05 | openstackgerrit | melanie witt proposed openstack/nova stable/rocky: Make scheduler.utils.setup_instance_group query all cells https://review.openstack.org/599732 | |
| 18:18:09 | cdent | thanks | |
| 18:18:15 | mriedem | online_data_migrations trying to hit the projects table, | |
| 18:18:20 | mriedem | but i don't know which db it's trying to hit | |
| 18:18:30 | mriedem | prometheanfire: you're testing here is on rocky yes? | |
| 18:18:33 | mriedem | *your | |
| 18:18:40 | prometheanfire | yes | |
| 18:19:20 | mriedem | i believe devstack in rocky is configuring the placement_database, but this should still work without placement_database, otherwise grenade wouldn't work | |
| 18:20:13 | cdent | mriedem: devstack will only use placement_database is a flag var is set, which we set in nova-next, but not elsewhere (unless someone else has changed it) | |
| 18:20:29 | cdent | #PLACEMENT_DB_ENABLED=True | |
| 18:20:45 | mriedem | ok | |
| 18:21:49 | prometheanfire | no way I can get more debug code? | |
| 18:22:24 | prometheanfire | wasn't there a sqlalchemy setting for more debug? | |
| 18:22:35 | mriedem | yes connection_debug or something | |
| 18:22:50 | mriedem | https://docs.openstack.org/nova/latest/configuration/config.html#api_database.connection_debug | |
| 18:23:11 | prometheanfire | connection_trace = False (Boolean) Add Python stack traces to SQL as comment strings. | |
| 18:23:14 | prometheanfire | that one?? | |
| 18:23:28 | mriedem | there is also connection_debug | |
| 18:23:31 | mriedem | i don't know which is better | |
| 18:23:34 | prometheanfire | ah, that one | |
| 18:23:40 | mriedem | the banner hides everything | |
| 18:24:05 | mriedem | cdent: i don't see what calls this https://github.com/openstack/nova/blob/master/nova/api/openstack/placement/db_api.py#L27 except for the placement wsgi code | |
| 18:25:33 | cdent | mriedem: there's a similar thing in nova-manage for db_sync https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L861-L866 . If that got missed elsewhere, could be a problem | |
| 18:25:48 | prometheanfire | I'm fine re-creating the cell, but I think my instances would be dead at that point | |
| 18:25:50 | mriedem | looks like it used to happen down in the db api code https://review.openstack.org/#/c/541435/ | |
| 18:25:52 | mriedem | before ^ | |
| 18:26:08 | mriedem | prometheanfire: this doesn't have anything to do with the cell db | |
| 18:27:16 | cdent | What was the original command that started this investingation? | |
| 18:27:17 | prometheanfire | ok | |
| 18:27:22 | jaypipes | mriedem: are we able to get all the instance info from the nova API db for create_incomplete_consumers() then? | |
| 18:27:27 | mriedem | cdent: nova-manage db online_data_migrations | |
| 18:27:28 | prometheanfire | cdent: top of paste | |
| 18:27:37 | mriedem | jaypipes: no | |
| 18:27:37 | cdent | thanks | |
| 18:27:52 | mriedem | jaypipes: you need the instance.user_id right? | |
| 18:28:01 | jaypipes | mriedem: and instance.project_id. | |
| 18:28:10 | jaypipes | mriedem: that's why I presumed we needed to hit the cell DB. | |
| 18:28:10 | mriedem | we have the project_id in the instance_mappings table in the API DB | |
| 18:28:13 | mriedem | but not the user_id | |
| 18:28:42 | mriedem | the online data migration, which you wrote i might add :) - relies on the consumer information in the allocations table, which is populated via running nova | |
| 18:29:21 | mriedem | my guess is https://review.openstack.org/#/c/541435/ regressed something but i have no idea how prometheanfire would hit this but we don't in the gate | |
| 18:29:33 | jaypipes | mriedem: we're talking about populating placement with missing *nova* information. | |
| 18:29:37 | prometheanfire | mriedem: old and jankey install | |
| 18:29:48 | prometheanfire | is there a way to re-init the cell db? | |
| 18:30:20 | prometheanfire | I think my cell db for cell1 is the same as the general nova db | |
| 18:30:25 | prometheanfire | not sure that's a good thing :| | |
| 18:30:32 | mriedem | jaypipes: i don't know what you're talking about. looking at the online data migration, clearly it doesn't care about nova instance information from the cell dbs | |
| 18:30:42 | mriedem | prometheanfire: yes that's expected | |
| 18:30:55 | prometheanfire | ok, good | |
| 18:31:03 | mriedem | jaypipes: this https://github.com/openstack/nova/blob/master/nova/api/openstack/placement/objects/consumer.py#L44 | |
| 18:31:04 | prometheanfire | cell0 has it's own db | |
| 18:31:09 | mriedem | prometheanfire: yes | |
| 18:31:10 | prometheanfire | and nova_api | |
| 18:31:17 | cdent | mriedem: that seems like a good guess. is there a chance we don't have tests that exercise the online migrations? | |
| 18:31:26 | mriedem | cdent: devstack runs them | |
| 18:31:44 | mriedem | https://github.com/openstack-dev/devstack/blob/5da7e4a22ede5f3049e7607a54a0f5ca2b413a29/lib/nova#L787 | |