| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-09-04 | |||
| 17:43:19 | prometheanfire | same | |
| 17:44:22 | prometheanfire | sean-k-mooney: does placement look at the cell_mappings table of the api db? | |
| 17:45:05 | sean-k-mooney | prometheanfire: i dont think placement does but if you have not do the cellsv2 discover host thing this might have issues | |
| 17:46:58 | sean-k-mooney | prometheanfire: actull i think the output is missleading | |
| 17:47:24 | sean-k-mooney | prometheanfire: i get http://paste.openstack.org/show/729436/ | |
| 17:48:05 | prometheanfire | sean-k-mooney: that's diferent than mine | |
| 17:49:04 | prometheanfire | nova-manage cell_v2 list_hosts | |
| 17:49:10 | prometheanfire | shows my hosts as not mapped to a cell | |
| 17:49:13 | sean-k-mooney | yes it is but 1 i got a success and its an all in one devstrck that is passing tempest test so it works | |
| 17:49:28 | sean-k-mooney | so i think the test might be wrong | |
| 17:51:07 | sean-k-mooney | hum ok well ill try to look into this more tomorow. my brain is fried for today so im going to call it a day o/ | |
| 17:52:33 | prometheanfire | k | |
| 17:57:15 | prometheanfire | list cells backtraces | |
| 17:57:17 | prometheanfire | http://paste.openstack.org/show/729437/ | |
| 17:57:26 | mriedem | prometheanfire: you still don't have a resource provider in placement but no errors in the n-cpu logs? | |
| 17:58:59 | prometheanfire | mriedem: shows my two nodes | |
| 17:59:29 | mriedem | huh your cells must not have names | |
| 17:59:34 | prometheanfire | my api_db has two entries for cell_mappings | |
| 17:59:36 | prometheanfire | select * from cell_mappings; | |
| 17:59:38 | mriedem | which is the TypeError | |
| 17:59:51 | prometheanfire | ya, one of them has no name | |
| 17:59:59 | mriedem | ok that's a bug clearly | |
| 18:00:07 | mriedem | you want to report it or shall i? | |
| 18:01:29 | prometheanfire | I'm still figuring stuff out | |
| 18:01:53 | prometheanfire | I think a lot of my bugs in this area are because I moved to cells before it was ready | |
| 18:02:39 | prometheanfire | list_hosts now has the cell name :D | |
| 18:02:48 | prometheanfire | mriedem: can I delete cell0 or is that still used? | |
| 18:03:33 | mriedem | cell0 is required | |
| 18:03:44 | mriedem | it's where instances records are created for things that fail to schedule | |
| 18:03:55 | prometheanfire | k | |
| 18:04:00 | mriedem | "for cell in sorted(cell_mappings, key=lambda _cell: _cell.name):" | |
| 18:04:02 | mriedem | oops | |
| 18:04:07 | mriedem | "because I moved to cells before it was ready" | |
| 18:04:12 | mriedem | as in you moved in ocata? or rocky? | |
| 18:04:16 | mriedem | or... | |
| 18:04:28 | mriedem | and i'm assuming list_cells works now b/c you set a name in the DB? | |
| 18:04:35 | prometheanfire | it does | |
| 18:04:55 | mriedem | i just want to make sure we have a nova bug to track that, i'll open it | |
| 18:05:09 | prometheanfire | online data migrations fail though | |
| 18:06:21 | prometheanfire | it keeps on trying to look up the projects table in the nova database, not the nova_api database | |
| 18:06:40 | mriedem | projects table? | |
| 18:06:51 | mriedem | got a paste? | |
| 18:06:55 | melwitt | must be the placement db | |
| 18:07:10 | melwitt | (projects table) | |
| 18:07:28 | prometheanfire | mriedem: http://paste.openstack.org/show/729438/ | |
| 18:07:34 | mriedem | ok so the create_incomplete_consumers online data migration | |
| 18:07:53 | mriedem | prometheanfire: is the placement db defined in nova.conf separately? in [placement_database]? | |
| 18:08:02 | prometheanfire | mriedem: atm, no | |
| 18:08:16 | mriedem | did you run nova-manage api_db sync before running online_data_migrations? | |
| 18:08:46 | mriedem | you've gotta sync the api db schema before running the online data migrations | |
| 18:08:51 | prometheanfire | yep | |
| 18:08:55 | openstack | Launchpad bug 1790695 in OpenStack Compute (nova) "TypeError in nova-manage cell_v2 list_cells if a cell does not have a name" [High,Triaged] | |
| 18:08:55 | mriedem | https://bugs.launchpad.net/nova/+bug/1790695 btw | |
| 18:09:45 | prometheanfire | where does create_incomplete_consumers get the db that it's connecting to? | |
| 18:11:37 | prometheanfire | also, running nova-manage with --debug doesn't print much debug info :P | |
| 18:13:50 | mriedem | create_incomplete_consumers is getting an admin RequestContext which won't have any db connection set on it | |
| 18:14:02 | mriedem | that context should then be changed by @db_api.placement_context_manager.writer | |
| 18:14:28 | jaypipes | prometheanfire, mriedem: create_incomplete_consumers would need to query both the API database (which == placement DB) as well as the nova cell DBs, right? | |
| 18:14:37 | mriedem | which should use the api_database if placement_database isn't configured | |
| 18:15:53 | mriedem | jaypipes: there is nothing about that online data migration that is iterating cell dbs | |
| 18:16:06 | mriedem | and no it doesn't need to hit the cell dbs | |
| 18:16:16 | mriedem | it creates missing consumer records for existing allocations records | |
| 18:16:44 | mriedem | what i don't know is if this is trying to connect to the placement database or the api db | |
| 18:17:05 | mriedem | cdent wrote the placement db stuff so he'd likely need to look | |
| 18:17:06 | jaypipes | mriedem: those are the same thing, no? | |
| 18:17:10 | mriedem | no | |
| 18:17:14 | mriedem | not if you configure the placement db | |
| 18:17:30 | jaypipes | but isn't that a deliberate thing? has prometheanfire deliberately configured a placement DB? | |
| 18:17:33 | cdent | mriedem: what's up? | |
| 18:17:35 | prometheanfire | they are the same for me | |
| 18:17:39 | jaypipes | right... | |
| 18:17:40 | mriedem | jaypipes: no he hasn't, | |
| 18:17:46 | mriedem | but i don't know that the code is correctly looking at this | |
| 18:17:55 | cdent | which code? | |
| 18:18:03 | mriedem | cdent: http://paste.openstack.org/show/729438/ | |
| 18:18:04 | prometheanfire | that placement_database stuff only went in for rocky | |
| 18:18:04 | openstackgerrit | melanie witt proposed openstack/nova stable/rocky: Add functional test for affinity with multiple cells https://review.openstack.org/599731 | |
| 18:18:05 | openstackgerrit | melanie witt proposed openstack/nova stable/rocky: Make scheduler.utils.setup_instance_group query all cells https://review.openstack.org/599732 | |
| 18:18:09 | cdent | thanks | |
| 18:18:15 | mriedem | online_data_migrations trying to hit the projects table, | |
| 18:18:20 | mriedem | but i don't know which db it's trying to hit | |
| 18:18:30 | mriedem | prometheanfire: you're testing here is on rocky yes? | |
| 18:18:33 | mriedem | *your | |
| 18:18:40 | prometheanfire | yes | |
| 18:19:20 | mriedem | i believe devstack in rocky is configuring the placement_database, but this should still work without placement_database, otherwise grenade wouldn't work | |
| 18:20:13 | cdent | mriedem: devstack will only use placement_database is a flag var is set, which we set in nova-next, but not elsewhere (unless someone else has changed it) | |
| 18:20:29 | cdent | #PLACEMENT_DB_ENABLED=True | |
| 18:20:45 | mriedem | ok | |
| 18:21:49 | prometheanfire | no way I can get more debug code? | |
| 18:22:24 | prometheanfire | wasn't there a sqlalchemy setting for more debug? | |
| 18:22:35 | mriedem | yes connection_debug or something | |
| 18:22:50 | mriedem | https://docs.openstack.org/nova/latest/configuration/config.html#api_database.connection_debug | |
| 18:23:11 | prometheanfire | connection_trace = False (Boolean) Add Python stack traces to SQL as comment strings. | |
| 18:23:14 | prometheanfire | that one?? | |
| 18:23:28 | mriedem | there is also connection_debug | |
| 18:23:31 | mriedem | i don't know which is better | |
| 18:23:34 | prometheanfire | ah, that one | |
| 18:23:40 | mriedem | the banner hides everything | |
| 18:24:05 | mriedem | cdent: i don't see what calls this https://github.com/openstack/nova/blob/master/nova/api/openstack/placement/db_api.py#L27 except for the placement wsgi code | |
| 18:25:33 | cdent | mriedem: there's a similar thing in nova-manage for db_sync https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L861-L866 . If that got missed elsewhere, could be a problem | |