| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-09-04 | |||
| 17:17:44 | sean-k-mooney | prometheanfire: https://github.com/openstack/osc-placement for future reference. it should be on pypi too | |
| 17:17:44 | prometheanfire | mriedem: which errors you looking for? | |
| 17:17:52 | mriedem | prometheanfire: umm, any errors? | |
| 17:18:14 | mriedem | presumably "failed to create resource provider in placement" or something like that | |
| 17:18:16 | prometheanfire | mriedem: none now, I rebuilt all my systems from the ground up to fix the odd python issue | |
| 17:18:23 | SamYaple | mriedem: sorry, i said backport. i mean hack something in based on that commit. i stay pretty close to upstream | |
| 17:18:40 | prometheanfire | mriedem: testing now via ' su - nova -c "/usr/lib/python-exec/python3.5/nova-compute --config-file /etc/nova/nova-compute.conf --config-file /etc/nova/nova.conf" ' | |
| 17:19:03 | mriedem | SamYaple: ok, well, good luck. :) if that series of backports on pike also applies to ocata (i'm not sure if they all do), we could also backport them to ocata upstream | |
| 17:19:07 | mriedem | all official like | |
| 17:20:53 | sean-k-mooney | prometheanfire: you should get somting like "ResourceProviderCreationFailed" if the placement info is missing/wrong in the conf | |
| 17:20:58 | prometheanfire | mriedem: http://paste.openstack.org/show/729433/ | |
| 17:21:00 | prometheanfire | lgtm | |
| 17:21:19 | melwitt | . | |
| 17:21:20 | sean-k-mooney | prometheanfire: that what i get when i comment it out but the agent keeps running which i was surrprised by | |
| 17:22:55 | openstackgerrit | Merged openstack/nova master: Removing pip-missing-reqs from default tox jobs https://review.openstack.org/599442 | |
| 17:23:10 | mriedem | prometheanfire: it's likely logged at debug | |
| 17:23:18 | prometheanfire | ah, k | |
| 17:23:23 | mriedem | do you see the resource provider in placement now? | |
| 17:23:48 | prometheanfire | mriedem: I'm packaging osc-placement now | |
| 17:24:00 | SamYaple | mriedem: ack. will do | |
| 17:24:04 | sean-k-mooney | mriedem: no its logging at error if it fails | |
| 17:26:51 | prometheanfire | sean-k-mooney: mriedem does osc-placement need to be installed on the compute host? | |
| 17:27:28 | cdent | prometheanfire: no, it's only an openstack client plugin, for humans | |
| 17:27:50 | prometheanfire | that's what I thought | |
| 17:32:53 | sean-k-mooney | prometheanfire: its not needed but it makes debuging placment issues simpeler as you dont have to dive into the db | |
| 17:34:04 | prometheanfire | openstack resource provider list - shows my two nodes | |
| 17:34:38 | sean-k-mooney | prometheanfire: then placement is happy with them. what was the warning you were getting with nova status-check? | |
| 17:35:29 | prometheanfire | one is generation 76 and one is generation 34 though | |
| 17:35:59 | prometheanfire | sean-k-mooney: http://paste.openstack.org/show/729434/ | |
| 17:36:59 | sean-k-mooney | cdent: do you no if nova-status upgrade check uses the api or try to connect to the db directly? | |
| 17:37:23 | cdent | sean-k-mooney: api. it talks to the db for nova stuff, api for placement stuff | |
| 17:37:54 | cdent | sean-k-mooney: yes, it is useful for debugging, but it doesn't have to be _on_ the compute node... | |
| 17:38:32 | sean-k-mooney | prometheanfire: cdent yes. im wondering if nova-status if not finding the placement endoint or somthing in prometheanfire case | |
| 17:38:56 | sean-k-mooney | cdent: sorry ^ was ment for you | |
| 17:39:43 | sean-k-mooney | prometheanfire: do you have the placement setting in the nova.conf also or jsut /etc/nova/nova-cpu.conf? | |
| 17:40:14 | prometheanfire | sean-k-mooney: I have it in nova.conf for all hosts (nova-api and compute hosts) | |
| 17:41:00 | prometheanfire | online-data migrations fail too, I think it's looking in the wrong db here | |
| 17:41:28 | sean-k-mooney | ok i was going to suggest using --config-file=/path/to/conf to make sure its not reading values form another location | |
| 17:41:44 | cdent | sorry sean-k-mooney, I haven't been tracking prometheanfire's situation, just jumping at random points. in too many conversations at once | |
| 17:42:08 | sean-k-mooney | cdent: no worries. | |
| 17:42:54 | sean-k-mooney | prometheanfire: i would try "nova-status --debug --config-dir /etc/nova/nova.conf upgrade check" | |
| 17:43:19 | prometheanfire | same | |
| 17:44:22 | prometheanfire | sean-k-mooney: does placement look at the cell_mappings table of the api db? | |
| 17:45:05 | sean-k-mooney | prometheanfire: i dont think placement does but if you have not do the cellsv2 discover host thing this might have issues | |
| 17:46:58 | sean-k-mooney | prometheanfire: actull i think the output is missleading | |
| 17:47:24 | sean-k-mooney | prometheanfire: i get http://paste.openstack.org/show/729436/ | |
| 17:48:05 | prometheanfire | sean-k-mooney: that's diferent than mine | |
| 17:49:04 | prometheanfire | nova-manage cell_v2 list_hosts | |
| 17:49:10 | prometheanfire | shows my hosts as not mapped to a cell | |
| 17:49:13 | sean-k-mooney | yes it is but 1 i got a success and its an all in one devstrck that is passing tempest test so it works | |
| 17:49:28 | sean-k-mooney | so i think the test might be wrong | |
| 17:51:07 | sean-k-mooney | hum ok well ill try to look into this more tomorow. my brain is fried for today so im going to call it a day o/ | |
| 17:52:33 | prometheanfire | k | |
| 17:57:15 | prometheanfire | list cells backtraces | |
| 17:57:17 | prometheanfire | http://paste.openstack.org/show/729437/ | |
| 17:57:26 | mriedem | prometheanfire: you still don't have a resource provider in placement but no errors in the n-cpu logs? | |
| 17:58:59 | prometheanfire | mriedem: shows my two nodes | |
| 17:59:29 | mriedem | huh your cells must not have names | |
| 17:59:34 | prometheanfire | my api_db has two entries for cell_mappings | |
| 17:59:36 | prometheanfire | select * from cell_mappings; | |
| 17:59:38 | mriedem | which is the TypeError | |
| 17:59:51 | prometheanfire | ya, one of them has no name | |
| 17:59:59 | mriedem | ok that's a bug clearly | |
| 18:00:07 | mriedem | you want to report it or shall i? | |
| 18:01:29 | prometheanfire | I'm still figuring stuff out | |
| 18:01:53 | prometheanfire | I think a lot of my bugs in this area are because I moved to cells before it was ready | |
| 18:02:39 | prometheanfire | list_hosts now has the cell name :D | |
| 18:02:48 | prometheanfire | mriedem: can I delete cell0 or is that still used? | |
| 18:03:33 | mriedem | cell0 is required | |
| 18:03:44 | mriedem | it's where instances records are created for things that fail to schedule | |
| 18:03:55 | prometheanfire | k | |
| 18:04:00 | mriedem | "for cell in sorted(cell_mappings, key=lambda _cell: _cell.name):" | |
| 18:04:02 | mriedem | oops | |
| 18:04:07 | mriedem | "because I moved to cells before it was ready" | |
| 18:04:12 | mriedem | as in you moved in ocata? or rocky? | |
| 18:04:16 | mriedem | or... | |
| 18:04:28 | mriedem | and i'm assuming list_cells works now b/c you set a name in the DB? | |
| 18:04:35 | prometheanfire | it does | |
| 18:04:55 | mriedem | i just want to make sure we have a nova bug to track that, i'll open it | |
| 18:05:09 | prometheanfire | online data migrations fail though | |
| 18:06:21 | prometheanfire | it keeps on trying to look up the projects table in the nova database, not the nova_api database | |
| 18:06:40 | mriedem | projects table? | |
| 18:06:51 | mriedem | got a paste? | |
| 18:06:55 | melwitt | must be the placement db | |
| 18:07:10 | melwitt | (projects table) | |
| 18:07:28 | prometheanfire | mriedem: http://paste.openstack.org/show/729438/ | |
| 18:07:34 | mriedem | ok so the create_incomplete_consumers online data migration | |
| 18:07:53 | mriedem | prometheanfire: is the placement db defined in nova.conf separately? in [placement_database]? | |
| 18:08:02 | prometheanfire | mriedem: atm, no | |
| 18:08:16 | mriedem | did you run nova-manage api_db sync before running online_data_migrations? | |
| 18:08:46 | mriedem | you've gotta sync the api db schema before running the online data migrations | |
| 18:08:51 | prometheanfire | yep | |
| 18:08:55 | openstack | Launchpad bug 1790695 in OpenStack Compute (nova) "TypeError in nova-manage cell_v2 list_cells if a cell does not have a name" [High,Triaged] | |
| 18:08:55 | mriedem | https://bugs.launchpad.net/nova/+bug/1790695 btw | |
| 18:09:45 | prometheanfire | where does create_incomplete_consumers get the db that it's connecting to? | |
| 18:11:37 | prometheanfire | also, running nova-manage with --debug doesn't print much debug info :P | |
| 18:13:50 | mriedem | create_incomplete_consumers is getting an admin RequestContext which won't have any db connection set on it | |
| 18:14:02 | mriedem | that context should then be changed by @db_api.placement_context_manager.writer | |
| 18:14:28 | jaypipes | prometheanfire, mriedem: create_incomplete_consumers would need to query both the API database (which == placement DB) as well as the nova cell DBs, right? | |
| 18:14:37 | mriedem | which should use the api_database if placement_database isn't configured | |