Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-07-24
17:35:33 s-dean sorry not it> Nova*
17:36:27 s-dean nova.api.openstack.extensions [req-3ed5c135-64bd-4c39-934d-3d3d7a32a6b5 14d1055c2e2d4f0eab6739671b2688a8 cf7b827f55de4406b9325ab3505b973d - default default] Unexpected exception in API method
17:39:38 melwitt s-dean: I think that's an RPC timeout, so you will want to check your transport_url for your cell1 and make sure it's correct
17:40:07 s-dean ok cheers will do that now
17:49:09 s-dean yeah, cell1's RPC mapping was wrong, i have changed it and restarted all nova services, no dice :/
17:49:19 s-dean still same error
17:50:17 s-dean can i ask would it be better to implement SSL/TLS after successfully setting up openstack unencrypted ?
17:50:27 dansmith s-dean: I'm not following along, but you're sure that the cell1 mapping points to the same rpc endpoint that your other services do?
17:51:51 melwitt s-dean: did you change it using "nova-manage cell_v2 update_cell"? as shown https://docs.openstack.org/nova/latest/cli/nova-manage.html#man-page-cells-v2
17:51:54 mriedem melwitt: i think https://blueprints.launchpad.net/nova/+spec/cells-count-resources-to-check-quota-in-api is done
17:52:05 mriedem https://review.openstack.org/#/c/446243/ was the last change to merge
17:52:24 s-dean no i did not
17:52:59 melwitt s-dean: yeah you need to use the nova-manage commands to get it set right in the database. you can use "nova-manage cell_v2 list_cells --verbose" to see what you currently have
17:53:12 s-dean ok cheers
17:53:22 melwitt these are stored in the database now because the API needs to know how to contact each of the cells you have
17:53:58 melwitt mriedem: agreed, I just didn't mean to set the Completed state because I think usually you do that
17:54:12 melwitt but it does it automatically if you put Implemented
17:55:05 mriedem melwitt: i like to yeah because i have a spreadsheet keeping track of when things are closed out
17:55:08 mriedem b/c i'm a nerd
17:55:35 melwitt mriedem: heh. I would probably do the same thing. don't want to mess up your system
17:56:01 mriedem just please be sure to put scissors, pens and dusters back where you got them if you're ever at my house
17:56:13 mriedem and windex
17:56:17 melwitt lol
18:00:05 s-dean do ineed to sync the db's after running that update command
18:00:32 melwitt s-dean: no, just need to restart your nova-api services I think
18:00:52 dansmith unless he never properly sync'd it as a result
18:01:15 s-dean yeah, still timing out
18:01:36 melwitt hm
18:02:14 openstackgerrit Matt Riedemann proposed openstack/nova master: Provide hints when nova-manage db sync fails to sync cell0 https://review.openstack.org/486660
18:03:18 dansmith s-dean: so if you grep transport_url out of your configs on all nodes, they match exactly what is in the db for the cell1 transport_url?
18:06:35 s-dean ok so i can confirm, that on my controller node the URL's are correct, however on the compute node it is not because of: https://docs.openstack.org/security-guide/messaging/security.html
18:07:32 mriedem sdague: did you ever report a bug for the request log thing with uwsgi?
18:07:52 sdague mriedem: I don't think I did actually
18:10:07 mriedem sdague: ok, did the same thing need to be done in other projects like cinder?
18:10:18 mriedem if so, could be good to track with a bug for those other projects too
18:13:50 dansmith s-dean: so...the compute node is not correct?
18:14:00 s-dean does the username need to be consistent across all nodes, or is having different users for each compute node ok ?
18:14:13 s-dean yes i gave the compute node its on users
18:14:15 s-dean user
18:14:23 s-dean and assigned it all permissions
18:14:31 s-dean own*
18:14:59 dansmith s-dean: they can be different, they just all have to point at the same vhost of course
18:15:12 s-dean in my case my vhost is /
18:15:31 dansmith as long as that is the same for all the nodes then that's fine
18:16:34 dansmith s-dean: aside from that, we need more context on which thing is failing, so more detailed logs, and a sanitized dump of your config and cell_mappings table are needed
18:17:09 s-dean paste bin urls ok ?
18:17:19 dansmith yeah
18:18:55 s-dean https://pastebin.com/WEu7JCWW
18:20:20 dansmith s-dean: are you running n-net?
18:20:36 dansmith s-dean: and looks like you're configured for cellsv1, is that intentional or did you think that was related?
18:20:57 dansmith because this timeout is trying to call the cellsv1 service, which if you're not running it, will always time out :)
18:21:55 s-dean oh ok
18:22:06 s-dean i didnt know i was running cells v1
18:22:24 s-dean is this because of the following config line compute_api_class=nova.compute.cells_api.ComputeCellsAPI
18:22:32 s-dean https://bugs.launchpad.net/nova/+bug/1706118
18:22:32 s-dean my config is here
18:22:33 dansmith s-dean: you definitely know if you're running cellsv1 because it's a huge friggin nightmare,
18:22:34 openstack Launchpad bug 1706118 in OpenStack Compute (nova) ocata "Adding TLS Connection String To Database Connection Causes malformed URL in cell_mappings DB" [High,Confirmed]
18:22:44 dansmith so I'm guessing maybe you configured that while trying to resolve this problem?
18:23:00 s-dean yeah i have been poking about quite a bit
18:23:18 dansmith s-dean: this is an upgrade from a previously working deployment, right?
18:23:57 s-dean no fresh install, but because of that DB error, i have been trying to debug and find the right settings
18:24:13 dansmith oh, okay
18:24:16 dansmith s-dean: well, you went down a bad path
18:24:32 openstackgerrit Matt Riedemann proposed openstack/nova master: deprecate ``wsgi_log_format`` config variable https://review.openstack.org/486623
18:24:33 openstackgerrit Matt Riedemann proposed openstack/nova master: request_log addition for running under uwsgi https://review.openstack.org/485602
18:24:34 s-dean haha, is my install screwed ?
18:24:34 dansmith s-dean: might want to start fresh if it's not too much trouble and avoid any of that cellsv1 stuff, depending
18:24:43 s-dean ok i will od
18:24:45 s-dean do
18:24:54 mriedem fear leads to anger, anger leads to hate, hate leads to cells v1
18:24:55 dansmith it's definitely rescue-able, but if you're trying to nail down a procedure it'll be confusing from here
18:25:04 dansmith cellsv1 leads to?
18:25:13 mriedem showing up in nova freenode irc?
18:25:15 openstackgerrit Ken'ichi Ohmichi proposed openstack/nova master: Remove the useless FakeExt https://review.openstack.org/486415
18:25:21 s-dean xD
18:25:26 mriedem which is equivalent to the dark side
18:25:38 openstackgerrit Ken'ichi Ohmichi proposed openstack/nova master: Remove the useless extension block_device_mapping_v1 object https://review.openstack.org/486069
18:25:49 openstackgerrit Ken'ichi Ohmichi proposed openstack/nova master: Remove the useless fake ExtensionManager from API unittests https://review.openstack.org/486416
18:25:52 s-dean awww man wish i came here sooner, i dont like asking for help i prefer to bash it out, and use google
18:25:58 openstackgerrit Ken'ichi Ohmichi proposed openstack/nova master: Move the note about '/os-volume_boot' to the correct place https://review.openstack.org/486071
18:27:04 mriedem sdague: requestlog thing is approved
18:27:13 mriedem sdague: you want to skim over this hot little number? https://review.openstack.org/#/c/485435/
18:27:20 openstackgerrit Ed Leafe proposed openstack/nova master: Migrate Ironic Flavors https://review.openstack.org/484949
18:35:51 dansmith jaypipes: edleafe: I was thinking that the ironic node class was required now.. do we really need to concern ourselves with that not being set?
18:35:58 s-dean shouldnt that config option be removed from the docs ?
18:36:03 s-dean https://docs.openstack.org/ocata/config-reference/compute/cells.html
18:36:13 dansmith s-dean: we can't yet
18:36:27 dansmith s-dean: people still run that stuff and have to for a bit longer
18:36:55 s-dean a fair enough
18:42:22 mriedem s-dean: we're in the process of moving those docs into nova, at which point we can put a big fat warning at the top saying that's for cells v1
18:43:04 edleafe jaypipes: regarding your comments about splitting the patch for https://review.openstack.org/#/c/486215/, do you want me to build the list of alternates, but continue to only return the selected hosts? And then change all the method signatures in https://review.openstack.org/#/c/486253/ ?
18:43:57 edleafe dansmith: I asked dtantsur|afk about that. He said that while adding the resource_class attribute was required, populating it completely was not
18:44:27 dansmith edleafe: okay so, we should check it and if it's not present, log a warning and bail?
18:45:38 edleafe dansmith: if it's not present, just skip
18:46:26 dansmith edleafe: we have to log a warning, since not having it set will lead to brokenness later, but.. your code currently is not skipping right?
18:46:47 dansmith in PS10 I mean
18:47:26 edleafe dansmith: correct. Want me to add some safety stuff there?
18:47:57 dansmith edleafe: yeah, warn and skip if unset, test for those cases and then I think we can land this and argue about the batching after
18:48:38 s-dean mriedem: that would be brilliant. before i head home for the night, can i specify in keystone, glance and nova their database connections to be localhost skipping the need for TLS, or do they need to be specified with a network address like in the docs? do the other service such as neutron and and nova-compute need to access the DB over the network?

Earlier   Later