| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-07-24 | |||
| 16:29:13 | melwitt | mriedem: your comments addressed ^ | |
| 16:31:05 | s-dean | mriedem: https://bugs.launchpad.net/nova/+bug/1706118 | |
| 16:31:06 | openstack | Launchpad bug 1706118 in OpenStack Compute (nova) "Adding TLS Connection String To Database Connection Causes malformed URL in cell_mappings DB" [Undecided,New] | |
| 16:32:38 | edleafe | dansmith: still not clear how nodes could be batched in the ironic driver. Wouldn't that require some way of persisting what has been done already? | |
| 16:47:37 | melwitt | mriedem: I went to change the counting quotas blueprint to Implemented and it automatically changed it to Complete (I didn't know it would do that) https://blueprints.launchpad.net/nova/+spec/cells-count-resources-to-check-quota-in-api | |
| 16:48:12 | melwitt | sorry about that | |
| 16:50:13 | melwitt | oh, it let me change it back | |
| 16:50:46 | melwitt | phew. | |
| 16:52:53 | dansmith | edleafe: by batching I mean do 25 at a time, in that thread | |
| 16:53:20 | dansmith | edleafe: like for i in range(0, len(uuids), 25): do_instances(uuids[i:i+25]) | |
| 16:56:24 | edleafe | batching which? I'm going through the nodes 1 by 1, and creating the instances one at a time | |
| 16:56:53 | edleafe | I thought that doing it that way would satisfy your memory explosion concern | |
| 16:58:42 | dansmith | edleafe: it does, but now we're hammering the database and conductor with one-by-one queries on every single boot | |
| 17:16:37 | melwitt | dansmith, mriedem: I was wondering whether we're gonna try to get this list instances across cells merge done this week? https://review.openstack.org/#/c/463618/ | |
| 17:17:03 | dansmith | melwitt: I talked to mdbooth last week and it sounded like it wasn't going to happen | |
| 17:17:15 | dansmith | melwitt: especially since to do it right I think we'll need some refactoring of our db layer | |
| 17:17:49 | openstackgerrit | Sean Dague proposed openstack/nova master: Suppress some test warnings https://review.openstack.org/486700 | |
| 17:18:10 | melwitt | dansmith: oh. I took a quick look at the patch but wasn't sure the detail of what's left to do with it and whether one of us can finish it up | |
| 17:18:32 | melwitt | one of us could be me, I mean | |
| 17:18:38 | dansmith | well, we could, but I don't think it's a reasonable thing to throw in at the last minute | |
| 17:19:14 | melwitt | k | |
| 17:21:41 | openstackgerrit | Chris Friesen proposed openstack/nova master: Ensure we unshelve in the cell the instance is mapped https://review.openstack.org/486208 | |
| 17:22:29 | cfriesen | I added some assertions to existing tests here ^....if you want entirely new tests I can do that, but I wasn't sure it was worth the overhead. | |
| 17:35:01 | s-dean | hi, any pointers on how i may solve this issue ? MessagingTimeout: Timed out waiting for a reply to message ID | |
| 17:35:17 | s-dean | its telling me to submit a bug report | |
| 17:35:33 | s-dean | sorry not it> Nova* | |
| 17:36:27 | s-dean | nova.api.openstack.extensions [req-3ed5c135-64bd-4c39-934d-3d3d7a32a6b5 14d1055c2e2d4f0eab6739671b2688a8 cf7b827f55de4406b9325ab3505b973d - default default] Unexpected exception in API method | |
| 17:39:38 | melwitt | s-dean: I think that's an RPC timeout, so you will want to check your transport_url for your cell1 and make sure it's correct | |
| 17:40:07 | s-dean | ok cheers will do that now | |
| 17:49:09 | s-dean | yeah, cell1's RPC mapping was wrong, i have changed it and restarted all nova services, no dice :/ | |
| 17:49:19 | s-dean | still same error | |
| 17:50:17 | s-dean | can i ask would it be better to implement SSL/TLS after successfully setting up openstack unencrypted ? | |
| 17:50:27 | dansmith | s-dean: I'm not following along, but you're sure that the cell1 mapping points to the same rpc endpoint that your other services do? | |
| 17:51:51 | melwitt | s-dean: did you change it using "nova-manage cell_v2 update_cell"? as shown https://docs.openstack.org/nova/latest/cli/nova-manage.html#man-page-cells-v2 | |
| 17:51:54 | mriedem | melwitt: i think https://blueprints.launchpad.net/nova/+spec/cells-count-resources-to-check-quota-in-api is done | |
| 17:52:05 | mriedem | https://review.openstack.org/#/c/446243/ was the last change to merge | |
| 17:52:24 | s-dean | no i did not | |
| 17:52:59 | melwitt | s-dean: yeah you need to use the nova-manage commands to get it set right in the database. you can use "nova-manage cell_v2 list_cells --verbose" to see what you currently have | |
| 17:53:12 | s-dean | ok cheers | |
| 17:53:22 | melwitt | these are stored in the database now because the API needs to know how to contact each of the cells you have | |
| 17:53:58 | melwitt | mriedem: agreed, I just didn't mean to set the Completed state because I think usually you do that | |
| 17:54:12 | melwitt | but it does it automatically if you put Implemented | |
| 17:55:05 | mriedem | melwitt: i like to yeah because i have a spreadsheet keeping track of when things are closed out | |
| 17:55:08 | mriedem | b/c i'm a nerd | |
| 17:55:35 | melwitt | mriedem: heh. I would probably do the same thing. don't want to mess up your system | |
| 17:56:01 | mriedem | just please be sure to put scissors, pens and dusters back where you got them if you're ever at my house | |
| 17:56:13 | mriedem | and windex | |
| 17:56:17 | melwitt | lol | |
| 18:00:05 | s-dean | do ineed to sync the db's after running that update command | |
| 18:00:32 | melwitt | s-dean: no, just need to restart your nova-api services I think | |
| 18:00:52 | dansmith | unless he never properly sync'd it as a result | |
| 18:01:15 | s-dean | yeah, still timing out | |
| 18:01:36 | melwitt | hm | |
| 18:02:14 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Provide hints when nova-manage db sync fails to sync cell0 https://review.openstack.org/486660 | |
| 18:03:18 | dansmith | s-dean: so if you grep transport_url out of your configs on all nodes, they match exactly what is in the db for the cell1 transport_url? | |
| 18:06:35 | s-dean | ok so i can confirm, that on my controller node the URL's are correct, however on the compute node it is not because of: https://docs.openstack.org/security-guide/messaging/security.html | |
| 18:07:32 | mriedem | sdague: did you ever report a bug for the request log thing with uwsgi? | |
| 18:07:52 | sdague | mriedem: I don't think I did actually | |
| 18:10:07 | mriedem | sdague: ok, did the same thing need to be done in other projects like cinder? | |
| 18:10:18 | mriedem | if so, could be good to track with a bug for those other projects too | |
| 18:13:50 | dansmith | s-dean: so...the compute node is not correct? | |
| 18:14:00 | s-dean | does the username need to be consistent across all nodes, or is having different users for each compute node ok ? | |
| 18:14:13 | s-dean | yes i gave the compute node its on users | |
| 18:14:15 | s-dean | user | |
| 18:14:23 | s-dean | and assigned it all permissions | |
| 18:14:31 | s-dean | own* | |
| 18:14:59 | dansmith | s-dean: they can be different, they just all have to point at the same vhost of course | |
| 18:15:12 | s-dean | in my case my vhost is / | |
| 18:15:31 | dansmith | as long as that is the same for all the nodes then that's fine | |
| 18:16:34 | dansmith | s-dean: aside from that, we need more context on which thing is failing, so more detailed logs, and a sanitized dump of your config and cell_mappings table are needed | |
| 18:17:09 | s-dean | paste bin urls ok ? | |
| 18:17:19 | dansmith | yeah | |
| 18:18:55 | s-dean | https://pastebin.com/WEu7JCWW | |
| 18:20:20 | dansmith | s-dean: are you running n-net? | |
| 18:20:36 | dansmith | s-dean: and looks like you're configured for cellsv1, is that intentional or did you think that was related? | |
| 18:20:57 | dansmith | because this timeout is trying to call the cellsv1 service, which if you're not running it, will always time out :) | |
| 18:21:55 | s-dean | oh ok | |
| 18:22:06 | s-dean | i didnt know i was running cells v1 | |
| 18:22:24 | s-dean | is this because of the following config line compute_api_class=nova.compute.cells_api.ComputeCellsAPI | |
| 18:22:32 | s-dean | https://bugs.launchpad.net/nova/+bug/1706118 | |
| 18:22:32 | s-dean | my config is here | |
| 18:22:33 | dansmith | s-dean: you definitely know if you're running cellsv1 because it's a huge friggin nightmare, | |
| 18:22:34 | openstack | Launchpad bug 1706118 in OpenStack Compute (nova) ocata "Adding TLS Connection String To Database Connection Causes malformed URL in cell_mappings DB" [High,Confirmed] | |
| 18:22:44 | dansmith | so I'm guessing maybe you configured that while trying to resolve this problem? | |
| 18:23:00 | s-dean | yeah i have been poking about quite a bit | |
| 18:23:18 | dansmith | s-dean: this is an upgrade from a previously working deployment, right? | |
| 18:23:57 | s-dean | no fresh install, but because of that DB error, i have been trying to debug and find the right settings | |
| 18:24:13 | dansmith | oh, okay | |
| 18:24:16 | dansmith | s-dean: well, you went down a bad path | |
| 18:24:32 | openstackgerrit | Matt Riedemann proposed openstack/nova master: deprecate ``wsgi_log_format`` config variable https://review.openstack.org/486623 | |
| 18:24:33 | openstackgerrit | Matt Riedemann proposed openstack/nova master: request_log addition for running under uwsgi https://review.openstack.org/485602 | |
| 18:24:34 | s-dean | haha, is my install screwed ? | |
| 18:24:34 | dansmith | s-dean: might want to start fresh if it's not too much trouble and avoid any of that cellsv1 stuff, depending | |
| 18:24:43 | s-dean | ok i will od | |
| 18:24:45 | s-dean | do | |
| 18:24:54 | mriedem | fear leads to anger, anger leads to hate, hate leads to cells v1 | |
| 18:24:55 | dansmith | it's definitely rescue-able, but if you're trying to nail down a procedure it'll be confusing from here | |
| 18:25:04 | dansmith | cellsv1 leads to? | |
| 18:25:13 | mriedem | showing up in nova freenode irc? | |
| 18:25:15 | openstackgerrit | Ken'ichi Ohmichi proposed openstack/nova master: Remove the useless FakeExt https://review.openstack.org/486415 | |