Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-07-24
16:09:01 dansmith edleafe: the move from eventlet is purely a nova-api thing for wsgi stuff, fwiw
16:10:03 openstackgerrit Chris Friesen proposed openstack/nova master: Add validation for cpu_realtime_mask https://review.openstack.org/468203
16:10:04 openstackgerrit Chris Friesen proposed openstack/nova master: Tweak the cpu_realtime_mask handling slightly https://review.openstack.org/461456
16:14:44 mdbooth kashyap: https://review.openstack.org/#/c/485752/ Can you remind me where we rewrite the guest xml post migration?
16:15:44 mdbooth kashyap: NM, I didn't read your commit message carefully enough
16:27:48 openstackgerrit melanie witt proposed openstack/nova master: Add online migration to move quotas to API database https://review.openstack.org/410946
16:29:13 melwitt mriedem: your comments addressed ^
16:31:05 s-dean mriedem: https://bugs.launchpad.net/nova/+bug/1706118
16:31:06 openstack Launchpad bug 1706118 in OpenStack Compute (nova) "Adding TLS Connection String To Database Connection Causes malformed URL in cell_mappings DB" [Undecided,New]
16:32:38 edleafe dansmith: still not clear how nodes could be batched in the ironic driver. Wouldn't that require some way of persisting what has been done already?
16:47:37 melwitt mriedem: I went to change the counting quotas blueprint to Implemented and it automatically changed it to Complete (I didn't know it would do that) https://blueprints.launchpad.net/nova/+spec/cells-count-resources-to-check-quota-in-api
16:48:12 melwitt sorry about that
16:50:13 melwitt oh, it let me change it back
16:50:46 melwitt phew.
16:52:53 dansmith edleafe: by batching I mean do 25 at a time, in that thread
16:53:20 dansmith edleafe: like for i in range(0, len(uuids), 25): do_instances(uuids[i:i+25])
16:56:24 edleafe batching which? I'm going through the nodes 1 by 1, and creating the instances one at a time
16:56:53 edleafe I thought that doing it that way would satisfy your memory explosion concern
16:58:42 dansmith edleafe: it does, but now we're hammering the database and conductor with one-by-one queries on every single boot
17:16:37 melwitt dansmith, mriedem: I was wondering whether we're gonna try to get this list instances across cells merge done this week? https://review.openstack.org/#/c/463618/
17:17:03 dansmith melwitt: I talked to mdbooth last week and it sounded like it wasn't going to happen
17:17:15 dansmith melwitt: especially since to do it right I think we'll need some refactoring of our db layer
17:17:49 openstackgerrit Sean Dague proposed openstack/nova master: Suppress some test warnings https://review.openstack.org/486700
17:18:10 melwitt dansmith: oh. I took a quick look at the patch but wasn't sure the detail of what's left to do with it and whether one of us can finish it up
17:18:32 melwitt one of us could be me, I mean
17:18:38 dansmith well, we could, but I don't think it's a reasonable thing to throw in at the last minute
17:19:14 melwitt k
17:21:41 openstackgerrit Chris Friesen proposed openstack/nova master: Ensure we unshelve in the cell the instance is mapped https://review.openstack.org/486208
17:22:29 cfriesen I added some assertions to existing tests here ^....if you want entirely new tests I can do that, but I wasn't sure it was worth the overhead.
17:35:01 s-dean hi, any pointers on how i may solve this issue ? MessagingTimeout: Timed out waiting for a reply to message ID
17:35:17 s-dean its telling me to submit a bug report
17:35:33 s-dean sorry not it> Nova*
17:36:27 s-dean nova.api.openstack.extensions [req-3ed5c135-64bd-4c39-934d-3d3d7a32a6b5 14d1055c2e2d4f0eab6739671b2688a8 cf7b827f55de4406b9325ab3505b973d - default default] Unexpected exception in API method
17:39:38 melwitt s-dean: I think that's an RPC timeout, so you will want to check your transport_url for your cell1 and make sure it's correct
17:40:07 s-dean ok cheers will do that now
17:49:09 s-dean yeah, cell1's RPC mapping was wrong, i have changed it and restarted all nova services, no dice :/
17:49:19 s-dean still same error
17:50:17 s-dean can i ask would it be better to implement SSL/TLS after successfully setting up openstack unencrypted ?
17:50:27 dansmith s-dean: I'm not following along, but you're sure that the cell1 mapping points to the same rpc endpoint that your other services do?
17:51:51 melwitt s-dean: did you change it using "nova-manage cell_v2 update_cell"? as shown https://docs.openstack.org/nova/latest/cli/nova-manage.html#man-page-cells-v2
17:51:54 mriedem melwitt: i think https://blueprints.launchpad.net/nova/+spec/cells-count-resources-to-check-quota-in-api is done
17:52:05 mriedem https://review.openstack.org/#/c/446243/ was the last change to merge
17:52:24 s-dean no i did not
17:52:59 melwitt s-dean: yeah you need to use the nova-manage commands to get it set right in the database. you can use "nova-manage cell_v2 list_cells --verbose" to see what you currently have
17:53:12 s-dean ok cheers
17:53:22 melwitt these are stored in the database now because the API needs to know how to contact each of the cells you have
17:53:58 melwitt mriedem: agreed, I just didn't mean to set the Completed state because I think usually you do that
17:54:12 melwitt but it does it automatically if you put Implemented
17:55:05 mriedem melwitt: i like to yeah because i have a spreadsheet keeping track of when things are closed out
17:55:08 mriedem b/c i'm a nerd
17:55:35 melwitt mriedem: heh. I would probably do the same thing. don't want to mess up your system
17:56:01 mriedem just please be sure to put scissors, pens and dusters back where you got them if you're ever at my house
17:56:13 mriedem and windex
17:56:17 melwitt lol
18:00:05 s-dean do ineed to sync the db's after running that update command
18:00:32 melwitt s-dean: no, just need to restart your nova-api services I think
18:00:52 dansmith unless he never properly sync'd it as a result
18:01:15 s-dean yeah, still timing out
18:01:36 melwitt hm
18:02:14 openstackgerrit Matt Riedemann proposed openstack/nova master: Provide hints when nova-manage db sync fails to sync cell0 https://review.openstack.org/486660
18:03:18 dansmith s-dean: so if you grep transport_url out of your configs on all nodes, they match exactly what is in the db for the cell1 transport_url?
18:06:35 s-dean ok so i can confirm, that on my controller node the URL's are correct, however on the compute node it is not because of: https://docs.openstack.org/security-guide/messaging/security.html
18:07:32 mriedem sdague: did you ever report a bug for the request log thing with uwsgi?
18:07:52 sdague mriedem: I don't think I did actually
18:10:07 mriedem sdague: ok, did the same thing need to be done in other projects like cinder?
18:10:18 mriedem if so, could be good to track with a bug for those other projects too
18:13:50 dansmith s-dean: so...the compute node is not correct?
18:14:00 s-dean does the username need to be consistent across all nodes, or is having different users for each compute node ok ?
18:14:13 s-dean yes i gave the compute node its on users
18:14:15 s-dean user
18:14:23 s-dean and assigned it all permissions
18:14:31 s-dean own*
18:14:59 dansmith s-dean: they can be different, they just all have to point at the same vhost of course
18:15:12 s-dean in my case my vhost is /
18:15:31 dansmith as long as that is the same for all the nodes then that's fine
18:16:34 dansmith s-dean: aside from that, we need more context on which thing is failing, so more detailed logs, and a sanitized dump of your config and cell_mappings table are needed
18:17:09 s-dean paste bin urls ok ?
18:17:19 dansmith yeah
18:18:55 s-dean https://pastebin.com/WEu7JCWW
18:20:20 dansmith s-dean: are you running n-net?
18:20:36 dansmith s-dean: and looks like you're configured for cellsv1, is that intentional or did you think that was related?
18:20:57 dansmith because this timeout is trying to call the cellsv1 service, which if you're not running it, will always time out :)
18:21:55 s-dean oh ok
18:22:06 s-dean i didnt know i was running cells v1
18:22:24 s-dean is this because of the following config line compute_api_class=nova.compute.cells_api.ComputeCellsAPI
18:22:32 s-dean my config is here
18:22:32 s-dean https://bugs.launchpad.net/nova/+bug/1706118
18:22:33 dansmith s-dean: you definitely know if you're running cellsv1 because it's a huge friggin nightmare,
18:22:34 openstack Launchpad bug 1706118 in OpenStack Compute (nova) ocata "Adding TLS Connection String To Database Connection Causes malformed URL in cell_mappings DB" [High,Confirmed]
18:22:44 dansmith so I'm guessing maybe you configured that while trying to resolve this problem?
18:23:00 s-dean yeah i have been poking about quite a bit
18:23:18 dansmith s-dean: this is an upgrade from a previously working deployment, right?
18:23:57 s-dean no fresh install, but because of that DB error, i have been trying to debug and find the right settings
18:24:13 dansmith oh, okay
18:24:16 dansmith s-dean: well, you went down a bad path
18:24:32 openstackgerrit Matt Riedemann proposed openstack/nova master: deprecate ``wsgi_log_format`` config variable https://review.openstack.org/486623
18:24:33 openstackgerrit Matt Riedemann proposed openstack/nova master: request_log addition for running under uwsgi https://review.openstack.org/485602
18:24:34 dansmith s-dean: might want to start fresh if it's not too much trouble and avoid any of that cellsv1 stuff, depending
18:24:34 s-dean haha, is my install screwed ?
18:24:43 s-dean ok i will od

Earlier   Later