Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-02-07
10:19:57 mdbooth Should we be logging authentication tokens?
10:19:59 mdbooth e.g.:
10:20:10 mdbooth application/json" -H "X-Auth-Token: {SHA1}dfc5e07945c13b29514dd78d29b374cf2e5d3a1f" _http_log_request /usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py:347
10:20:10 mdbooth 2017-02-06 16:37:53.021 [n-api.log] 29609 DEBUG cinderclient.v2.client [req-f47e58e1-65a9-44b3-b9ea-90959d0255ab tempest-TestVolumeSwap-787392718 tempest-TestVolumeSwap-787392718] REQ: curl -g -i -X GET http://10.24.164.111:8776/v2/72077d0aec734621ad21d12d3ffbc0e9/volumes/b4c90b8c-fac7-4bfd-b4be-a641ac137a1c -H "X-Service-Token: {SHA1}161c3fe7af35615c3f7a8136153d893316c4afa5" -H "User-Agent: python-cinderclient" -H "Accept:
10:20:55 mdbooth What's the lifetime of one of those tokens?
10:20:58 markus_z bauzas: Do you have a minute?
10:21:05 markus_z bauzas: It's about https://review.openstack.org/#/c/420079/1
10:21:10 bauzas markus_z: sure, but I'm a bit off this week
10:21:20 rpodolyaka mdbooth: we specifically store a hash of a token, so that it (i.e. a valid token) does not show up in logs, don?
10:21:25 rpodolyaka * don't we
10:21:49 bauzas markus_z: mmm, I should abandon this change
10:21:50 mdbooth rpodolyaka: Well the above it a cut/paste from my logs
10:21:51 rpodolyaka but you still can track usage of a particular token in logs
10:21:58 rpodolyaka but it's a hash
10:22:00 rpodolyaka not a token itself
10:22:07 rpodolyaka or I am missing something obvious? :)
10:22:14 markus_z bauzas: I was wondering about the alternatives to this change.
10:22:19 mdbooth rpodolyaka: I'm wondering the same thing :)
10:22:32 mdbooth If it's the thing we send to cinder, does it matter if it's a hash or not?
10:22:45 mdbooth Does it, eg, include a nonce component so it changes every time?
10:22:50 markus_z bauzas: Our s390 CI is broken because of "No hosts found to map to cell, exiting." and I don't find a solution for that.
10:22:58 rpodolyaka mdbooth: it's not what goes over the wire
10:23:09 rpodolyaka it's just the way it's shown in logs
10:23:25 rpodolyaka i.e. a sha1 hash
10:23:42 mdbooth rpodolyaka: Ah... so that's logging an sha1 of what went over the wire?
10:23:45 openstack Launchpad bug 1656276 in OpenStack Compute (nova) "Error running nova-manage cell_v2 simple_cell_setup when configuring nova with puppet-nova" [Undecided,In progress] - Assigned to Sylvain Bauza (sylvain-bauza)
10:23:45 bauzas markus_z: see the bug https://bugs.launchpad.net/nova/+bug/1656276
10:23:53 rpodolyaka mdbooth: yep
10:24:02 bauzas markus_z: that's the consensus now https://git.openstack.org/cgit/openstack/puppet-nova/commit/?id=dc2f3a358663c0fb97795666b16dec63cfcc3872
10:24:03 mdbooth Are you sure about that? Seems kinda weird we wouldn't just elide it.
10:24:42 mdbooth In other places we replace passwords with ***
10:25:08 rpodolyaka mdbooth: strace of `nova list` http://paste.openstack.org/raw/597914/
10:25:19 rpodolyaka well, the relevant part of it :)
10:25:49 mdbooth rpodolyaka: Cool, that doesn't look like an sha1 :)
10:25:53 mdbooth rpodolyaka: Thanks
10:25:56 rpodolyaka np!
10:27:14 bauzas markus_z: also http://docs.openstack.org/developer/nova/cells.html#first-time-setup now differs from http://docs.openstack.org/developer/nova/cells.html#upgrade-minimal
10:27:21 markus_z bauzas: We see that issue when we setup devstack (with devstack-gate). I'm not sure how that change applies to that situation.
10:28:24 openstackgerrit Bob Ball proposed openstack/nova master: XenAPI: fix memory usage miscalculation https://review.openstack.org/394857
10:28:40 bauzas markus_z: tbc, unless you're upgrading a newton cloud, you now need to stop using single_cell_setup and rather use map_cell0, then discover_hosts()
10:31:11 markus_z bauzas: Do I need to configure that in Devstacks local.conf? As said, it's for our CI.
10:34:46 bauzas markus_z: what do you mean by local.conf ?
10:35:03 bauzas markus_z: you don't really need to modify your workflow
10:35:20 bauzas if you're using devstack, it starts n-cpu because running simple_cell_setup AFAIK
10:35:31 bauzas which is then all good
10:35:49 bauzas the problem here is when you start n-cpu after running simple_cell_setup
10:36:09 bauzas then that call considers finding no hosts is an error
10:38:48 markus_z bauzas: To make sure I understand it: This call is the correct one https://github.com/openstack-dev/devstack/blob/bd14f7f23e01968aba70f0025b85dc15f110abc1/lib/nova#L952
10:38:51 markus_z ?
10:40:00 markus_z And then nova-manage "discover_hosts" is called? https://github.com/openstack/nova/blob/9eb7bcea05ce2447496f6561caa1ae46b800f85d/nova/cmd/manage.py#L1379-L1379
10:41:17 bauzas discover_hosts is when you add a compute *after* doing the cellsv2 setup
10:41:24 bauzas markus_z: ^
10:42:02 bauzas for the moment, there is no auto-registration of hosts in the cellsv2 world, but we need their mappings to order to place instances onto them
10:44:52 markus_z bauzas: I still don't get it :/ At the end of January there has to be a change which introduced something in that area which our CI isn't prepared to deal with.
10:46:51 markus_z bauzas: Do I understand it correctly that "n-cpu" has to run *before* simple_cell_setup is called?
10:47:42 bauzas markus_z: leave me 10 mins and we could hangout if you wish, okay
10:47:43 bauzas ?
10:49:11 markus_z bauzas: Let me dig more into the issue on my side before I'm consuming you time, but thanks a lot for the offer.
10:56:11 johnthetubaguy markus_z: did you see the new docs that just merged by the way
10:56:20 johnthetubaguy markus_z: they explain things a lot better
10:57:02 johnthetubaguy markus_z: the thing that confused me is that "db sync" created cell0, only after you first did the create_cell0 thing, it could be a different thing you hit
10:57:28 mdbooth lyarwood: Hey, so detach_volume races with delete()
10:57:42 tdurakov johnthetubaguy: hi, which docs?
10:58:21 mdbooth i.e. if you detach_volume(), delete() really quickly, the delete will do the detach which is already in progress
10:58:28 johnthetubaguy tdurakov: https://github.com/openstack/nova/commit/da591bc543c84fb601e5d832ba6a28e38c036901
10:58:36 mdbooth lyarwood: Can you think of any bad things this might cause on the cinder side?
10:58:40 tdurakov previous time I've deployed multinode devstack, I need to run nova-manage cell_v2 simple_cell_setup after second compute is deployed
10:58:43 mdbooth johnthetubaguy: Same question ^^^^ ?
10:59:02 tdurakov today I've deployed new one, and it doesn't recognise second compute is added
10:59:11 tdurakov thanks for the link!
10:59:30 lyarwood mdbooth: about to jump on a call
10:59:34 johnthetubaguy tdurakov: markus_z: there is an ordering issue I suspect. You need to have started n-cpu to be able to discover it
10:59:37 mdbooth lyarwood: k
10:59:44 lyarwood mdbooth: I can't think of anything bad on the cinder cide
10:59:58 lyarwood mdbooth: but that is going to screw up host state and leave things hanging around with iSCSI/FC etc
11:00:18 lyarwood mdbooth: ah no wait we have locks around disconnect_volume
11:00:19 mdbooth Well it's going do to do 2 detaches
11:00:20 tdurakov johnthetubaguy: hm... I'm not sure how this should work
11:00:26 mdbooth So it's that going to extra cleanup?
11:00:27 johnthetubaguy mdbooth: lyarwood: unsure on cinder, I suspect its OK
11:00:49 lyarwood mdbooth: double terminate_connection calls should be fine as long as something has disconnected the volume from the host
11:00:49 tdurakov local.conf on compute only node already has endpoints of controller node
11:01:03 tdurakov so how I should reorder them properly
11:01:13 mdbooth lyarwood: Thanks
11:01:13 johnthetubaguy tdurakov: like I say, the docs do try cover that now, including the nova-manage man pages: https://github.com/openstack/nova/blob/5dad7eaef7f8562425cce6b233aed610ca2d3148/doc/source/man/nova-manage.rst
11:01:28 johnthetubaguy tdurakov: I would talk you through it, but I would like to test the docs :)
11:01:46 tdurakov johnthetubaguy: that's ok
11:01:50 tdurakov thank you
11:02:08 johnthetubaguy tdurakov: now worries, this is the rendered docs: http://docs.openstack.org/developer/nova/cells.html#step-by-step-for-common-use-cases
11:02:23 johnthetubaguy markus_z: http://docs.openstack.org/developer/nova/cells.html#step-by-step-for-common-use-cases and https://github.com/openstack/nova/blob/5dad7eaef7f8562425cce6b233aed610ca2d3148/doc/source/man/nova-manage.rst should be your friend
11:03:44 mdbooth Ah... attach/delete would be worse, which is probably why that takes an instance lock
11:04:47 johnthetubaguy mdbooth: I am wondering about the end API state vs what we have on the host, its not clear what we end up with
11:42:33 openstackgerrit Matthew Booth proposed openstack/nova master: Fix various races with detach_volume https://review.openstack.org/430213
11:42:56 openstackgerrit Sergey Nikitin proposed openstack/nova master: TrivialFix: replace list comprehension with 'for' https://review.openstack.org/430214
11:51:53 markus_z johnthetubaguy: thanks, I haven't yet read the new docs, I'll do that today.
11:52:38 johnthetubaguy markus_z: no worries, hopefully it should help
11:52:48 johnthetubaguy markus_z: if not we should fix the docs :)
11:58:39 openstackgerrit John Garbutt proposed openstack/nova master: WIP: Remove stuck live-migration detection https://review.openstack.org/430218
12:00:26 johnthetubaguy pkoniszewski: kick started the discussion around that bug in the above patch^
12:01:22 mdbooth johnthetubaguy: Haven't looked at what it does yet, but +87, -163 looks good :)

Earlier   Later