| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-02-07 | |||
| 09:40:03 | gcb | johnthetubaguy, I added gate-nova-tox-db-functional-py35-ubuntu-xenial as non-voting, maybe we can make it votable | |
| 09:40:25 | johnthetubaguy | gcb: I think that would be a good thing to try | |
| 09:41:10 | johnthetubaguy | gcb: you can always to a tempest skip list, like we have for cellsv1: https://github.com/openstack/nova/blob/master/devstack/tempest-dsvm-cells-rc | |
| 09:41:40 | johnthetubaguy | i.e. get a job that passes creating at least one instance, then slowly enable more goodness as the fixes go in | |
| 09:42:19 | gcb | I will be back after dinner :) | |
| 09:44:45 | openstackgerrit | John Garbutt proposed openstack/nova-specs master: WIP: Spec idea on the future of quotas for Nova https://review.openstack.org/429678 | |
| 10:05:21 | lyarwood | mdbooth: https://review.openstack.org/#/c/427364/ - I added the comments you requested btw, any chance of a quick review if you have time this morning? | |
| 10:05:56 | mdbooth | lyarwood: Got my head in something right now. Is there a chance of follow-on core review? | |
| 10:06:19 | mdbooth | (iow: will my review achieve anything this morning) | |
| 10:07:25 | lyarwood | mdbooth: ah, np, I was going to ask around now that we are past RC but don't go out of your way if you're in the middle of something | |
| 10:19:57 | mdbooth | Should we be logging authentication tokens? | |
| 10:19:59 | mdbooth | e.g.: | |
| 10:20:10 | mdbooth | application/json" -H "X-Auth-Token: {SHA1}dfc5e07945c13b29514dd78d29b374cf2e5d3a1f" _http_log_request /usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py:347 | |
| 10:20:10 | mdbooth | 2017-02-06 16:37:53.021 [n-api.log] 29609 DEBUG cinderclient.v2.client [req-f47e58e1-65a9-44b3-b9ea-90959d0255ab tempest-TestVolumeSwap-787392718 tempest-TestVolumeSwap-787392718] REQ: curl -g -i -X GET http://10.24.164.111:8776/v2/72077d0aec734621ad21d12d3ffbc0e9/volumes/b4c90b8c-fac7-4bfd-b4be-a641ac137a1c -H "X-Service-Token: {SHA1}161c3fe7af35615c3f7a8136153d893316c4afa5" -H "User-Agent: python-cinderclient" -H "Accept: | |
| 10:20:55 | mdbooth | What's the lifetime of one of those tokens? | |
| 10:20:58 | markus_z | bauzas: Do you have a minute? | |
| 10:21:05 | markus_z | bauzas: It's about https://review.openstack.org/#/c/420079/1 | |
| 10:21:10 | bauzas | markus_z: sure, but I'm a bit off this week | |
| 10:21:20 | rpodolyaka | mdbooth: we specifically store a hash of a token, so that it (i.e. a valid token) does not show up in logs, don? | |
| 10:21:25 | rpodolyaka | * don't we | |
| 10:21:49 | bauzas | markus_z: mmm, I should abandon this change | |
| 10:21:50 | mdbooth | rpodolyaka: Well the above it a cut/paste from my logs | |
| 10:21:51 | rpodolyaka | but you still can track usage of a particular token in logs | |
| 10:21:58 | rpodolyaka | but it's a hash | |
| 10:22:00 | rpodolyaka | not a token itself | |
| 10:22:07 | rpodolyaka | or I am missing something obvious? :) | |
| 10:22:14 | markus_z | bauzas: I was wondering about the alternatives to this change. | |
| 10:22:19 | mdbooth | rpodolyaka: I'm wondering the same thing :) | |
| 10:22:32 | mdbooth | If it's the thing we send to cinder, does it matter if it's a hash or not? | |
| 10:22:45 | mdbooth | Does it, eg, include a nonce component so it changes every time? | |
| 10:22:50 | markus_z | bauzas: Our s390 CI is broken because of "No hosts found to map to cell, exiting." and I don't find a solution for that. | |
| 10:22:58 | rpodolyaka | mdbooth: it's not what goes over the wire | |
| 10:23:09 | rpodolyaka | it's just the way it's shown in logs | |
| 10:23:25 | rpodolyaka | i.e. a sha1 hash | |
| 10:23:42 | mdbooth | rpodolyaka: Ah... so that's logging an sha1 of what went over the wire? | |
| 10:23:45 | openstack | Launchpad bug 1656276 in OpenStack Compute (nova) "Error running nova-manage cell_v2 simple_cell_setup when configuring nova with puppet-nova" [Undecided,In progress] - Assigned to Sylvain Bauza (sylvain-bauza) | |
| 10:23:45 | bauzas | markus_z: see the bug https://bugs.launchpad.net/nova/+bug/1656276 | |
| 10:23:53 | rpodolyaka | mdbooth: yep | |
| 10:24:02 | bauzas | markus_z: that's the consensus now https://git.openstack.org/cgit/openstack/puppet-nova/commit/?id=dc2f3a358663c0fb97795666b16dec63cfcc3872 | |
| 10:24:03 | mdbooth | Are you sure about that? Seems kinda weird we wouldn't just elide it. | |
| 10:24:42 | mdbooth | In other places we replace passwords with *** | |
| 10:25:08 | rpodolyaka | mdbooth: strace of `nova list` http://paste.openstack.org/raw/597914/ | |
| 10:25:19 | rpodolyaka | well, the relevant part of it :) | |
| 10:25:49 | mdbooth | rpodolyaka: Cool, that doesn't look like an sha1 :) | |
| 10:25:53 | mdbooth | rpodolyaka: Thanks | |
| 10:25:56 | rpodolyaka | np! | |
| 10:27:14 | bauzas | markus_z: also http://docs.openstack.org/developer/nova/cells.html#first-time-setup now differs from http://docs.openstack.org/developer/nova/cells.html#upgrade-minimal | |
| 10:27:21 | markus_z | bauzas: We see that issue when we setup devstack (with devstack-gate). I'm not sure how that change applies to that situation. | |
| 10:28:24 | openstackgerrit | Bob Ball proposed openstack/nova master: XenAPI: fix memory usage miscalculation https://review.openstack.org/394857 | |
| 10:28:40 | bauzas | markus_z: tbc, unless you're upgrading a newton cloud, you now need to stop using single_cell_setup and rather use map_cell0, then discover_hosts() | |
| 10:31:11 | markus_z | bauzas: Do I need to configure that in Devstacks local.conf? As said, it's for our CI. | |
| 10:34:46 | bauzas | markus_z: what do you mean by local.conf ? | |
| 10:35:03 | bauzas | markus_z: you don't really need to modify your workflow | |
| 10:35:20 | bauzas | if you're using devstack, it starts n-cpu because running simple_cell_setup AFAIK | |
| 10:35:31 | bauzas | which is then all good | |
| 10:35:49 | bauzas | the problem here is when you start n-cpu after running simple_cell_setup | |
| 10:36:09 | bauzas | then that call considers finding no hosts is an error | |
| 10:38:48 | markus_z | bauzas: To make sure I understand it: This call is the correct one https://github.com/openstack-dev/devstack/blob/bd14f7f23e01968aba70f0025b85dc15f110abc1/lib/nova#L952 | |
| 10:38:51 | markus_z | ? | |
| 10:40:00 | markus_z | And then nova-manage "discover_hosts" is called? https://github.com/openstack/nova/blob/9eb7bcea05ce2447496f6561caa1ae46b800f85d/nova/cmd/manage.py#L1379-L1379 | |
| 10:41:17 | bauzas | discover_hosts is when you add a compute *after* doing the cellsv2 setup | |
| 10:41:24 | bauzas | markus_z: ^ | |
| 10:42:02 | bauzas | for the moment, there is no auto-registration of hosts in the cellsv2 world, but we need their mappings to order to place instances onto them | |
| 10:44:52 | markus_z | bauzas: I still don't get it :/ At the end of January there has to be a change which introduced something in that area which our CI isn't prepared to deal with. | |
| 10:46:51 | markus_z | bauzas: Do I understand it correctly that "n-cpu" has to run *before* simple_cell_setup is called? | |
| 10:47:42 | bauzas | markus_z: leave me 10 mins and we could hangout if you wish, okay | |
| 10:47:43 | bauzas | ? | |
| 10:49:11 | markus_z | bauzas: Let me dig more into the issue on my side before I'm consuming you time, but thanks a lot for the offer. | |
| 10:56:11 | johnthetubaguy | markus_z: did you see the new docs that just merged by the way | |
| 10:56:20 | johnthetubaguy | markus_z: they explain things a lot better | |
| 10:57:02 | johnthetubaguy | markus_z: the thing that confused me is that "db sync" created cell0, only after you first did the create_cell0 thing, it could be a different thing you hit | |
| 10:57:28 | mdbooth | lyarwood: Hey, so detach_volume races with delete() | |
| 10:57:42 | tdurakov | johnthetubaguy: hi, which docs? | |
| 10:58:21 | mdbooth | i.e. if you detach_volume(), delete() really quickly, the delete will do the detach which is already in progress | |
| 10:58:28 | johnthetubaguy | tdurakov: https://github.com/openstack/nova/commit/da591bc543c84fb601e5d832ba6a28e38c036901 | |
| 10:58:36 | mdbooth | lyarwood: Can you think of any bad things this might cause on the cinder side? | |
| 10:58:40 | tdurakov | previous time I've deployed multinode devstack, I need to run nova-manage cell_v2 simple_cell_setup after second compute is deployed | |
| 10:58:43 | mdbooth | johnthetubaguy: Same question ^^^^ ? | |
| 10:59:02 | tdurakov | today I've deployed new one, and it doesn't recognise second compute is added | |
| 10:59:11 | tdurakov | thanks for the link! | |
| 10:59:30 | lyarwood | mdbooth: about to jump on a call | |
| 10:59:34 | johnthetubaguy | tdurakov: markus_z: there is an ordering issue I suspect. You need to have started n-cpu to be able to discover it | |
| 10:59:37 | mdbooth | lyarwood: k | |
| 10:59:44 | lyarwood | mdbooth: I can't think of anything bad on the cinder cide | |
| 10:59:58 | lyarwood | mdbooth: but that is going to screw up host state and leave things hanging around with iSCSI/FC etc | |
| 11:00:18 | lyarwood | mdbooth: ah no wait we have locks around disconnect_volume | |
| 11:00:19 | mdbooth | Well it's going do to do 2 detaches | |
| 11:00:20 | tdurakov | johnthetubaguy: hm... I'm not sure how this should work | |
| 11:00:26 | mdbooth | So it's that going to extra cleanup? | |
| 11:00:27 | johnthetubaguy | mdbooth: lyarwood: unsure on cinder, I suspect its OK | |
| 11:00:49 | lyarwood | mdbooth: double terminate_connection calls should be fine as long as something has disconnected the volume from the host | |
| 11:00:49 | tdurakov | local.conf on compute only node already has endpoints of controller node | |
| 11:01:03 | tdurakov | so how I should reorder them properly | |
| 11:01:13 | mdbooth | lyarwood: Thanks | |
| 11:01:13 | johnthetubaguy | tdurakov: like I say, the docs do try cover that now, including the nova-manage man pages: https://github.com/openstack/nova/blob/5dad7eaef7f8562425cce6b233aed610ca2d3148/doc/source/man/nova-manage.rst | |
| 11:01:28 | johnthetubaguy | tdurakov: I would talk you through it, but I would like to test the docs :) | |
| 11:01:46 | tdurakov | johnthetubaguy: that's ok | |
| 11:01:50 | tdurakov | thank you | |
| 11:02:08 | johnthetubaguy | tdurakov: now worries, this is the rendered docs: http://docs.openstack.org/developer/nova/cells.html#step-by-step-for-common-use-cases | |
| 11:02:23 | johnthetubaguy | markus_z: http://docs.openstack.org/developer/nova/cells.html#step-by-step-for-common-use-cases and https://github.com/openstack/nova/blob/5dad7eaef7f8562425cce6b233aed610ca2d3148/doc/source/man/nova-manage.rst should be your friend | |