| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-06-18 | |||
| 16:19:57 | sangeet | I have SSL enabled for keystone. My compute service fails to come up due to SSL 'certificate verify failed'. I am not sure if I am placing the certifcate at the correct place. Where should the certifciate go? I tried to palce it in /etc/nova/certs/ca.crt. it did not work. Then I set CA_CERTS, it still did not work. Any help will be highly apprecaited. I am running Stein | |
| 16:20:45 | sean-k-mooney | sangeet: you need to add it to the openerating systems certificat store | |
| 16:21:10 | sean-k-mooney | i dont think we support passing a ca directly to nova since now does not realy know anything about tls | |
| 16:21:55 | sangeet | nova-api is working fine for me. I placed the certs in /etc/nova/certs folder | |
| 16:22:46 | sean-k-mooney | is the api runnign on the host that generated teh cert | |
| 16:23:03 | sangeet | No | |
| 16:23:53 | sangeet | But I converted api to wsgi | |
| 16:23:55 | sean-k-mooney | the only ca config option i see are related to rabbitmq and we dont mention this in https://docs.openstack.org/nova/latest/admin/security.html | |
| 16:24:31 | sean-k-mooney | actully we have options for vendor data too | |
| 16:24:43 | sangeet | Issue is compuet is trying to get token from keystone and since keystone supports SSL it faile | |
| 16:25:42 | sean-k-mooney | sangeet: right but thats becasue you have not added the private ca to the operating systsms cert store | |
| 16:27:29 | sean-k-mooney | you should be adding the ca cert in /usr/local/share/ca-certificates/ and then do sudo update-ca-certificates | |
| 17:50:22 | sangeet | sean-k-mooney .. sorry had to run for an appointment. Should the cert be in /usr/local/share/ca-certificates/ or /var/lib/openstack/lib/python3.6/site-packages/certifi/cacert.pem? certifi.where() shows later. Also name "ca.crt" is the correct name? | |
| 17:53:57 | mordred | do we not expose the keystoneauth session ssl options? | |
| 17:54:27 | sean-k-mooney | mordred: maybe but that would be documented in keystoneauth | |
| 17:54:32 | sean-k-mooney | not nova | |
| 17:54:53 | sean-k-mooney | sangeet: it should be in /usr/local/share/ca-certificates/ | |
| 17:55:19 | sean-k-mooney | since they are your own addtional CA certs not one packaged by the distro | |
| 17:55:32 | mordred | sean-k-mooney: that won't necessarily work | |
| 17:55:45 | mordred | sean-k-mooney: python requests bundles a CA bundle and doesn;t' use system cas | |
| 17:55:49 | mordred | because MONKEYS | |
| 17:56:01 | sean-k-mooney | ... | |
| 17:56:07 | mordred | don't even get me started | |
| 17:56:35 | sean-k-mooney | ok so do we have docs for how to configure keystone with tls somehwere | |
| 17:56:40 | mordred | which is why it's important to be able to pass a path to a CA in config - I believe nova is doing a register_conf_options from ksa - so it should be possible to pass cafile in nova.conf ... | |
| 17:56:43 | sean-k-mooney | its not in the nova security docs | |
| 17:56:56 | mordred | well - that might be a bug - but let me see what I can find | |
| 17:57:00 | sean-k-mooney | mordred: ya i think we do | |
| 17:57:26 | sean-k-mooney | i guess it could be in the install docs i just did a google search and didnt find them | |
| 17:57:45 | mordred | register_ksa_opts | |
| 17:58:15 | mordred | that's in nova/conf/utils - and calls ks_loading.register_session_conf_options - which will register an option "cafile" | |
| 18:01:01 | mordred | it's going to be per-service - so I think it would be [identity]cafile=/etc/nova/certs/ca.crt - but really it would be best to put it in [networking]cafile= and [image]cafile and [placement]cafile= too ... | |
| 18:01:16 | mordred | I'm not 100% sure what the story would be if keystone and a given service need _different_ ca's | |
| 18:01:21 | mordred | sangeet: ^^ | |
| 18:02:34 | sangeet | thanks mordred .. so I onlty need to change the conf file as you suggested and it will be used by ksa automatically? | |
| 18:02:54 | mordred | sangeet: yes. at least I hope so - that's the theory :) | |
| 18:03:14 | sangeet | Let me try that .. my system is up. Thanks | |
| 18:03:21 | mordred | woot! | |
| 18:03:27 | mordred | we should really make a general [session] config section - having those options repeat for each service is a little weird | |
| 18:04:58 | mordred | sean-k-mooney: I feel like I should update the nova docs to include this information - but I'm not really sure where would be a good idea for that | |
| 18:06:22 | mordred | efried: isn't there docs somewhere about the ksa conf options | |
| 18:06:24 | mordred | ? | |
| 18:06:52 | efried | ... | |
| 18:07:32 | mordred | efried: like - now that you can configure session adapter stuff via service-types-authority names and the ksa options ... do we have docs about that in the nova docs? | |
| 18:08:00 | efried | I think I understand the question, I'm just trying to swap that back in from tape. | |
| 18:09:00 | efried | mordred: https://docs.openstack.org/nova/latest/configuration/config.html | |
| 18:09:21 | efried | If you search for e.g. `cafile` you'll find an entry for each $service that uses ksa. | |
| 18:09:51 | mordred | efried: ah - oh, that's probably generated from ksa by sphinx | |
| 18:09:51 | efried | six entries for `endpoint_override` | |
| 18:09:58 | efried | exactly | |
| 18:10:00 | mordred | so a git grep wasn't finding it - that makes sense | |
| 18:10:02 | mordred | efried: thanks! | |
| 18:10:04 | efried | yw | |
| 18:10:39 | efried | not sure about 'sphinx', but generated by doc build, yes. | |
| 18:11:31 | efried | ...and I think it comes in by virtue of the `list_opts()` methods in conf/*, e.g. https://github.com/openstack/nova/blob/master/nova/conf/glance.py#L173 | |
| 18:11:49 | efried | ...which as you can see uses ksa's methods for generating those | |
| 18:13:07 | mordred | efried: yah. \o/ yay | |
| 18:19:34 | sangeet | ten thousand thansk modred .. it worked. I am so exicted. QQ Do I need to put it under idenity also or neutron, glance and placement is enough? | |
| 18:19:53 | sangeet | Sorry *mordred ^^ | |
| 18:21:12 | sean-k-mooney | mordred: i was expecting to find it here https://docs.openstack.org/nova/latest/admin/security.html | |
| 18:21:37 | sean-k-mooney | mordred: although the install guide would make sense | |
| 18:22:15 | sean-k-mooney | efried: mordred i dont think the config guide is really helpful in this case | |
| 18:22:47 | sean-k-mooney | that is where i started but i did not find it mainly because i was looking for ca_ | |
| 18:22:55 | sean-k-mooney | but it was not obvious | |
| 18:23:25 | efried | I don't think we should describe the optinos in depth in the security guide, but it would be sane to refer to the config docs from there. | |
| 18:24:42 | sean-k-mooney | yep that is what i was thinking too altough i think having a secting in the install guide would make sense | |
| 18:24:49 | sean-k-mooney | e.g. how to isntall with tls? | |
| 18:27:07 | sangeet | I agree .. that would be an excellent idea | |
| 18:28:56 | mordred | sangeet: you'll likely need to put it in for each service you're using | |
| 18:29:24 | mordred | sangeet: so - yeah - I'd do identity I think | |
| 18:38:41 | sangeet | Thank you mordred | |
| 19:12:44 | sangeet | mordred .. oops now my nova-conductor is not liking it when I try to create a server. I have set cafiles as we discussed above. "OSError: Could not find a suitable TLS CA certificate bundle, invalid path: /etc/nova/certs/ca.crt" | |
| 19:13:21 | sangeet | It seems conductor is expecting the CA file to be at some other location | |
| 19:17:38 | mordred | sangeet: are those on the same machine? | |
| 19:17:50 | sangeet | differnt pods | |
| 19:18:06 | sangeet | the file exist | |
| 19:18:45 | mordred | hrm. I'm not sure about that one - maybe someone else will know | |
| 19:19:34 | sangeet | sean-k-mooney efried .. please help ^^ | |
| 19:20:31 | efried | I'm no expert here, so this is just a guess: | |
| 19:20:49 | efried | If you put this in [identity], it means all the nova services will try to use it when talking to keystone. | |
| 19:20:50 | sangeet | I am up for trying anything | |
| 19:21:05 | efried | So you need it on every node that's running any nova service (conductor, compute, scheduler, whatever) | |
| 19:21:15 | efried | it == the crt fil. | |
| 19:21:16 | efried | file | |
| 19:21:41 | sangeet | so put cafile=/etc/nova/certs/ca.crt under identity | |
| 19:21:53 | efried | eh? I thought that's what you did, and it didn't work | |
| 19:22:00 | efried | Let's back up. | |
| 19:22:06 | efried | What change did you make that's leading to this error? | |
| 19:22:07 | sangeet | I ut it in compute and not in conductore | |
| 19:22:30 | sangeet | Let me try to put it under identity also | |
| 19:22:39 | efried | waitwait | |
| 19:22:52 | efried | I haven't been following this conversation, so I don't want to lead you down a rabbit hole. | |
| 19:22:59 | efried | What exactly have you changed so far? | |
| 19:25:52 | sangeet | I have Keystone deployed with SSL. In my nova.conf I set cafile=/etc/nova/certs/ca.crt for [neutron], [glance], [keystone_authtoken], [placement] | |
| 19:26:53 | efried | Hm, okay, I'm not sure about [keystone_authtoken] -- that's to set up the server side of the keystone service. | |
| 19:27:29 | efried | But nova talks to some of those services from multiple places -- conductor, scheduler, compute. | |
| 19:27:40 | efried | I don't remember offhand which ones talk to which ones from where. | |
| 19:29:09 | efried | Also note that conductor and compute use different config files by default (unless that's changed, or unless you're on an old release), so if you're running both, you'll want to set up both files. | |
| 19:29:46 | efried | But honestly, beyond that, I'm really out of my depth. sean-k-mooney would probably be a better resource. He is EU, so maybe try again "tomorrow". | |
| 19:30:22 | sangeet | I am using stein and I do have different configs as they are running on a different pods | |