Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-11-04
09:45:33 EugenMayer this is what i have in nova-vnc https://gist.github.com/EugenMayer/82528fcfca6e22b818f865852606f28c - currently kind of double posting since i'am not sure this is a nova 'issue' or a kolla 'configuration' issue or a kolla 'deployment bug'
09:47:03 lyarwood EugenMayer: I'd think that's a kolla config bug tbh
09:47:44 lyarwood the URL returned from n-api is https right?
09:47:55 lyarwood when you do a console url show?
09:48:29 EugenMayer lyarwood how would i do that?
09:51:56 lyarwood EugenMayer: https://docs.openstack.org/python-openstackclient/latest/cli/command-objects/console-url.html#console-url-show openstack console url show $instance
09:54:46 EugenMayer lyarwood the URL looks good. HTTPS, right fqdn, 60800
09:55:55 EugenMayer telent on that port works, but i assume it talks http
09:56:18 EugenMayer Unable to establish SSL connection.'
09:56:18 EugenMayer wget returns 'GnuTLS: Error in the pull function.
09:56:37 lyarwood kk then it's the nova-novncproxy service that isn't configured correctly and thus a Kolla bug
09:59:52 EugenMayer lyarwood thank you. Already looking at the configuration. I assume the certificates are not bound properly
10:02:20 EugenMayer lyarwood https://gist.github.com/EugenMayer/fba4eb20a49ccd717ba70f38188a8e1e - i'am using officially signed certificates. Maybe missing https://docs.openstack.org/nova/xena/admin/remote-console-access.html#novnc-proxy-server-configuration /etc/pki/libvirt-vnc/server-cert.pem or something?
10:04:49 lyarwood that's to encrypt the connection between the proxy service and libvirtd
10:05:03 lyarwood I think your issue is between the user and the proxy service right?
10:10:28 EugenMayer yes
10:29:08 EugenMayer lyarwood any hints where to start
10:29:56 lyarwood EugenMayer: I'm not sure how Kolla configures things tbh but I'd start with the config file associated with the nova-novncproxy service itself
10:30:15 lyarwood EugenMayer: see if it differs from nova-api etc that are working with tls already
10:30:19 EugenMayer that's what i posted lyarwood - the problem is, not sure what to expect there
10:34:03 opendevreview Merged openstack/nova stable/xena: Store old_flavor already on source host during resize https://review.opendev.org/c/openstack/nova/+/810911
10:36:08 lyarwood EugenMayer: https://github.com/openstack/nova/blob/909cfc76369b94b026cf42b86fb5a310dce21a8c/nova/cmd/baseproxy.py#L48-L87 - so looking at the code it's using configurables like ssl_only and cert
10:37:43 EugenMayer which are missing in my case
10:37:51 lyarwood EugenMayer: iirc from an earlier pastebin websockify couldn't find the cert?
10:39:22 lyarwood I've lost the gist now but I'm sure it was listing a cert file
10:39:37 lyarwood so you must have cert set to something in the config used by the proxy service
10:39:46 lyarwood it's in the DEFAULT namespace btw, not under vnc
10:39:59 lyarwood just grep from ^cert
10:40:01 lyarwood for*
10:45:25 EugenMayer sorry i'am lost lyarwood, not sure which you mean https://gist.github.com/EugenMayer/058499029fbd298600a8efa634687c92 or https://gist.github.com/EugenMayer/fba4eb20a49ccd717ba70f38188a8e1e or https://gist.github.com/EugenMayer/82528fcfca6e22b818f865852606f28c - if nothing of this, which on do you need. Happy to hand over anything
10:45:48 lyarwood 2021-11-04 09:18:39.933 13300 INFO nova.console.websocketproxy [-] 10.0.1.1: SSL connection but '/self.pem' not found
10:46:20 lyarwood ^ that error suggests that the nova-novncproxy hasn't been configured correctly
10:50:53 EugenMayer yes, that is what i saw too. So 'cert' in noproxy under [vnc]
10:50:57 opendevreview Wenping Song proposed openstack/nova master: Support concurrently add hosts to aggregates https://review.opendev.org/c/openstack/nova/+/815105
10:51:11 lyarwood EugenMayer: no, it's outside of that in the default namespace AFAICT
10:51:40 lyarwood EugenMayer: if you grep for self.pem you should be able to find it
10:51:57 lyarwood EugenMayer: this is definitely a Kolla bug FWIW
10:52:18 lyarwood EugenMayer: it smells like it hasn't copied that cert into the container for the service or something?
10:53:51 EugenMayer it does copy those and maybe is missing one. Let me clear up my confusion Which part of the configuration is broken / missing ther certs. On the computes the nova-compute or on the controller the novncproxy
10:55:41 lyarwood EugenMayer: on the contrller, the novncproxy service
10:57:09 lyarwood EugenMayer: assuming the logs you shared were from the controller
10:57:26 lyarwood EugenMayer: regarding 2021-11-04 09:18:39.933 13300 INFO nova.console.websocketproxy [-] 10.0.1.1: SSL connection but '/self.pem' not found
10:58:12 lyarwood Yeah actually that can only be from the controller
11:02:42 EugenMayer lyarwood ok so i check the noproxy container and how the certificates are deployed and configured
11:05:04 lyarwood https://github.com/novnc/websockify/blob/master/README.md#encrypted-websocket-connections-wss - FWIW self.pem is the default cert websockify will try to load when it's asked to use SSL
11:06:01 lyarwood as I said before https://github.com/openstack/nova/blob/909cfc76369b94b026cf42b86fb5a310dce21a8c/nova/cmd/baseproxy.py#L48-L87 is where Nova tries to provide the correct cert when launching the novncproxy service
11:07:39 EugenMayer well ok now i guess that will be the issue. is self.pem a cert+private-key format?
11:08:08 EugenMayer ah --cert --key
11:10:18 lyarwood right you should already have these in your env?
11:10:24 lyarwood just under a different filename
11:10:34 lyarwood so just update the nova.conf used by the service to point to them
11:10:42 lyarwood cert=/path/to/cert
11:10:48 lyarwood key=/path/to/key
11:11:07 lyarwood and again, DEFAULT namespace so outside of the [vnc] section etc.
11:11:26 EugenMayer checking the ansible tasks right now (kollas)
11:16:35 lyarwood yeah it doesn't look like it has support tbh
11:16:43 EugenMayer https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/nova-cell/templates/nova.conf.j2
11:16:45 lyarwood looking at the config templates at least
11:16:46 EugenMayer yes, it's mising
11:17:46 EugenMayer lyarwood did the nova implemenation of novnc change since victoria?
11:18:34 lyarwood I don't think anything has that would change this behaviour tbh
11:18:45 EugenMayer in other words, looking back, that template never had cert/key as values set for TLS
11:19:04 EugenMayer so either it has never been supported at all - or it is a regression
11:20:17 lyarwood yeah I would assume this has never been supported by Kolla tbh, should be pretty trivial to correct however
11:21:22 EugenMayer yes it is just PITA to search for that, i you are clueless (like i'am). You never know what is supposed to work and what not, and how a working configuration does look like
11:22:59 EugenMayer lyarwood any idea what the path of self.pem looks like? i mean i do not assume /self.pem is really absolute here
11:24:33 lyarwood https://github.com/openstack/nova/blob/909cfc76369b94b026cf42b86fb5a310dce21a8c/nova/conf/novnc.py#L41-L52 looks like it's relative so it depends how kolla is launching the service
11:24:56 lyarwood but again updating the nova.conf used by the service to point to your actual key and cert is a better option here
11:25:24 EugenMayer yes sure, it is a little more work then that
11:26:00 EugenMayer i will need to volume-mount the certs first, i cannot just docker cp them, or they will be lost on upgrade. Then a config override for nova conf (conditional) and then mounting that certs
11:26:25 EugenMayer but still, HUGE help lyarwood, i should be able to handle the rest. Thank you big times!
11:26:37 lyarwood np good luck :)
11:26:49 EugenMayer thank you sir
12:24:29 opendevreview Vlad Gusev proposed openstack/nova stable/train: Reproduce bug 1897528 https://review.opendev.org/c/openstack/nova/+/792116
12:24:53 opendevreview Vlad Gusev proposed openstack/nova stable/train: Ignore PCI devices with 32bit domain https://review.opendev.org/c/openstack/nova/+/792117
12:25:26 opendevreview Vlad Gusev proposed openstack/nova stable/stein: Reproduce bug 1897528 https://review.opendev.org/c/openstack/nova/+/816656
12:26:16 opendevreview Vlad Gusev proposed openstack/nova stable/stein: Reproduce bug 1897528 https://review.opendev.org/c/openstack/nova/+/816656
12:37:35 opendevreview Vlad Gusev proposed openstack/nova stable/stein: Reproduce bug 1897528 https://review.opendev.org/c/openstack/nova/+/816656
12:37:36 opendevreview Vlad Gusev proposed openstack/nova stable/stein: Ignore PCI devices with 32bit domain https://review.opendev.org/c/openstack/nova/+/816682
13:20:08 opendevreview Vlad Gusev proposed openstack/nova stable/stein: Ignore PCI devices with 32bit domain https://review.opendev.org/c/openstack/nova/+/816682
13:24:49 opendevreview Merged openstack/nova master: compute: Update volume_id within connection_info during swap_volume https://review.opendev.org/c/openstack/nova/+/807025
13:24:57 opendevreview Merged openstack/nova master: fup: Move _wait_for_volume_{attach,detach} to os-volume_attachments https://review.opendev.org/c/openstack/nova/+/810775
13:25:06 opendevreview Merged openstack/nova master: fup: Refactor and simplify Cinder fixture GET volume mock https://review.opendev.org/c/openstack/nova/+/810776
14:20:30 opendevreview Merged openstack/nova master: Clean up allocations left by evacuation when deleting service https://review.opendev.org/c/openstack/nova/+/778696
14:20:40 opendevreview Merged openstack/nova stable/wallaby: Reproduce bug 1944759 https://review.opendev.org/c/openstack/nova/+/810912
14:53:26 gibi lyarwood: hi! it seems there is a variant of https://bugs.launchpad.net/nova/+bug/1931702 in https://zuul.opendev.org/t/openstack/build/582935ad35a348cf89dcb25bdc3be0ea/logs But the guest console log at volume detach is different now https://zuul.opendev.org/t/openstack/build/582935ad35a348cf89dcb25bdc3be0ea/log/controller/logs/tempest_log.txt#5444
14:53:31 gibi elodilles: ^^
14:53:46 gibi "[ 15.981709] virtio_blk virtio4: req.0:id 4 is not a head!"
14:54:05 gibi lyarwood: does it ring a bell for you?
14:56:31 lyarwood gibi: no I've not seen that before tbh
14:56:46 gibi lyarwood: ack, thanks
15:02:23 artom_ bauzas, hey, I think the Ironic folks would be really happy if we made https://review.opendev.org/c/openstack/nova/+/813263 a review priority...
15:03:20 sean-k-mooney i see
15:04:48 sean-k-mooney i think should be safe although it raise the question about oter life cyle events liek power on power off and had/soft reboot
15:05:02 artom sean-k-mooney, the Ironic patch?
15:05:25 sean-k-mooney yes
15:05:29 artom Yeah, I suppose it does, but from what I've seen, use of plug_vifs() is highly limited, so it's safe to make it a noop
15:05:44 sean-k-mooney no its not
15:05:59 sean-k-mooney we need to call it for the inial spawn
15:06:02 artom sean-k-mooney, I mean, look at my review notes inline, and tell me if I've missed something :)

Earlier   Later