| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-11-04 | |||
| 08:28:08 | jpodivin | gibi: anything specific I should look for? | |
| 08:28:28 | gibi | jpodivin: do you see requests resulting in 404 there? | |
| 08:28:38 | EugenMayer | when running nova backup on an image, i see pending operations in the UI. Is there any way to show this tasks and their detailed operations on the cli? glance taks-list and glance image-tasks do not show any pending tasks | |
| 08:29:13 | gibi | we had an issue with apache config not long ago resulted in wrong proxy config for placement https://review.opendev.org/c/openstack/devstack/+/811389 | |
| 08:29:31 | jpodivin | gibi: no, it looks fairly clean, couple of warnings but those seem related to deprecation. https://logserver.rdoproject.org/99/36499/6/check/periodic-tripleo-ci-centos-8-containers-multinode-compute-master-validation/91262ed/logs/subnode-1/var/log/containers/placement/placement.log.txt.gz | |
| 08:31:31 | jpodivin | gibi: found another log, that might be what you meant. Lists a bunch of requests but 404 https://logserver.rdoproject.org/99/36499/6/check/periodic-tripleo-ci-centos-8-containers-multinode-compute-master-validation/91262ed/logs/subnode-1/var/log/containers/httpd/placement/placement_wsgi_access.log.txt.gz | |
| 08:32:33 | jpodivin | Interesting thing is: it starts *after* the error occurs, not before. | |
| 08:32:40 | gibi | yeah, these logs seems to indicate your placement service works correctly and some clients (probably nova) was able to communicate with it | |
| 08:33:23 | gibi | if the error happens after the logs ends then it can be that your placement service was simply stopped / crashed | |
| 08:34:10 | jpodivin | gibi: looking at the time stamps, it looks like the error occurs a approximately one minute before the placement requests log starts . | |
| 08:34:50 | jpodivin | gibi: scrap that. It's actually about 3 seconds before | |
| 08:35:05 | jpodivin | still, before any communication is recorded . | |
| 08:35:25 | gibi | still it means that the placement server is not running when you client tried to read the supported versions from it | |
| 08:40:44 | lyarwood | bauzas / melwitt ; https://review.opendev.org/c/openstack/nova/+/807025 - would you mind hitting this and the fups on top this week? | |
| 08:45:31 | jpodivin | gibi: yes that does seem to be the case | |
| 08:45:50 | jpodivin | gibi: question is: why would that happen on master and not on wallaby. | |
| 09:06:37 | gibi | jpodivin: I have no idea | |
| 09:34:34 | lyarwood | sean-k-mooney: can you hit https://review.opendev.org/c/openstack/nova/+/811716 again when you get a chance | |
| 09:35:43 | EugenMayer | after deploying with kolla and tls, it seems like the GUI based TTY console is still behin to http, not https, while anything else is properly behind tls encryption. The point is, horizon tries to load from https:6080 .. but it fails (no socket). http:6080 has a socket, but the token seems to be invalid there. Any hints what could be wrong, i | |
| 09:35:43 | EugenMayer | assume this is related to the nova service? | |
| 09:36:50 | opendevreview | Lee Yarwood proposed openstack/nova master: WIP configdrive: Move mkisofs_cmd default to mkisofs https://review.opendev.org/c/openstack/nova/+/808921 | |
| 09:36:59 | EugenMayer | when looking at the nova-compute configuration i see https://gist.github.com/EugenMayer/058499029fbd298600a8efa634687c92 | |
| 09:37:52 | bauzas | gibi: done, https://blueprints.launchpad.net/nova/+spec/qos-minimum-guaranteed-packet-rate is now Implemented | |
| 09:37:59 | gibi | bauzas: thanks | |
| 09:38:03 | bauzas | (thanks for noticing me) | |
| 09:38:10 | bauzas | lyarwood: ack, will look | |
| 09:38:13 | EugenMayer | soseems like https://bugzilla.redhat.com/show_bug.cgi?id=1722089 is related | |
| 09:40:12 | opendevreview | Lee Yarwood proposed openstack/nova master: DNM/WIP libvirt: Default x86_64 instances to the q35 machine type https://review.opendev.org/c/openstack/nova/+/816629 | |
| 09:41:04 | kashyap | lyarwood: --^ I'm curious too... :) Thx for posting it | |
| 09:41:07 | lyarwood | EugenMayer: the downstream Red hat bug isn't related | |
| 09:42:14 | EugenMayer | lyarwood thank you. Not sure what i miss then, reading https://docs.openstack.org/nova/xena/admin/remote-console-access.html | |
| 09:42:31 | EugenMayer | a little hard to understand what kolla did and did not | |
| 09:45:33 | EugenMayer | this is what i have in nova-vnc https://gist.github.com/EugenMayer/82528fcfca6e22b818f865852606f28c - currently kind of double posting since i'am not sure this is a nova 'issue' or a kolla 'configuration' issue or a kolla 'deployment bug' | |
| 09:47:03 | lyarwood | EugenMayer: I'd think that's a kolla config bug tbh | |
| 09:47:44 | lyarwood | the URL returned from n-api is https right? | |
| 09:47:55 | lyarwood | when you do a console url show? | |
| 09:48:29 | EugenMayer | lyarwood how would i do that? | |
| 09:51:56 | lyarwood | EugenMayer: https://docs.openstack.org/python-openstackclient/latest/cli/command-objects/console-url.html#console-url-show openstack console url show $instance | |
| 09:54:46 | EugenMayer | lyarwood the URL looks good. HTTPS, right fqdn, 60800 | |
| 09:55:55 | EugenMayer | telent on that port works, but i assume it talks http | |
| 09:56:18 | EugenMayer | wget returns 'GnuTLS: Error in the pull function. | |
| 09:56:18 | EugenMayer | Unable to establish SSL connection.' | |
| 09:56:37 | lyarwood | kk then it's the nova-novncproxy service that isn't configured correctly and thus a Kolla bug | |
| 09:59:52 | EugenMayer | lyarwood thank you. Already looking at the configuration. I assume the certificates are not bound properly | |
| 10:02:20 | EugenMayer | lyarwood https://gist.github.com/EugenMayer/fba4eb20a49ccd717ba70f38188a8e1e - i'am using officially signed certificates. Maybe missing https://docs.openstack.org/nova/xena/admin/remote-console-access.html#novnc-proxy-server-configuration /etc/pki/libvirt-vnc/server-cert.pem or something? | |
| 10:04:49 | lyarwood | that's to encrypt the connection between the proxy service and libvirtd | |
| 10:05:03 | lyarwood | I think your issue is between the user and the proxy service right? | |
| 10:10:28 | EugenMayer | yes | |
| 10:29:08 | EugenMayer | lyarwood any hints where to start | |
| 10:29:56 | lyarwood | EugenMayer: I'm not sure how Kolla configures things tbh but I'd start with the config file associated with the nova-novncproxy service itself | |
| 10:30:15 | lyarwood | EugenMayer: see if it differs from nova-api etc that are working with tls already | |
| 10:30:19 | EugenMayer | that's what i posted lyarwood - the problem is, not sure what to expect there | |
| 10:34:03 | opendevreview | Merged openstack/nova stable/xena: Store old_flavor already on source host during resize https://review.opendev.org/c/openstack/nova/+/810911 | |
| 10:36:08 | lyarwood | EugenMayer: https://github.com/openstack/nova/blob/909cfc76369b94b026cf42b86fb5a310dce21a8c/nova/cmd/baseproxy.py#L48-L87 - so looking at the code it's using configurables like ssl_only and cert | |
| 10:37:43 | EugenMayer | which are missing in my case | |
| 10:37:51 | lyarwood | EugenMayer: iirc from an earlier pastebin websockify couldn't find the cert? | |
| 10:39:22 | lyarwood | I've lost the gist now but I'm sure it was listing a cert file | |
| 10:39:37 | lyarwood | so you must have cert set to something in the config used by the proxy service | |
| 10:39:46 | lyarwood | it's in the DEFAULT namespace btw, not under vnc | |
| 10:39:59 | lyarwood | just grep from ^cert | |
| 10:40:01 | lyarwood | for* | |
| 10:45:25 | EugenMayer | sorry i'am lost lyarwood, not sure which you mean https://gist.github.com/EugenMayer/058499029fbd298600a8efa634687c92 or https://gist.github.com/EugenMayer/fba4eb20a49ccd717ba70f38188a8e1e or https://gist.github.com/EugenMayer/82528fcfca6e22b818f865852606f28c - if nothing of this, which on do you need. Happy to hand over anything | |
| 10:45:48 | lyarwood | 2021-11-04 09:18:39.933 13300 INFO nova.console.websocketproxy [-] 10.0.1.1: SSL connection but '/self.pem' not found | |
| 10:46:20 | lyarwood | ^ that error suggests that the nova-novncproxy hasn't been configured correctly | |
| 10:50:53 | EugenMayer | yes, that is what i saw too. So 'cert' in noproxy under [vnc] | |
| 10:50:57 | opendevreview | Wenping Song proposed openstack/nova master: Support concurrently add hosts to aggregates https://review.opendev.org/c/openstack/nova/+/815105 | |
| 10:51:11 | lyarwood | EugenMayer: no, it's outside of that in the default namespace AFAICT | |
| 10:51:40 | lyarwood | EugenMayer: if you grep for self.pem you should be able to find it | |
| 10:51:57 | lyarwood | EugenMayer: this is definitely a Kolla bug FWIW | |
| 10:52:18 | lyarwood | EugenMayer: it smells like it hasn't copied that cert into the container for the service or something? | |
| 10:53:51 | EugenMayer | it does copy those and maybe is missing one. Let me clear up my confusion Which part of the configuration is broken / missing ther certs. On the computes the nova-compute or on the controller the novncproxy | |
| 10:55:41 | lyarwood | EugenMayer: on the contrller, the novncproxy service | |
| 10:57:09 | lyarwood | EugenMayer: assuming the logs you shared were from the controller | |
| 10:57:26 | lyarwood | EugenMayer: regarding 2021-11-04 09:18:39.933 13300 INFO nova.console.websocketproxy [-] 10.0.1.1: SSL connection but '/self.pem' not found | |
| 10:58:12 | lyarwood | Yeah actually that can only be from the controller | |
| 11:02:42 | EugenMayer | lyarwood ok so i check the noproxy container and how the certificates are deployed and configured | |
| 11:05:04 | lyarwood | https://github.com/novnc/websockify/blob/master/README.md#encrypted-websocket-connections-wss - FWIW self.pem is the default cert websockify will try to load when it's asked to use SSL | |
| 11:06:01 | lyarwood | as I said before https://github.com/openstack/nova/blob/909cfc76369b94b026cf42b86fb5a310dce21a8c/nova/cmd/baseproxy.py#L48-L87 is where Nova tries to provide the correct cert when launching the novncproxy service | |
| 11:07:39 | EugenMayer | well ok now i guess that will be the issue. is self.pem a cert+private-key format? | |
| 11:08:08 | EugenMayer | ah --cert --key | |
| 11:10:18 | lyarwood | right you should already have these in your env? | |
| 11:10:24 | lyarwood | just under a different filename | |
| 11:10:34 | lyarwood | so just update the nova.conf used by the service to point to them | |
| 11:10:42 | lyarwood | cert=/path/to/cert | |
| 11:10:48 | lyarwood | key=/path/to/key | |
| 11:11:07 | lyarwood | and again, DEFAULT namespace so outside of the [vnc] section etc. | |
| 11:11:26 | EugenMayer | checking the ansible tasks right now (kollas) | |
| 11:16:35 | lyarwood | yeah it doesn't look like it has support tbh | |
| 11:16:43 | EugenMayer | https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/nova-cell/templates/nova.conf.j2 | |
| 11:16:45 | lyarwood | looking at the config templates at least | |
| 11:16:46 | EugenMayer | yes, it's mising | |
| 11:17:46 | EugenMayer | lyarwood did the nova implemenation of novnc change since victoria? | |
| 11:18:34 | lyarwood | I don't think anything has that would change this behaviour tbh | |
| 11:18:45 | EugenMayer | in other words, looking back, that template never had cert/key as values set for TLS | |
| 11:19:04 | EugenMayer | so either it has never been supported at all - or it is a regression | |
| 11:20:17 | lyarwood | yeah I would assume this has never been supported by Kolla tbh, should be pretty trivial to correct however | |
| 11:21:22 | EugenMayer | yes it is just PITA to search for that, i you are clueless (like i'am). You never know what is supposed to work and what not, and how a working configuration does look like | |
| 11:22:59 | EugenMayer | lyarwood any idea what the path of self.pem looks like? i mean i do not assume /self.pem is really absolute here | |
| 11:24:33 | lyarwood | https://github.com/openstack/nova/blob/909cfc76369b94b026cf42b86fb5a310dce21a8c/nova/conf/novnc.py#L41-L52 looks like it's relative so it depends how kolla is launching the service | |
| 11:24:56 | lyarwood | but again updating the nova.conf used by the service to point to your actual key and cert is a better option here | |