| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-11-04 | |||
| 08:07:39 | opendevreview | Balazs Gibizer proposed openstack/nova stable/xena: Prevent leaked eventlets to send notifications https://review.opendev.org/c/openstack/nova/+/816487 | |
| 08:08:47 | opendevreview | Balazs Gibizer proposed openstack/nova stable/xena: Avoid unbound instance_uuid var during delete https://review.opendev.org/c/openstack/nova/+/816488 | |
| 08:09:33 | gibi | bauzas: the reno for the pps work has landed so you can marke the | |
| 08:09:39 | gibi | https://blueprints.launchpad.net/nova/+spec/qos-minimum-guaranteed-packet-rate | |
| 08:09:43 | gibi | bp as implemented | |
| 08:09:46 | gibi | \o/ | |
| 08:11:02 | opendevreview | Balazs Gibizer proposed openstack/nova master: Enable min pps tempest testing in nova-next https://review.opendev.org/c/openstack/nova/+/811748 | |
| 08:15:45 | lyarwood | gibi++ | |
| 08:16:03 | gibi | lyarwood: thanks for the reviews yestarday | |
| 08:16:15 | lyarwood | np, still catching up | |
| 08:20:46 | gibi | I hope you enjoyed the explanation of the fix for https://launchpad.net/bugs/1946339 . It was a real adventure for me in eventlet/greenlet land. | |
| 08:21:53 | jpodivin | Hi everyone. Has anyone ever encountered: "The placement service for 192.168.24.100:regionOne exists but does not have any supported versions." ? | |
| 08:22:27 | jpodivin | It's hitting master in rdo, but wallaby somehow avoids it. | |
| 08:26:08 | gibi | jpodivin: hi, could you try to simply send a GET request to the root of you placement service? | |
| 08:26:47 | gibi | you should see something like https://paste.opendev.org/show/810375/ | |
| 08:27:00 | jpodivin | gibi: hm, sadly it's a proposed CI job, so the nodes are already down. | |
| 08:27:07 | jpodivin | althought I do have the logs stored | |
| 08:27:25 | gibi | then it would be good to check the placement service logs | |
| 08:28:08 | jpodivin | gibi: anything specific I should look for? | |
| 08:28:28 | gibi | jpodivin: do you see requests resulting in 404 there? | |
| 08:28:38 | EugenMayer | when running nova backup on an image, i see pending operations in the UI. Is there any way to show this tasks and their detailed operations on the cli? glance taks-list and glance image-tasks do not show any pending tasks | |
| 08:29:13 | gibi | we had an issue with apache config not long ago resulted in wrong proxy config for placement https://review.opendev.org/c/openstack/devstack/+/811389 | |
| 08:29:31 | jpodivin | gibi: no, it looks fairly clean, couple of warnings but those seem related to deprecation. https://logserver.rdoproject.org/99/36499/6/check/periodic-tripleo-ci-centos-8-containers-multinode-compute-master-validation/91262ed/logs/subnode-1/var/log/containers/placement/placement.log.txt.gz | |
| 08:31:31 | jpodivin | gibi: found another log, that might be what you meant. Lists a bunch of requests but 404 https://logserver.rdoproject.org/99/36499/6/check/periodic-tripleo-ci-centos-8-containers-multinode-compute-master-validation/91262ed/logs/subnode-1/var/log/containers/httpd/placement/placement_wsgi_access.log.txt.gz | |
| 08:32:33 | jpodivin | Interesting thing is: it starts *after* the error occurs, not before. | |
| 08:32:40 | gibi | yeah, these logs seems to indicate your placement service works correctly and some clients (probably nova) was able to communicate with it | |
| 08:33:23 | gibi | if the error happens after the logs ends then it can be that your placement service was simply stopped / crashed | |
| 08:34:10 | jpodivin | gibi: looking at the time stamps, it looks like the error occurs a approximately one minute before the placement requests log starts . | |
| 08:34:50 | jpodivin | gibi: scrap that. It's actually about 3 seconds before | |
| 08:35:05 | jpodivin | still, before any communication is recorded . | |
| 08:35:25 | gibi | still it means that the placement server is not running when you client tried to read the supported versions from it | |
| 08:40:44 | lyarwood | bauzas / melwitt ; https://review.opendev.org/c/openstack/nova/+/807025 - would you mind hitting this and the fups on top this week? | |
| 08:45:31 | jpodivin | gibi: yes that does seem to be the case | |
| 08:45:50 | jpodivin | gibi: question is: why would that happen on master and not on wallaby. | |
| 09:06:37 | gibi | jpodivin: I have no idea | |
| 09:34:34 | lyarwood | sean-k-mooney: can you hit https://review.opendev.org/c/openstack/nova/+/811716 again when you get a chance | |
| 09:35:43 | EugenMayer | after deploying with kolla and tls, it seems like the GUI based TTY console is still behin to http, not https, while anything else is properly behind tls encryption. The point is, horizon tries to load from https:6080 .. but it fails (no socket). http:6080 has a socket, but the token seems to be invalid there. Any hints what could be wrong, i | |
| 09:35:43 | EugenMayer | assume this is related to the nova service? | |
| 09:36:50 | opendevreview | Lee Yarwood proposed openstack/nova master: WIP configdrive: Move mkisofs_cmd default to mkisofs https://review.opendev.org/c/openstack/nova/+/808921 | |
| 09:36:59 | EugenMayer | when looking at the nova-compute configuration i see https://gist.github.com/EugenMayer/058499029fbd298600a8efa634687c92 | |
| 09:37:52 | bauzas | gibi: done, https://blueprints.launchpad.net/nova/+spec/qos-minimum-guaranteed-packet-rate is now Implemented | |
| 09:37:59 | gibi | bauzas: thanks | |
| 09:38:03 | bauzas | (thanks for noticing me) | |
| 09:38:10 | bauzas | lyarwood: ack, will look | |
| 09:38:13 | EugenMayer | soseems like https://bugzilla.redhat.com/show_bug.cgi?id=1722089 is related | |
| 09:40:12 | opendevreview | Lee Yarwood proposed openstack/nova master: DNM/WIP libvirt: Default x86_64 instances to the q35 machine type https://review.opendev.org/c/openstack/nova/+/816629 | |
| 09:41:04 | kashyap | lyarwood: --^ I'm curious too... :) Thx for posting it | |
| 09:41:07 | lyarwood | EugenMayer: the downstream Red hat bug isn't related | |
| 09:42:14 | EugenMayer | lyarwood thank you. Not sure what i miss then, reading https://docs.openstack.org/nova/xena/admin/remote-console-access.html | |
| 09:42:31 | EugenMayer | a little hard to understand what kolla did and did not | |
| 09:45:33 | EugenMayer | this is what i have in nova-vnc https://gist.github.com/EugenMayer/82528fcfca6e22b818f865852606f28c - currently kind of double posting since i'am not sure this is a nova 'issue' or a kolla 'configuration' issue or a kolla 'deployment bug' | |
| 09:47:03 | lyarwood | EugenMayer: I'd think that's a kolla config bug tbh | |
| 09:47:44 | lyarwood | the URL returned from n-api is https right? | |
| 09:47:55 | lyarwood | when you do a console url show? | |
| 09:48:29 | EugenMayer | lyarwood how would i do that? | |
| 09:51:56 | lyarwood | EugenMayer: https://docs.openstack.org/python-openstackclient/latest/cli/command-objects/console-url.html#console-url-show openstack console url show $instance | |
| 09:54:46 | EugenMayer | lyarwood the URL looks good. HTTPS, right fqdn, 60800 | |
| 09:55:55 | EugenMayer | telent on that port works, but i assume it talks http | |
| 09:56:18 | EugenMayer | wget returns 'GnuTLS: Error in the pull function. | |
| 09:56:18 | EugenMayer | Unable to establish SSL connection.' | |
| 09:56:37 | lyarwood | kk then it's the nova-novncproxy service that isn't configured correctly and thus a Kolla bug | |
| 09:59:52 | EugenMayer | lyarwood thank you. Already looking at the configuration. I assume the certificates are not bound properly | |
| 10:02:20 | EugenMayer | lyarwood https://gist.github.com/EugenMayer/fba4eb20a49ccd717ba70f38188a8e1e - i'am using officially signed certificates. Maybe missing https://docs.openstack.org/nova/xena/admin/remote-console-access.html#novnc-proxy-server-configuration /etc/pki/libvirt-vnc/server-cert.pem or something? | |
| 10:04:49 | lyarwood | that's to encrypt the connection between the proxy service and libvirtd | |
| 10:05:03 | lyarwood | I think your issue is between the user and the proxy service right? | |
| 10:10:28 | EugenMayer | yes | |
| 10:29:08 | EugenMayer | lyarwood any hints where to start | |
| 10:29:56 | lyarwood | EugenMayer: I'm not sure how Kolla configures things tbh but I'd start with the config file associated with the nova-novncproxy service itself | |
| 10:30:15 | lyarwood | EugenMayer: see if it differs from nova-api etc that are working with tls already | |
| 10:30:19 | EugenMayer | that's what i posted lyarwood - the problem is, not sure what to expect there | |
| 10:34:03 | opendevreview | Merged openstack/nova stable/xena: Store old_flavor already on source host during resize https://review.opendev.org/c/openstack/nova/+/810911 | |
| 10:36:08 | lyarwood | EugenMayer: https://github.com/openstack/nova/blob/909cfc76369b94b026cf42b86fb5a310dce21a8c/nova/cmd/baseproxy.py#L48-L87 - so looking at the code it's using configurables like ssl_only and cert | |
| 10:37:43 | EugenMayer | which are missing in my case | |
| 10:37:51 | lyarwood | EugenMayer: iirc from an earlier pastebin websockify couldn't find the cert? | |
| 10:39:22 | lyarwood | I've lost the gist now but I'm sure it was listing a cert file | |
| 10:39:37 | lyarwood | so you must have cert set to something in the config used by the proxy service | |
| 10:39:46 | lyarwood | it's in the DEFAULT namespace btw, not under vnc | |
| 10:39:59 | lyarwood | just grep from ^cert | |
| 10:40:01 | lyarwood | for* | |
| 10:45:25 | EugenMayer | sorry i'am lost lyarwood, not sure which you mean https://gist.github.com/EugenMayer/058499029fbd298600a8efa634687c92 or https://gist.github.com/EugenMayer/fba4eb20a49ccd717ba70f38188a8e1e or https://gist.github.com/EugenMayer/82528fcfca6e22b818f865852606f28c - if nothing of this, which on do you need. Happy to hand over anything | |
| 10:45:48 | lyarwood | 2021-11-04 09:18:39.933 13300 INFO nova.console.websocketproxy [-] 10.0.1.1: SSL connection but '/self.pem' not found | |
| 10:46:20 | lyarwood | ^ that error suggests that the nova-novncproxy hasn't been configured correctly | |
| 10:50:53 | EugenMayer | yes, that is what i saw too. So 'cert' in noproxy under [vnc] | |
| 10:50:57 | opendevreview | Wenping Song proposed openstack/nova master: Support concurrently add hosts to aggregates https://review.opendev.org/c/openstack/nova/+/815105 | |
| 10:51:11 | lyarwood | EugenMayer: no, it's outside of that in the default namespace AFAICT | |
| 10:51:40 | lyarwood | EugenMayer: if you grep for self.pem you should be able to find it | |
| 10:51:57 | lyarwood | EugenMayer: this is definitely a Kolla bug FWIW | |
| 10:52:18 | lyarwood | EugenMayer: it smells like it hasn't copied that cert into the container for the service or something? | |
| 10:53:51 | EugenMayer | it does copy those and maybe is missing one. Let me clear up my confusion Which part of the configuration is broken / missing ther certs. On the computes the nova-compute or on the controller the novncproxy | |
| 10:55:41 | lyarwood | EugenMayer: on the contrller, the novncproxy service | |
| 10:57:09 | lyarwood | EugenMayer: assuming the logs you shared were from the controller | |
| 10:57:26 | lyarwood | EugenMayer: regarding 2021-11-04 09:18:39.933 13300 INFO nova.console.websocketproxy [-] 10.0.1.1: SSL connection but '/self.pem' not found | |
| 10:58:12 | lyarwood | Yeah actually that can only be from the controller | |
| 11:02:42 | EugenMayer | lyarwood ok so i check the noproxy container and how the certificates are deployed and configured | |
| 11:05:04 | lyarwood | https://github.com/novnc/websockify/blob/master/README.md#encrypted-websocket-connections-wss - FWIW self.pem is the default cert websockify will try to load when it's asked to use SSL | |
| 11:06:01 | lyarwood | as I said before https://github.com/openstack/nova/blob/909cfc76369b94b026cf42b86fb5a310dce21a8c/nova/cmd/baseproxy.py#L48-L87 is where Nova tries to provide the correct cert when launching the novncproxy service | |
| 11:07:39 | EugenMayer | well ok now i guess that will be the issue. is self.pem a cert+private-key format? | |
| 11:08:08 | EugenMayer | ah --cert --key | |
| 11:10:18 | lyarwood | right you should already have these in your env? | |
| 11:10:24 | lyarwood | just under a different filename | |