| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-11-04 | |||
| 10:04:49 | lyarwood | that's to encrypt the connection between the proxy service and libvirtd | |
| 10:05:03 | lyarwood | I think your issue is between the user and the proxy service right? | |
| 10:10:28 | EugenMayer | yes | |
| 10:29:08 | EugenMayer | lyarwood any hints where to start | |
| 10:29:56 | lyarwood | EugenMayer: I'm not sure how Kolla configures things tbh but I'd start with the config file associated with the nova-novncproxy service itself | |
| 10:30:15 | lyarwood | EugenMayer: see if it differs from nova-api etc that are working with tls already | |
| 10:30:19 | EugenMayer | that's what i posted lyarwood - the problem is, not sure what to expect there | |
| 10:34:03 | opendevreview | Merged openstack/nova stable/xena: Store old_flavor already on source host during resize https://review.opendev.org/c/openstack/nova/+/810911 | |
| 10:36:08 | lyarwood | EugenMayer: https://github.com/openstack/nova/blob/909cfc76369b94b026cf42b86fb5a310dce21a8c/nova/cmd/baseproxy.py#L48-L87 - so looking at the code it's using configurables like ssl_only and cert | |
| 10:37:43 | EugenMayer | which are missing in my case | |
| 10:37:51 | lyarwood | EugenMayer: iirc from an earlier pastebin websockify couldn't find the cert? | |
| 10:39:22 | lyarwood | I've lost the gist now but I'm sure it was listing a cert file | |
| 10:39:37 | lyarwood | so you must have cert set to something in the config used by the proxy service | |
| 10:39:46 | lyarwood | it's in the DEFAULT namespace btw, not under vnc | |
| 10:39:59 | lyarwood | just grep from ^cert | |
| 10:40:01 | lyarwood | for* | |
| 10:45:25 | EugenMayer | sorry i'am lost lyarwood, not sure which you mean https://gist.github.com/EugenMayer/058499029fbd298600a8efa634687c92 or https://gist.github.com/EugenMayer/fba4eb20a49ccd717ba70f38188a8e1e or https://gist.github.com/EugenMayer/82528fcfca6e22b818f865852606f28c - if nothing of this, which on do you need. Happy to hand over anything | |
| 10:45:48 | lyarwood | 2021-11-04 09:18:39.933 13300 INFO nova.console.websocketproxy [-] 10.0.1.1: SSL connection but '/self.pem' not found | |
| 10:46:20 | lyarwood | ^ that error suggests that the nova-novncproxy hasn't been configured correctly | |
| 10:50:53 | EugenMayer | yes, that is what i saw too. So 'cert' in noproxy under [vnc] | |
| 10:50:57 | opendevreview | Wenping Song proposed openstack/nova master: Support concurrently add hosts to aggregates https://review.opendev.org/c/openstack/nova/+/815105 | |
| 10:51:11 | lyarwood | EugenMayer: no, it's outside of that in the default namespace AFAICT | |
| 10:51:40 | lyarwood | EugenMayer: if you grep for self.pem you should be able to find it | |
| 10:51:57 | lyarwood | EugenMayer: this is definitely a Kolla bug FWIW | |
| 10:52:18 | lyarwood | EugenMayer: it smells like it hasn't copied that cert into the container for the service or something? | |
| 10:53:51 | EugenMayer | it does copy those and maybe is missing one. Let me clear up my confusion Which part of the configuration is broken / missing ther certs. On the computes the nova-compute or on the controller the novncproxy | |
| 10:55:41 | lyarwood | EugenMayer: on the contrller, the novncproxy service | |
| 10:57:09 | lyarwood | EugenMayer: assuming the logs you shared were from the controller | |
| 10:57:26 | lyarwood | EugenMayer: regarding 2021-11-04 09:18:39.933 13300 INFO nova.console.websocketproxy [-] 10.0.1.1: SSL connection but '/self.pem' not found | |
| 10:58:12 | lyarwood | Yeah actually that can only be from the controller | |
| 11:02:42 | EugenMayer | lyarwood ok so i check the noproxy container and how the certificates are deployed and configured | |
| 11:05:04 | lyarwood | https://github.com/novnc/websockify/blob/master/README.md#encrypted-websocket-connections-wss - FWIW self.pem is the default cert websockify will try to load when it's asked to use SSL | |
| 11:06:01 | lyarwood | as I said before https://github.com/openstack/nova/blob/909cfc76369b94b026cf42b86fb5a310dce21a8c/nova/cmd/baseproxy.py#L48-L87 is where Nova tries to provide the correct cert when launching the novncproxy service | |
| 11:07:39 | EugenMayer | well ok now i guess that will be the issue. is self.pem a cert+private-key format? | |
| 11:08:08 | EugenMayer | ah --cert --key | |
| 11:10:18 | lyarwood | right you should already have these in your env? | |
| 11:10:24 | lyarwood | just under a different filename | |
| 11:10:34 | lyarwood | so just update the nova.conf used by the service to point to them | |
| 11:10:42 | lyarwood | cert=/path/to/cert | |
| 11:10:48 | lyarwood | key=/path/to/key | |
| 11:11:07 | lyarwood | and again, DEFAULT namespace so outside of the [vnc] section etc. | |
| 11:11:26 | EugenMayer | checking the ansible tasks right now (kollas) | |
| 11:16:35 | lyarwood | yeah it doesn't look like it has support tbh | |
| 11:16:43 | EugenMayer | https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/nova-cell/templates/nova.conf.j2 | |
| 11:16:45 | lyarwood | looking at the config templates at least | |
| 11:16:46 | EugenMayer | yes, it's mising | |
| 11:17:46 | EugenMayer | lyarwood did the nova implemenation of novnc change since victoria? | |
| 11:18:34 | lyarwood | I don't think anything has that would change this behaviour tbh | |
| 11:18:45 | EugenMayer | in other words, looking back, that template never had cert/key as values set for TLS | |
| 11:19:04 | EugenMayer | so either it has never been supported at all - or it is a regression | |
| 11:20:17 | lyarwood | yeah I would assume this has never been supported by Kolla tbh, should be pretty trivial to correct however | |
| 11:21:22 | EugenMayer | yes it is just PITA to search for that, i you are clueless (like i'am). You never know what is supposed to work and what not, and how a working configuration does look like | |
| 11:22:59 | EugenMayer | lyarwood any idea what the path of self.pem looks like? i mean i do not assume /self.pem is really absolute here | |
| 11:24:33 | lyarwood | https://github.com/openstack/nova/blob/909cfc76369b94b026cf42b86fb5a310dce21a8c/nova/conf/novnc.py#L41-L52 looks like it's relative so it depends how kolla is launching the service | |
| 11:24:56 | lyarwood | but again updating the nova.conf used by the service to point to your actual key and cert is a better option here | |
| 11:25:24 | EugenMayer | yes sure, it is a little more work then that | |
| 11:26:00 | EugenMayer | i will need to volume-mount the certs first, i cannot just docker cp them, or they will be lost on upgrade. Then a config override for nova conf (conditional) and then mounting that certs | |
| 11:26:25 | EugenMayer | but still, HUGE help lyarwood, i should be able to handle the rest. Thank you big times! | |
| 11:26:37 | lyarwood | np good luck :) | |
| 11:26:49 | EugenMayer | thank you sir | |
| 12:24:29 | opendevreview | Vlad Gusev proposed openstack/nova stable/train: Reproduce bug 1897528 https://review.opendev.org/c/openstack/nova/+/792116 | |
| 12:24:53 | opendevreview | Vlad Gusev proposed openstack/nova stable/train: Ignore PCI devices with 32bit domain https://review.opendev.org/c/openstack/nova/+/792117 | |
| 12:25:26 | opendevreview | Vlad Gusev proposed openstack/nova stable/stein: Reproduce bug 1897528 https://review.opendev.org/c/openstack/nova/+/816656 | |
| 12:26:16 | opendevreview | Vlad Gusev proposed openstack/nova stable/stein: Reproduce bug 1897528 https://review.opendev.org/c/openstack/nova/+/816656 | |
| 12:37:35 | opendevreview | Vlad Gusev proposed openstack/nova stable/stein: Reproduce bug 1897528 https://review.opendev.org/c/openstack/nova/+/816656 | |
| 12:37:36 | opendevreview | Vlad Gusev proposed openstack/nova stable/stein: Ignore PCI devices with 32bit domain https://review.opendev.org/c/openstack/nova/+/816682 | |
| 13:20:08 | opendevreview | Vlad Gusev proposed openstack/nova stable/stein: Ignore PCI devices with 32bit domain https://review.opendev.org/c/openstack/nova/+/816682 | |
| 13:24:49 | opendevreview | Merged openstack/nova master: compute: Update volume_id within connection_info during swap_volume https://review.opendev.org/c/openstack/nova/+/807025 | |
| 13:24:57 | opendevreview | Merged openstack/nova master: fup: Move _wait_for_volume_{attach,detach} to os-volume_attachments https://review.opendev.org/c/openstack/nova/+/810775 | |
| 13:25:06 | opendevreview | Merged openstack/nova master: fup: Refactor and simplify Cinder fixture GET volume mock https://review.opendev.org/c/openstack/nova/+/810776 | |
| 14:20:30 | opendevreview | Merged openstack/nova master: Clean up allocations left by evacuation when deleting service https://review.opendev.org/c/openstack/nova/+/778696 | |
| 14:20:40 | opendevreview | Merged openstack/nova stable/wallaby: Reproduce bug 1944759 https://review.opendev.org/c/openstack/nova/+/810912 | |
| 14:53:26 | gibi | lyarwood: hi! it seems there is a variant of https://bugs.launchpad.net/nova/+bug/1931702 in https://zuul.opendev.org/t/openstack/build/582935ad35a348cf89dcb25bdc3be0ea/logs But the guest console log at volume detach is different now https://zuul.opendev.org/t/openstack/build/582935ad35a348cf89dcb25bdc3be0ea/log/controller/logs/tempest_log.txt#5444 | |
| 14:53:31 | gibi | elodilles: ^^ | |
| 14:53:46 | gibi | "[ 15.981709] virtio_blk virtio4: req.0:id 4 is not a head!" | |
| 14:54:05 | gibi | lyarwood: does it ring a bell for you? | |
| 14:56:31 | lyarwood | gibi: no I've not seen that before tbh | |
| 14:56:46 | gibi | lyarwood: ack, thanks | |
| 15:02:23 | artom_ | bauzas, hey, I think the Ironic folks would be really happy if we made https://review.opendev.org/c/openstack/nova/+/813263 a review priority... | |
| 15:03:20 | sean-k-mooney | i see | |
| 15:04:48 | sean-k-mooney | i think should be safe although it raise the question about oter life cyle events liek power on power off and had/soft reboot | |
| 15:05:02 | artom | sean-k-mooney, the Ironic patch? | |
| 15:05:25 | sean-k-mooney | yes | |
| 15:05:29 | artom | Yeah, I suppose it does, but from what I've seen, use of plug_vifs() is highly limited, so it's safe to make it a noop | |
| 15:05:44 | sean-k-mooney | no its not | |
| 15:05:59 | sean-k-mooney | we need to call it for the inial spawn | |
| 15:06:02 | artom | sean-k-mooney, I mean, look at my review notes inline, and tell me if I've missed something :) | |
| 15:06:20 | sean-k-mooney | we do not need to call it in init_host for ironci | |
| 15:06:34 | sean-k-mooney | but it cant jsut be a noop without change the spwan workflow | |
| 15:07:06 | sean-k-mooney | we use it on inital boot to ensure that the networkign if fully configured by the backend before we power on the ironic host | |
| 15:07:27 | artom | sean-k-mooney, maybe you're thinking of a slightly differently named method? | |
| 15:07:35 | sean-k-mooney | no im not | |
| 15:08:00 | artom | In the compute manager, it's only called from _init_instance(), which is only called from init_host() | |
| 15:08:05 | artom | That's it, nothing on spawn | |
| 15:08:14 | sean-k-mooney | correct its not | |
| 15:08:25 | sean-k-mooney | but we also call plug_vifs during spwan | |
| 15:08:31 | sean-k-mooney | so you cant just make plug_vifs a noop | |
| 15:09:03 | artom | From where? | |
| 15:09:06 | sean-k-mooney | it will mean during spawn we will not actully set up the networking proerly they have hacked around this here https://review.opendev.org/c/openstack/nova/+/813263/3/nova/virt/ironic/driver.py#1606 | |
| 15:09:28 | sean-k-mooney | by starting to use _plug_vifs to actully invoke the ironic api | |