| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-11-04 | |||
| 10:57:26 | lyarwood | EugenMayer: regarding 2021-11-04 09:18:39.933 13300 INFO nova.console.websocketproxy [-] 10.0.1.1: SSL connection but '/self.pem' not found | |
| 10:58:12 | lyarwood | Yeah actually that can only be from the controller | |
| 11:02:42 | EugenMayer | lyarwood ok so i check the noproxy container and how the certificates are deployed and configured | |
| 11:05:04 | lyarwood | https://github.com/novnc/websockify/blob/master/README.md#encrypted-websocket-connections-wss - FWIW self.pem is the default cert websockify will try to load when it's asked to use SSL | |
| 11:06:01 | lyarwood | as I said before https://github.com/openstack/nova/blob/909cfc76369b94b026cf42b86fb5a310dce21a8c/nova/cmd/baseproxy.py#L48-L87 is where Nova tries to provide the correct cert when launching the novncproxy service | |
| 11:07:39 | EugenMayer | well ok now i guess that will be the issue. is self.pem a cert+private-key format? | |
| 11:08:08 | EugenMayer | ah --cert --key | |
| 11:10:18 | lyarwood | right you should already have these in your env? | |
| 11:10:24 | lyarwood | just under a different filename | |
| 11:10:34 | lyarwood | so just update the nova.conf used by the service to point to them | |
| 11:10:42 | lyarwood | cert=/path/to/cert | |
| 11:10:48 | lyarwood | key=/path/to/key | |
| 11:11:07 | lyarwood | and again, DEFAULT namespace so outside of the [vnc] section etc. | |
| 11:11:26 | EugenMayer | checking the ansible tasks right now (kollas) | |
| 11:16:35 | lyarwood | yeah it doesn't look like it has support tbh | |
| 11:16:43 | EugenMayer | https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/nova-cell/templates/nova.conf.j2 | |
| 11:16:45 | lyarwood | looking at the config templates at least | |
| 11:16:46 | EugenMayer | yes, it's mising | |
| 11:17:46 | EugenMayer | lyarwood did the nova implemenation of novnc change since victoria? | |
| 11:18:34 | lyarwood | I don't think anything has that would change this behaviour tbh | |
| 11:18:45 | EugenMayer | in other words, looking back, that template never had cert/key as values set for TLS | |
| 11:19:04 | EugenMayer | so either it has never been supported at all - or it is a regression | |
| 11:20:17 | lyarwood | yeah I would assume this has never been supported by Kolla tbh, should be pretty trivial to correct however | |
| 11:21:22 | EugenMayer | yes it is just PITA to search for that, i you are clueless (like i'am). You never know what is supposed to work and what not, and how a working configuration does look like | |
| 11:22:59 | EugenMayer | lyarwood any idea what the path of self.pem looks like? i mean i do not assume /self.pem is really absolute here | |
| 11:24:33 | lyarwood | https://github.com/openstack/nova/blob/909cfc76369b94b026cf42b86fb5a310dce21a8c/nova/conf/novnc.py#L41-L52 looks like it's relative so it depends how kolla is launching the service | |
| 11:24:56 | lyarwood | but again updating the nova.conf used by the service to point to your actual key and cert is a better option here | |
| 11:25:24 | EugenMayer | yes sure, it is a little more work then that | |
| 11:26:00 | EugenMayer | i will need to volume-mount the certs first, i cannot just docker cp them, or they will be lost on upgrade. Then a config override for nova conf (conditional) and then mounting that certs | |
| 11:26:25 | EugenMayer | but still, HUGE help lyarwood, i should be able to handle the rest. Thank you big times! | |
| 11:26:37 | lyarwood | np good luck :) | |
| 11:26:49 | EugenMayer | thank you sir | |
| 12:24:29 | opendevreview | Vlad Gusev proposed openstack/nova stable/train: Reproduce bug 1897528 https://review.opendev.org/c/openstack/nova/+/792116 | |
| 12:24:53 | opendevreview | Vlad Gusev proposed openstack/nova stable/train: Ignore PCI devices with 32bit domain https://review.opendev.org/c/openstack/nova/+/792117 | |
| 12:25:26 | opendevreview | Vlad Gusev proposed openstack/nova stable/stein: Reproduce bug 1897528 https://review.opendev.org/c/openstack/nova/+/816656 | |
| 12:26:16 | opendevreview | Vlad Gusev proposed openstack/nova stable/stein: Reproduce bug 1897528 https://review.opendev.org/c/openstack/nova/+/816656 | |
| 12:37:35 | opendevreview | Vlad Gusev proposed openstack/nova stable/stein: Reproduce bug 1897528 https://review.opendev.org/c/openstack/nova/+/816656 | |
| 12:37:36 | opendevreview | Vlad Gusev proposed openstack/nova stable/stein: Ignore PCI devices with 32bit domain https://review.opendev.org/c/openstack/nova/+/816682 | |
| 13:20:08 | opendevreview | Vlad Gusev proposed openstack/nova stable/stein: Ignore PCI devices with 32bit domain https://review.opendev.org/c/openstack/nova/+/816682 | |
| 13:24:49 | opendevreview | Merged openstack/nova master: compute: Update volume_id within connection_info during swap_volume https://review.opendev.org/c/openstack/nova/+/807025 | |
| 13:24:57 | opendevreview | Merged openstack/nova master: fup: Move _wait_for_volume_{attach,detach} to os-volume_attachments https://review.opendev.org/c/openstack/nova/+/810775 | |
| 13:25:06 | opendevreview | Merged openstack/nova master: fup: Refactor and simplify Cinder fixture GET volume mock https://review.opendev.org/c/openstack/nova/+/810776 | |
| 14:20:30 | opendevreview | Merged openstack/nova master: Clean up allocations left by evacuation when deleting service https://review.opendev.org/c/openstack/nova/+/778696 | |
| 14:20:40 | opendevreview | Merged openstack/nova stable/wallaby: Reproduce bug 1944759 https://review.opendev.org/c/openstack/nova/+/810912 | |
| 14:53:26 | gibi | lyarwood: hi! it seems there is a variant of https://bugs.launchpad.net/nova/+bug/1931702 in https://zuul.opendev.org/t/openstack/build/582935ad35a348cf89dcb25bdc3be0ea/logs But the guest console log at volume detach is different now https://zuul.opendev.org/t/openstack/build/582935ad35a348cf89dcb25bdc3be0ea/log/controller/logs/tempest_log.txt#5444 | |
| 14:53:31 | gibi | elodilles: ^^ | |
| 14:53:46 | gibi | "[ 15.981709] virtio_blk virtio4: req.0:id 4 is not a head!" | |
| 14:54:05 | gibi | lyarwood: does it ring a bell for you? | |
| 14:56:31 | lyarwood | gibi: no I've not seen that before tbh | |
| 14:56:46 | gibi | lyarwood: ack, thanks | |
| 15:02:23 | artom_ | bauzas, hey, I think the Ironic folks would be really happy if we made https://review.opendev.org/c/openstack/nova/+/813263 a review priority... | |
| 15:03:20 | sean-k-mooney | i see | |
| 15:04:48 | sean-k-mooney | i think should be safe although it raise the question about oter life cyle events liek power on power off and had/soft reboot | |
| 15:05:02 | artom | sean-k-mooney, the Ironic patch? | |
| 15:05:25 | sean-k-mooney | yes | |
| 15:05:29 | artom | Yeah, I suppose it does, but from what I've seen, use of plug_vifs() is highly limited, so it's safe to make it a noop | |
| 15:05:44 | sean-k-mooney | no its not | |
| 15:05:59 | sean-k-mooney | we need to call it for the inial spawn | |
| 15:06:02 | artom | sean-k-mooney, I mean, look at my review notes inline, and tell me if I've missed something :) | |
| 15:06:20 | sean-k-mooney | we do not need to call it in init_host for ironci | |
| 15:06:34 | sean-k-mooney | but it cant jsut be a noop without change the spwan workflow | |
| 15:07:06 | sean-k-mooney | we use it on inital boot to ensure that the networkign if fully configured by the backend before we power on the ironic host | |
| 15:07:27 | artom | sean-k-mooney, maybe you're thinking of a slightly differently named method? | |
| 15:07:35 | sean-k-mooney | no im not | |
| 15:08:00 | artom | In the compute manager, it's only called from _init_instance(), which is only called from init_host() | |
| 15:08:05 | artom | That's it, nothing on spawn | |
| 15:08:14 | sean-k-mooney | correct its not | |
| 15:08:25 | sean-k-mooney | but we also call plug_vifs during spwan | |
| 15:08:31 | sean-k-mooney | so you cant just make plug_vifs a noop | |
| 15:09:03 | artom | From where? | |
| 15:09:06 | sean-k-mooney | it will mean during spawn we will not actully set up the networking proerly they have hacked around this here https://review.opendev.org/c/openstack/nova/+/813263/3/nova/virt/ironic/driver.py#1606 | |
| 15:09:28 | sean-k-mooney | by starting to use _plug_vifs to actully invoke the ironic api | |
| 15:09:34 | sean-k-mooney | for interface attach | |
| 15:10:23 | artom | That's just inlining what plug_vifs() used to do into attach(), no? | |
| 15:10:35 | bauzas | artom: I can mark it as a Review-Priority for me | |
| 15:11:40 | artom | bauzas, PTL's discretion and all that :) I was just making a request | |
| 15:12:00 | bauzas | any core can set this flag | |
| 15:12:08 | bauzas | ... for the moment | |
| 15:12:18 | artom | But you're the core-iest of cores | |
| 15:12:19 | bauzas | I'm about to write a doc change for it, hopefully tomorow | |
| 15:12:47 | bauzas | artom: nah, as sean-k-mooney said, I'm just a "cat herder" or if you prefer, some French guy yelling in the wind | |
| 15:13:05 | gibi | sean-k-mooney, artom: I read that ironic related nova patch, I see that it is correct and does not affect spawn, but now I'm affraid that sean-k-mooney has things I'm missing | |
| 15:13:07 | artom | Meow. | |
| 15:13:19 | artom | gibi, you and me both | |
| 15:13:24 | gibi | we need more cats | |
| 15:13:33 | bauzas | I have a dog | |
| 15:13:39 | gibi | then you are out | |
| 15:13:41 | gibi | :P | |
| 15:13:49 | bauzas | cats are selfish | |
| 15:13:50 | artom | Ionesco says dogs are cats | |
| 15:13:56 | sean-k-mooney | gibi: im mostly uncofrotable with changing the meaing of plug_vifs to be honest | |
| 15:13:59 | artom | He also says that Socrates was a cat | |
| 15:14:12 | sean-k-mooney | im currently reviewing the spawn path | |
| 15:14:22 | artom | sean-k-mooney, I don't think that's our call to make, every driver can do what they want | |
| 15:14:26 | bauzas | artom: the only merit to cats is that they help to prove some theorem | |
| 15:14:40 | bauzas | about quantic nature | |
| 15:15:30 | artom | Only if they're in boxes | |
| 15:16:58 | sean-k-mooney | artom: its used here https://github.com/openstack/nova/blob/fded762f4df26ff5706438a66da33ff966f833c6/nova/virt/ironic/driver.py#L1910 | |
| 15:17:04 | sean-k-mooney | which is used by the compute manger here https://github.com/openstack/nova/blob/fded762f4df26ff5706438a66da33ff966f833c6/nova/compute/manager.py#L2597 | |
| 15:17:17 | sean-k-mooney | in _build_resources | |