| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-03-01 | |||
| 15:35:55 | dansmith | the host won't even use the ssh key to try to ssh to a remote if it's not one of the allowed key types | |
| 15:36:05 | dansmith | the error messaging for this has been terrible | |
| 15:36:14 | dansmith | because it shows that it's trying the key, but it's not | |
| 15:36:21 | dansmith | I struggled with this when I upgraded myself | |
| 15:36:25 | dansmith | yeah | |
| 15:42:58 | bauzas | dansmith: -1 on https://review.opendev.org/c/openstack/grenade/+/875940 | |
| 15:43:12 | bauzas | IMHO, you need to specify another keytype to be safe | |
| 15:43:26 | bauzas | I mean another algorithm | |
| 15:43:42 | dansmith | that's been the default for years now right? | |
| 15:43:56 | bauzas | https://transang.me/ssh-handshake-is-rejected-with-no-mutual-signature-algorithm-error/ | |
| 15:44:03 | dansmith | oh, maybe not: | |
| 15:44:14 | dansmith | "If invoked without any arguments, ssh-keygen will gen‐ | |
| 15:44:15 | bauzas | no, I think we continue defaulting to ssh-rsa | |
| 15:44:16 | dansmith | erate an RSA key." | |
| 15:44:24 | dansmith | how stupid, when ssh will reject the default | |
| 15:44:29 | dansmith | ffs :) | |
| 15:44:53 | bauzas | another option on the client side is to invoke to accept the key | |
| 15:45:00 | bauzas | +o PubkeyAcceptedKeyTypes +ssh-rsa | |
| 15:45:04 | bauzas | when you ssh | |
| 15:45:15 | dansmith | right I mentioned that above as another option | |
| 15:45:19 | bauzas | or add it in your ssh config | |
| 15:45:39 | bauzas | but I'd prefer to switch to the latest ed25519 if not edcsa | |
| 15:45:41 | dansmith | but since we're already using a deprecated thing, it seems better to just move to what people should be doing, and since this wasn't generated by the old devstack, | |
| 15:45:48 | bauzas | yeah | |
| 15:45:49 | dansmith | it's not an upgrade-y thing like I mentioned | |
| 15:45:56 | bauzas | agreed | |
| 15:48:30 | opendevreview | Jorge San Emeterio proposed openstack/nova master: WIP: Have schema for 'lock' action be applied to all microversions. https://review.opendev.org/c/openstack/nova/+/875653 | |
| 15:55:50 | opendevreview | Jorge San Emeterio proposed openstack/nova master: WIP: Have schema for 'lock' action be applied to all microversions. https://review.opendev.org/c/openstack/nova/+/875653 | |
| 15:58:16 | opendevreview | Jorge San Emeterio proposed openstack/nova master: WIP: Have schema for 'lock' action be applied to all microversions. https://review.opendev.org/c/openstack/nova/+/875653 | |
| 15:59:10 | bauzas | gmann: fwiw, I added the new RBAC policies mention in the prelude https://review.opendev.org/c/openstack/nova/+/875380/3/releasenotes/notes/antelope-prelude-4a99907b00e739f8.yaml#49 | |
| 15:59:17 | bauzas | feel free to vote on it | |
| 15:59:53 | bauzas | gmann: also, I forgot about it because the original change wasn't correctly tracked on the nova side : no blueprint at least | |
| 15:59:55 | bauzas | hence my miss | |
| 16:10:14 | clarkb | bauzas: dansmith: RSA is fine with openssh. THe issue is openssh + rsa + sha1 | |
| 16:10:39 | bauzas | I see | |
| 16:10:49 | bauzas | ssh-rsa | |
| 16:10:53 | dansmith | clarkb: oh, so rsa with ssh-keygen that uses sha256 would be fine/ | |
| 16:11:13 | bauzas | by default ssh-rsa uses sha1 yes | |
| 16:11:24 | clarkb | dansmith: at keygen time the sha version isn't a concern. It is negotiated at connection time | |
| 16:11:42 | clarkb | but ya ssh-rsa is ssh + sha1. rsa-sha-256 or whatever the name is is the sha2 variant | |
| 16:12:03 | dansmith | okay I'm confused then | |
| 16:12:18 | bauzas | clarkb: I -1 on https://review.opendev.org/c/openstack/grenade/+/875940/1/projects/70_cinder/resources.sh#121 | |
| 16:12:29 | clarkb | for a concrete example gerri's sshd didn't know how to negotiate rsa + sha2 with clients until recently (we opendev addressed that and now it works). This broke fedora who deprecated rsa + sha1 before everyone else | |
| 16:12:34 | bauzas | clarkb: do you think it would be better to rather use rsa with sha256 ? | |
| 16:12:54 | clarkb | today fedora users can use those same exact rsa keys to talk to gerrit because the gerrit sshd will now negotiate with the fedora ssh client to use sha2 | |
| 16:13:10 | clarkb | bauzas: if ou want to use rsa then ya that is he non deprecated option | |
| 16:13:49 | clarkb | one consideration here is fips testing. fips doesn't allow ed25519 keys. Which means you're stuck using ecdsa or rsa | |
| 16:14:04 | clarkb | Personally when all this started becoming a problem I switched my personal things to ed25519 as I don't care about fips at home | |
| 16:14:32 | bauzas | clarkb: ok, then your preference would be "-t rsa -E sha256 " correct ? | |
| 16:14:39 | bauzas | for ssh-keygen | |
| 16:14:51 | clarkb | bauzas: the sha version shouldn't matter at keygen time its all dynamic protocol negoatiation | |
| 16:15:47 | clarkb | bauzas: is this talking to cirros? | |
| 16:16:22 | clarkb | if so cirros' dropbear server may not support sha2 negotiation which would be a similar problem to what we addressed in Gerrit | |
| 16:17:12 | bauzas | clarkb: context https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f59/875773/3/check/grenade-skip-level-continuous/f597bdf/controller/logs/grenade.sh_log.txt | |
| 16:17:49 | bauzas | clarkb: coming from https://github.com/openstack/grenade/blob/master/projects/70_cinder/resources.sh#L186 failing | |
| 16:18:01 | clarkb | ya thats cirros. And the test node must be jammy? | |
| 16:18:02 | bauzas | we generate ssh-rsa keys by default with jammy | |
| 16:18:07 | bauzas | yup | |
| 16:18:46 | clarkb | I think your options are: use newer cirros with newer dropbear, use ecdsa/ed25519 key if dropbear supports these, explicitly override configs to allow rsa + sha1 on the jammy side to talk to dropbear that only knows this combo | |
| 16:19:49 | clarkb | Part of the problem here is that when everyone deprecated sha1 they kept the fallback default in protocol negotiation as rsa + sha1 despite knowing it will not work. It would've been better if they updated the fallback default to be rsa + sha2 which would have worked with gerrit its only problem was understanding how to negotiate things, not lack of sha2 support | |
| 16:21:39 | clarkb | dropbear version 2020.79 added support for both ed25519 keys and rsa sha2 | |
| 16:22:40 | clarkb | the easiest thing is probably using ecdsa keys and that would also satisfiy fips | |
| 16:27:53 | bauzas | clarkb: cool, then | |
| 16:33:07 | opendevreview | Alexey Stupnikov proposed openstack/nova stable/train: reenable greendns in nova. https://review.opendev.org/c/openstack/nova/+/833438 | |
| 17:08:21 | gmann | bauzas: thanks for RBAC things in prelude. I will check | |
| 17:08:29 | bauzas | cool | |
| 17:09:54 | dansmith | clarkb: okay, so you're saying the different key type ends up using a different hash algo in the negotiation and that's what sidesteps the problem? | |
| 17:09:58 | dansmith | I thought it was the key type itself | |
| 17:10:15 | clarkb | dansmith: correct | |
| 17:10:22 | dansmith | hrm | |
| 17:10:27 | clarkb | all the newer keys use hashes that are secure enough to have not been replaced | |
| 17:10:35 | clarkb | rsa didn't so has extra negotiation requirements when setting up a connection | |
| 17:10:46 | dansmith | but I thought you said the hash wasn't part of the key? | |
| 17:10:48 | clarkb | its the key exchange extensions bit of the ssh protocol | |
| 17:10:59 | clarkb | sorry correct to the first statement not to the second | |
| 17:11:10 | clarkb | this is not part of the key itself. It is part of how the key is used at connection time | |
| 17:11:15 | dansmith | because I thought even ssh'ing between two newer machines still fails with the old key | |
| 17:11:31 | clarkb | it shouldn't | |
| 17:12:23 | dansmith | that hasn't been my experience, but perhaps I was just missing what the linkage was | |
| 17:13:01 | clarkb | I think cirros + dropbear and gerrit + mina sshd may have created confusion around that? But new openssh to new opensshd should be fine | |
| 17:13:22 | dansmith | I had to enable two things over the course of the last few years: | |
| 17:13:23 | dansmith | PubkeyAcceptedKeyTypes +ssh-rsa | |
| 17:13:29 | dansmith | and | |
| 17:13:30 | dansmith | KexAlgorithms +diffie-hellman-group1-sha1,diffie-hellman-group14-sha1 | |
| 17:14:38 | dansmith | seems like the latter must be the part you're talking about | |
| 17:14:57 | dansmith | but at some point, something (one of my systems, which might have been macos) stopped even considering rsa keys | |
| 17:17:00 | dansmith | although some docs make the PubkeyAcceptedTypes sound like maybe it's not the key type? I dunno, it's just confusing (to me) :) | |
| 17:26:21 | bauzas | iiuc, it's a matter of negotiation like clarkb said | |
| 17:26:44 | bauzas | the current interim solution allows the existing keys to continue to work by asking the server to negociate with them | |
| 17:27:06 | bauzas | later, "they" could drop support of those keys and then the pubkeyacceptedtypes hint wouldn't work | |
| 17:27:43 | bauzas | the idea of that option hint is to leave a graceful period to ops to generate new keys and deliver their public ones into the servers | |
| 17:28:14 | bauzas | but like every other migration helper, people will rather use the workaround instead of migrating their keys | |
| 17:28:26 | bauzas | until the full removal arrives | |
| 17:35:24 | dansmith | bauzas: you see that documented somewhere/ | |
| 17:35:39 | dansmith | because if I read what clarkb is saying, there's nothing wrong with rsa keys themselves... | |
| 17:35:52 | bauzas | well, I need to remember that situation | |
| 17:36:20 | dansmith | back in the day, I migrated all my keys to dsa because that was "the way" and then dsa was dropped and I had to migrate back | |
| 17:36:26 | dansmith | so forgive me if I'm a bit skeptical :D | |
| 17:37:16 | clarkb | correct you should not need to generate new keys to fix this | |
| 17:37:28 | clarkb | its purely a I'm making a connection we need to negotiate details and can't agree on those details problem | |