| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-03-01 | |||
| 15:25:23 | dansmith | it's master grenade | |
| 15:25:49 | bauzas | against a master devstack then ? | |
| 15:25:56 | bauzas | oh | |
| 15:26:35 | bauzas | it defaults to the previous release before upgrading | |
| 15:26:37 | bauzas | right? | |
| 15:26:50 | dansmith | not a master devstack | |
| 15:26:52 | bauzas | I mean the grenade base | |
| 15:27:16 | dansmith | at this point in the phase, we've run devstack from yoga, we're running grenade from master to do some things against it, then upgrade to devstack master | |
| 15:27:53 | bauzas | https://github.com/openstack/grenade/blob/master/grenaderc#L36 | |
| 15:28:05 | bauzas | ok | |
| 15:28:07 | bauzas | so | |
| 15:28:18 | bauzas | we created a yoga devstack, we're running grenade against it | |
| 15:28:23 | bauzas | which will create a key | |
| 15:28:39 | bauzas | and after this, we would upgrade to devstack master | |
| 15:28:58 | bauzas | so the environment that OSC calls is a Yoga nova-API service | |
| 15:29:06 | bauzas | amirite ? | |
| 15:29:22 | dansmith | yes | |
| 15:31:29 | dansmith | https://review.opendev.org/c/openstack/grenade/+/875940 | |
| 15:31:35 | dansmith | that ^ is what I'm thinking | |
| 15:32:47 | bauzas | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f59/875773/3/check/grenade-skip-level-continuous/f597bdf/controller/logs/grenade.sh_log.txt | |
| 15:32:53 | bauzas | I see the problem | |
| 15:32:58 | bauzas | but this is using jammy | |
| 15:33:03 | opendevreview | Dan Smith proposed openstack/nova master: Add continuous skip-level job for nova https://review.opendev.org/c/openstack/nova/+/875773 | |
| 15:33:15 | bauzas | so I guess ssh-rsa keys are no longer accepted | |
| 15:33:33 | bauzas | since you created them with yoga, it was accepted by Nova | |
| 15:33:43 | bauzas | and the guest was accordingly created | |
| 15:34:01 | dansmith | I don't think it has anything to do with yoga, | |
| 15:34:16 | dansmith | because even if we were running on jammy, we'd still be able to generate because of the old microversion right>? | |
| 15:34:46 | bauzas | you're true | |
| 15:34:48 | bauzas | nothing related | |
| 15:35:03 | bauzas | we continue to accept to blindly create ssh-rsa keys by default | |
| 15:35:08 | bauzas | if you don't opt(in | |
| 15:35:23 | bauzas | but, jammy guests should no longer accept it | |
| 15:35:27 | bauzas | lemme verify this | |
| 15:35:38 | dansmith | it's not jammy guests, it's hosts | |
| 15:35:51 | bauzas | oooh, then I understand | |
| 15:35:55 | dansmith | the host won't even use the ssh key to try to ssh to a remote if it's not one of the allowed key types | |
| 15:36:05 | dansmith | the error messaging for this has been terrible | |
| 15:36:14 | dansmith | because it shows that it's trying the key, but it's not | |
| 15:36:21 | dansmith | I struggled with this when I upgraded myself | |
| 15:36:25 | dansmith | yeah | |
| 15:42:58 | bauzas | dansmith: -1 on https://review.opendev.org/c/openstack/grenade/+/875940 | |
| 15:43:12 | bauzas | IMHO, you need to specify another keytype to be safe | |
| 15:43:26 | bauzas | I mean another algorithm | |
| 15:43:42 | dansmith | that's been the default for years now right? | |
| 15:43:56 | bauzas | https://transang.me/ssh-handshake-is-rejected-with-no-mutual-signature-algorithm-error/ | |
| 15:44:03 | dansmith | oh, maybe not: | |
| 15:44:14 | dansmith | "If invoked without any arguments, ssh-keygen will gen‐ | |
| 15:44:15 | bauzas | no, I think we continue defaulting to ssh-rsa | |
| 15:44:16 | dansmith | erate an RSA key." | |
| 15:44:24 | dansmith | how stupid, when ssh will reject the default | |
| 15:44:29 | dansmith | ffs :) | |
| 15:44:53 | bauzas | another option on the client side is to invoke to accept the key | |
| 15:45:00 | bauzas | +o PubkeyAcceptedKeyTypes +ssh-rsa | |
| 15:45:04 | bauzas | when you ssh | |
| 15:45:15 | dansmith | right I mentioned that above as another option | |
| 15:45:19 | bauzas | or add it in your ssh config | |
| 15:45:39 | bauzas | but I'd prefer to switch to the latest ed25519 if not edcsa | |
| 15:45:41 | dansmith | but since we're already using a deprecated thing, it seems better to just move to what people should be doing, and since this wasn't generated by the old devstack, | |
| 15:45:48 | bauzas | yeah | |
| 15:45:49 | dansmith | it's not an upgrade-y thing like I mentioned | |
| 15:45:56 | bauzas | agreed | |
| 15:48:30 | opendevreview | Jorge San Emeterio proposed openstack/nova master: WIP: Have schema for 'lock' action be applied to all microversions. https://review.opendev.org/c/openstack/nova/+/875653 | |
| 15:55:50 | opendevreview | Jorge San Emeterio proposed openstack/nova master: WIP: Have schema for 'lock' action be applied to all microversions. https://review.opendev.org/c/openstack/nova/+/875653 | |
| 15:58:16 | opendevreview | Jorge San Emeterio proposed openstack/nova master: WIP: Have schema for 'lock' action be applied to all microversions. https://review.opendev.org/c/openstack/nova/+/875653 | |
| 15:59:10 | bauzas | gmann: fwiw, I added the new RBAC policies mention in the prelude https://review.opendev.org/c/openstack/nova/+/875380/3/releasenotes/notes/antelope-prelude-4a99907b00e739f8.yaml#49 | |
| 15:59:17 | bauzas | feel free to vote on it | |
| 15:59:53 | bauzas | gmann: also, I forgot about it because the original change wasn't correctly tracked on the nova side : no blueprint at least | |
| 15:59:55 | bauzas | hence my miss | |
| 16:10:14 | clarkb | bauzas: dansmith: RSA is fine with openssh. THe issue is openssh + rsa + sha1 | |
| 16:10:39 | bauzas | I see | |
| 16:10:49 | bauzas | ssh-rsa | |
| 16:10:53 | dansmith | clarkb: oh, so rsa with ssh-keygen that uses sha256 would be fine/ | |
| 16:11:13 | bauzas | by default ssh-rsa uses sha1 yes | |
| 16:11:24 | clarkb | dansmith: at keygen time the sha version isn't a concern. It is negotiated at connection time | |
| 16:11:42 | clarkb | but ya ssh-rsa is ssh + sha1. rsa-sha-256 or whatever the name is is the sha2 variant | |
| 16:12:03 | dansmith | okay I'm confused then | |
| 16:12:18 | bauzas | clarkb: I -1 on https://review.opendev.org/c/openstack/grenade/+/875940/1/projects/70_cinder/resources.sh#121 | |
| 16:12:29 | clarkb | for a concrete example gerri's sshd didn't know how to negotiate rsa + sha2 with clients until recently (we opendev addressed that and now it works). This broke fedora who deprecated rsa + sha1 before everyone else | |
| 16:12:34 | bauzas | clarkb: do you think it would be better to rather use rsa with sha256 ? | |
| 16:12:54 | clarkb | today fedora users can use those same exact rsa keys to talk to gerrit because the gerrit sshd will now negotiate with the fedora ssh client to use sha2 | |
| 16:13:10 | clarkb | bauzas: if ou want to use rsa then ya that is he non deprecated option | |
| 16:13:49 | clarkb | one consideration here is fips testing. fips doesn't allow ed25519 keys. Which means you're stuck using ecdsa or rsa | |
| 16:14:04 | clarkb | Personally when all this started becoming a problem I switched my personal things to ed25519 as I don't care about fips at home | |
| 16:14:32 | bauzas | clarkb: ok, then your preference would be "-t rsa -E sha256 " correct ? | |
| 16:14:39 | bauzas | for ssh-keygen | |
| 16:14:51 | clarkb | bauzas: the sha version shouldn't matter at keygen time its all dynamic protocol negoatiation | |
| 16:15:47 | clarkb | bauzas: is this talking to cirros? | |
| 16:16:22 | clarkb | if so cirros' dropbear server may not support sha2 negotiation which would be a similar problem to what we addressed in Gerrit | |
| 16:17:12 | bauzas | clarkb: context https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f59/875773/3/check/grenade-skip-level-continuous/f597bdf/controller/logs/grenade.sh_log.txt | |
| 16:17:49 | bauzas | clarkb: coming from https://github.com/openstack/grenade/blob/master/projects/70_cinder/resources.sh#L186 failing | |
| 16:18:01 | clarkb | ya thats cirros. And the test node must be jammy? | |
| 16:18:02 | bauzas | we generate ssh-rsa keys by default with jammy | |
| 16:18:07 | bauzas | yup | |
| 16:18:46 | clarkb | I think your options are: use newer cirros with newer dropbear, use ecdsa/ed25519 key if dropbear supports these, explicitly override configs to allow rsa + sha1 on the jammy side to talk to dropbear that only knows this combo | |
| 16:19:49 | clarkb | Part of the problem here is that when everyone deprecated sha1 they kept the fallback default in protocol negotiation as rsa + sha1 despite knowing it will not work. It would've been better if they updated the fallback default to be rsa + sha2 which would have worked with gerrit its only problem was understanding how to negotiate things, not lack of sha2 support | |
| 16:21:39 | clarkb | dropbear version 2020.79 added support for both ed25519 keys and rsa sha2 | |
| 16:22:40 | clarkb | the easiest thing is probably using ecdsa keys and that would also satisfiy fips | |
| 16:27:53 | bauzas | clarkb: cool, then | |
| 16:33:07 | opendevreview | Alexey Stupnikov proposed openstack/nova stable/train: reenable greendns in nova. https://review.opendev.org/c/openstack/nova/+/833438 | |