| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-03-01 | |||
| 15:13:34 | sean-k-mooney | fedora reject rsa keys. osc will only generate rsa keys because that all nova does | |
| 15:13:56 | sean-k-mooney | so unless its gnereated in tempest itself it would fail if jammy rejects them | |
| 15:14:17 | sean-k-mooney | with that said im expecting that job to still use tempest master right | |
| 15:14:29 | dansmith | this is pre-tempest, this is grenade resource creation | |
| 15:14:30 | sean-k-mooney | we dont pin it or anything strange for grenade | |
| 15:14:57 | bauzas | dansmith: lemme look at your change | |
| 15:15:19 | dansmith | sean-k-mooney: https://github.com/openstack/grenade/blob/master/projects/70_cinder/resources.sh#L186 | |
| 15:15:29 | dansmith | sean-k-mooney: https://github.com/openstack/grenade/blob/master/projects/70_cinder/resources.sh#L121 | |
| 15:15:48 | dansmith | and when we try to ssh: | |
| 15:15:49 | dansmith | 2023-02-28 20:55:58.894 | debug1: Will attempt key: /opt/stack/save/cinder_key.pem explicit | |
| 15:15:49 | dansmith | 2023-02-28 20:55:58.897 | debug1: SSH2_MSG_SERVICE_ACCEPT received | |
| 15:15:59 | dansmith | ahh | |
| 15:16:04 | dansmith | 2023-02-28 20:55:58.902 | sign_and_send_pubkey: no mutual signature supported | |
| 15:16:06 | dansmith | there ^ | |
| 15:16:52 | dansmith | but surely grenade is running on jammy nodes already, so I'm not sure why it wouldn't have been configured to support the old keys | |
| 15:18:28 | sean-k-mooney | dansmith: proably not | |
| 15:18:42 | sean-k-mooney | for antelop i think it used focal | |
| 15:18:48 | dansmith | looks like not yeah | |
| 15:19:00 | dansmith | okay, well, I can fix that then I guess | |
| 15:19:08 | sean-k-mooney | so likely we jsut need to call ssh-keygen directly | |
| 15:19:12 | sean-k-mooney | instead of osc | |
| 15:19:33 | bauzas | wait | |
| 15:19:36 | dansmith | well, I was going to configure the host to allow that key type | |
| 15:19:38 | bauzas | https://github.com/openstack/grenade/blob/master/projects/70_cinder/resources.sh#L121 | |
| 15:19:47 | bauzas | we no longer support this by Zed | |
| 15:20:11 | bauzas | oh no, sorry | |
| 15:20:14 | dansmith | bauzas: but it works if you use the old microversion, which osc is doing right? | |
| 15:20:17 | bauzas | you're passing the key | |
| 15:20:32 | bauzas | dansmith: ah right too | |
| 15:20:41 | dansmith | also that :) | |
| 15:20:48 | bauzas | but I wonder, that's probably why we have a problem with rsa | |
| 15:21:14 | bauzas | could we maybe try to create ed keys ? | |
| 15:22:21 | dansmith | I mean, I guess, but it seems more upgrade-y to allow the use of the older key type during the upgrade | |
| 15:23:17 | dansmith | hmm, that keypair create *is* generating the key in nova right? | |
| 15:23:26 | dansmith | $CINDER_KEY is just the key name | |
| 15:24:04 | bauzas | yeah | |
| 15:24:19 | bauzas | and by default it creates a ssh-rsa key | |
| 15:24:22 | dansmith | looks like that's the only place we run keypair create, so maybe it's best to just generate it properly | |
| 15:24:33 | bauzas | but you're creating it in Yoga, right? | |
| 15:24:38 | bauzas | or in Zed ? | |
| 15:25:23 | dansmith | it's master grenade | |
| 15:25:49 | bauzas | against a master devstack then ? | |
| 15:25:56 | bauzas | oh | |
| 15:26:35 | bauzas | it defaults to the previous release before upgrading | |
| 15:26:37 | bauzas | right? | |
| 15:26:50 | dansmith | not a master devstack | |
| 15:26:52 | bauzas | I mean the grenade base | |
| 15:27:16 | dansmith | at this point in the phase, we've run devstack from yoga, we're running grenade from master to do some things against it, then upgrade to devstack master | |
| 15:27:53 | bauzas | https://github.com/openstack/grenade/blob/master/grenaderc#L36 | |
| 15:28:05 | bauzas | ok | |
| 15:28:07 | bauzas | so | |
| 15:28:18 | bauzas | we created a yoga devstack, we're running grenade against it | |
| 15:28:23 | bauzas | which will create a key | |
| 15:28:39 | bauzas | and after this, we would upgrade to devstack master | |
| 15:28:58 | bauzas | so the environment that OSC calls is a Yoga nova-API service | |
| 15:29:06 | bauzas | amirite ? | |
| 15:29:22 | dansmith | yes | |
| 15:31:29 | dansmith | https://review.opendev.org/c/openstack/grenade/+/875940 | |
| 15:31:35 | dansmith | that ^ is what I'm thinking | |
| 15:32:47 | bauzas | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f59/875773/3/check/grenade-skip-level-continuous/f597bdf/controller/logs/grenade.sh_log.txt | |
| 15:32:53 | bauzas | I see the problem | |
| 15:32:58 | bauzas | but this is using jammy | |
| 15:33:03 | opendevreview | Dan Smith proposed openstack/nova master: Add continuous skip-level job for nova https://review.opendev.org/c/openstack/nova/+/875773 | |
| 15:33:15 | bauzas | so I guess ssh-rsa keys are no longer accepted | |
| 15:33:33 | bauzas | since you created them with yoga, it was accepted by Nova | |
| 15:33:43 | bauzas | and the guest was accordingly created | |
| 15:34:01 | dansmith | I don't think it has anything to do with yoga, | |
| 15:34:16 | dansmith | because even if we were running on jammy, we'd still be able to generate because of the old microversion right>? | |
| 15:34:46 | bauzas | you're true | |
| 15:34:48 | bauzas | nothing related | |
| 15:35:03 | bauzas | we continue to accept to blindly create ssh-rsa keys by default | |
| 15:35:08 | bauzas | if you don't opt(in | |
| 15:35:23 | bauzas | but, jammy guests should no longer accept it | |
| 15:35:27 | bauzas | lemme verify this | |
| 15:35:38 | dansmith | it's not jammy guests, it's hosts | |
| 15:35:51 | bauzas | oooh, then I understand | |
| 15:35:55 | dansmith | the host won't even use the ssh key to try to ssh to a remote if it's not one of the allowed key types | |
| 15:36:05 | dansmith | the error messaging for this has been terrible | |
| 15:36:14 | dansmith | because it shows that it's trying the key, but it's not | |
| 15:36:21 | dansmith | I struggled with this when I upgraded myself | |
| 15:36:25 | dansmith | yeah | |
| 15:42:58 | bauzas | dansmith: -1 on https://review.opendev.org/c/openstack/grenade/+/875940 | |
| 15:43:12 | bauzas | IMHO, you need to specify another keytype to be safe | |
| 15:43:26 | bauzas | I mean another algorithm | |
| 15:43:42 | dansmith | that's been the default for years now right? | |
| 15:43:56 | bauzas | https://transang.me/ssh-handshake-is-rejected-with-no-mutual-signature-algorithm-error/ | |
| 15:44:03 | dansmith | oh, maybe not: | |
| 15:44:14 | dansmith | "If invoked without any arguments, ssh-keygen will gen‐ | |
| 15:44:15 | bauzas | no, I think we continue defaulting to ssh-rsa | |
| 15:44:16 | dansmith | erate an RSA key." | |
| 15:44:24 | dansmith | how stupid, when ssh will reject the default | |
| 15:44:29 | dansmith | ffs :) | |
| 15:44:53 | bauzas | another option on the client side is to invoke to accept the key | |
| 15:45:00 | bauzas | +o PubkeyAcceptedKeyTypes +ssh-rsa | |
| 15:45:04 | bauzas | when you ssh | |
| 15:45:15 | dansmith | right I mentioned that above as another option | |
| 15:45:19 | bauzas | or add it in your ssh config | |
| 15:45:39 | bauzas | but I'd prefer to switch to the latest ed25519 if not edcsa | |
| 15:45:41 | dansmith | but since we're already using a deprecated thing, it seems better to just move to what people should be doing, and since this wasn't generated by the old devstack, | |
| 15:45:48 | bauzas | yeah | |