| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-03-01 | |||
| 10:50:35 | opendevreview | OpenStack Release Bot proposed openstack/placement stable/2023.1: Update .gitreview for stable/2023.1 https://review.opendev.org/c/openstack/placement/+/875873 | |
| 10:50:38 | opendevreview | OpenStack Release Bot proposed openstack/placement stable/2023.1: Update TOX_CONSTRAINTS_FILE for stable/2023.1 https://review.opendev.org/c/openstack/placement/+/875874 | |
| 10:50:42 | opendevreview | OpenStack Release Bot proposed openstack/placement master: Update master for stable/2023.1 https://review.opendev.org/c/openstack/placement/+/875875 | |
| 10:58:35 | opendevreview | Jorge San Emeterio proposed openstack/nova master: WIP: Have schema for 'lock' action be applied to all microversions. https://review.opendev.org/c/openstack/nova/+/875653 | |
| 11:05:22 | bauzas | folks, placement-rc1 tag is done, master is now on Bobcat and Antelope is branched for Placement, I repeat, Antelope is branched for Placement | |
| 11:55:58 | gibi | bauzas: \0/ | |
| 12:00:02 | sean-k-mooney | a day ahead of schdule and all :P | |
| 12:00:08 | sean-k-mooney | but cool | |
| 12:01:06 | sean-k-mooney | the non client libs like os-vif and os-traits are also already branched | |
| 12:01:49 | sean-k-mooney | well at least os-vif is but i assume the same is true for the rest | |
| 13:00:34 | bauzas | sean-k-mooney: indeed, I approved https://review.opendev.org/c/openstack/releases/+/874450 | |
| 15:07:39 | dansmith | bauzas: so my skip-level job failed because we can't ssh to the guest | |
| 15:07:53 | dansmith | it's clearly working (reachable) but rejecting maybe the key or something | |
| 15:08:04 | dansmith | can you think of what about running on jammy would affect that? | |
| 15:12:03 | sean-k-mooney | could it be related to sha1/rsa keys | |
| 15:12:15 | sean-k-mooney | i think tempest is using ec keys now | |
| 15:12:24 | sean-k-mooney | but thats a guess | |
| 15:12:36 | dansmith | that's what I was thinking too, but we're generating the keys with osc | |
| 15:13:02 | dansmith | oh, but maybe jammy as the host refuses to use them? | |
| 15:13:30 | dansmith | I would think that if we needed a change, that'd be wrapped up in grenade already though... | |
| 15:13:34 | sean-k-mooney | fedora reject rsa keys. osc will only generate rsa keys because that all nova does | |
| 15:13:56 | sean-k-mooney | so unless its gnereated in tempest itself it would fail if jammy rejects them | |
| 15:14:17 | sean-k-mooney | with that said im expecting that job to still use tempest master right | |
| 15:14:29 | dansmith | this is pre-tempest, this is grenade resource creation | |
| 15:14:30 | sean-k-mooney | we dont pin it or anything strange for grenade | |
| 15:14:57 | bauzas | dansmith: lemme look at your change | |
| 15:15:19 | dansmith | sean-k-mooney: https://github.com/openstack/grenade/blob/master/projects/70_cinder/resources.sh#L186 | |
| 15:15:29 | dansmith | sean-k-mooney: https://github.com/openstack/grenade/blob/master/projects/70_cinder/resources.sh#L121 | |
| 15:15:48 | dansmith | and when we try to ssh: | |
| 15:15:49 | dansmith | 2023-02-28 20:55:58.897 | debug1: SSH2_MSG_SERVICE_ACCEPT received | |
| 15:15:49 | dansmith | 2023-02-28 20:55:58.894 | debug1: Will attempt key: /opt/stack/save/cinder_key.pem explicit | |
| 15:15:59 | dansmith | ahh | |
| 15:16:04 | dansmith | 2023-02-28 20:55:58.902 | sign_and_send_pubkey: no mutual signature supported | |
| 15:16:06 | dansmith | there ^ | |
| 15:16:52 | dansmith | but surely grenade is running on jammy nodes already, so I'm not sure why it wouldn't have been configured to support the old keys | |
| 15:18:28 | sean-k-mooney | dansmith: proably not | |
| 15:18:42 | sean-k-mooney | for antelop i think it used focal | |
| 15:18:48 | dansmith | looks like not yeah | |
| 15:19:00 | dansmith | okay, well, I can fix that then I guess | |
| 15:19:08 | sean-k-mooney | so likely we jsut need to call ssh-keygen directly | |
| 15:19:12 | sean-k-mooney | instead of osc | |
| 15:19:33 | bauzas | wait | |
| 15:19:36 | dansmith | well, I was going to configure the host to allow that key type | |
| 15:19:38 | bauzas | https://github.com/openstack/grenade/blob/master/projects/70_cinder/resources.sh#L121 | |
| 15:19:47 | bauzas | we no longer support this by Zed | |
| 15:20:11 | bauzas | oh no, sorry | |
| 15:20:14 | dansmith | bauzas: but it works if you use the old microversion, which osc is doing right? | |
| 15:20:17 | bauzas | you're passing the key | |
| 15:20:32 | bauzas | dansmith: ah right too | |
| 15:20:41 | dansmith | also that :) | |
| 15:20:48 | bauzas | but I wonder, that's probably why we have a problem with rsa | |
| 15:21:14 | bauzas | could we maybe try to create ed keys ? | |
| 15:22:21 | dansmith | I mean, I guess, but it seems more upgrade-y to allow the use of the older key type during the upgrade | |
| 15:23:17 | dansmith | hmm, that keypair create *is* generating the key in nova right? | |
| 15:23:26 | dansmith | $CINDER_KEY is just the key name | |
| 15:24:04 | bauzas | yeah | |
| 15:24:19 | bauzas | and by default it creates a ssh-rsa key | |
| 15:24:22 | dansmith | looks like that's the only place we run keypair create, so maybe it's best to just generate it properly | |
| 15:24:33 | bauzas | but you're creating it in Yoga, right? | |
| 15:24:38 | bauzas | or in Zed ? | |
| 15:25:23 | dansmith | it's master grenade | |
| 15:25:49 | bauzas | against a master devstack then ? | |
| 15:25:56 | bauzas | oh | |
| 15:26:35 | bauzas | it defaults to the previous release before upgrading | |
| 15:26:37 | bauzas | right? | |
| 15:26:50 | dansmith | not a master devstack | |
| 15:26:52 | bauzas | I mean the grenade base | |
| 15:27:16 | dansmith | at this point in the phase, we've run devstack from yoga, we're running grenade from master to do some things against it, then upgrade to devstack master | |
| 15:27:53 | bauzas | https://github.com/openstack/grenade/blob/master/grenaderc#L36 | |
| 15:28:05 | bauzas | ok | |
| 15:28:07 | bauzas | so | |
| 15:28:18 | bauzas | we created a yoga devstack, we're running grenade against it | |
| 15:28:23 | bauzas | which will create a key | |
| 15:28:39 | bauzas | and after this, we would upgrade to devstack master | |
| 15:28:58 | bauzas | so the environment that OSC calls is a Yoga nova-API service | |
| 15:29:06 | bauzas | amirite ? | |
| 15:29:22 | dansmith | yes | |
| 15:31:29 | dansmith | https://review.opendev.org/c/openstack/grenade/+/875940 | |
| 15:31:35 | dansmith | that ^ is what I'm thinking | |
| 15:32:47 | bauzas | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f59/875773/3/check/grenade-skip-level-continuous/f597bdf/controller/logs/grenade.sh_log.txt | |
| 15:32:53 | bauzas | I see the problem | |
| 15:32:58 | bauzas | but this is using jammy | |
| 15:33:03 | opendevreview | Dan Smith proposed openstack/nova master: Add continuous skip-level job for nova https://review.opendev.org/c/openstack/nova/+/875773 | |
| 15:33:15 | bauzas | so I guess ssh-rsa keys are no longer accepted | |
| 15:33:33 | bauzas | since you created them with yoga, it was accepted by Nova | |
| 15:33:43 | bauzas | and the guest was accordingly created | |
| 15:34:01 | dansmith | I don't think it has anything to do with yoga, | |
| 15:34:16 | dansmith | because even if we were running on jammy, we'd still be able to generate because of the old microversion right>? | |
| 15:34:46 | bauzas | you're true | |
| 15:34:48 | bauzas | nothing related | |
| 15:35:03 | bauzas | we continue to accept to blindly create ssh-rsa keys by default | |
| 15:35:08 | bauzas | if you don't opt(in | |
| 15:35:23 | bauzas | but, jammy guests should no longer accept it | |
| 15:35:27 | bauzas | lemme verify this | |
| 15:35:38 | dansmith | it's not jammy guests, it's hosts | |
| 15:35:51 | bauzas | oooh, then I understand | |
| 15:35:55 | dansmith | the host won't even use the ssh key to try to ssh to a remote if it's not one of the allowed key types | |
| 15:36:05 | dansmith | the error messaging for this has been terrible | |
| 15:36:14 | dansmith | because it shows that it's trying the key, but it's not | |
| 15:36:21 | dansmith | I struggled with this when I upgraded myself | |