| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-04-12 | |||
| 09:37:36 | rpittau | elod: thans for pointing to that change, maybe asking gmann could be worth it | |
| 09:37:36 | sean-k-mooney | https://zuul.opendev.org/t/openstack/build/90de727c38d54f4b80f6998ff3b19c17/log/job-output.txt#48455 | |
| 09:37:53 | sean-k-mooney | interesting so it install the correct verion and here it upgrade beyond the correct version | |
| 09:38:27 | sean-k-mooney | well its the victoria version | |
| 09:40:41 | sean-k-mooney | hum i guess that is after devstack completed initally | |
| 09:41:13 | sean-k-mooney | elod: bauzas is grenade ment to be enabled in the first run? | |
| 09:41:43 | sean-k-mooney | it look like devstack finsihedn and then it imediatly started runnign grenade without any tempest test in between | |
| 09:45:09 | sean-k-mooney | elod: bauzas what happening is its doing a normal devstack install and then its upgrading ironic only and then running tempest | |
| 09:45:10 | elod | there should be a smoke test. at least in nova's grenade jobs there is | |
| 09:45:29 | sean-k-mooney | right that is what i expect | |
| 09:45:57 | sean-k-mooney | but for some reason this is going strait to the upgrade of ironic and does not upgrade other services and then runs tempest | |
| 09:46:28 | sean-k-mooney | so upgrading ironic pulls in the victoria verion of oslo.messaging which is incompatible with ussui of everything else | |
| 09:47:30 | sean-k-mooney | actully its upgrading ironic then tempest nou tnot the rest | |
| 09:55:21 | sean-k-mooney | elod: the smoke tests are disabled https://zuul.opendev.org/t/openstack/build/90de727c38d54f4b80f6998ff3b19c17/log/controller/logs/grenade_localrc.txt#16 | |
| 09:55:43 | sean-k-mooney | for both the pre and post upgrade | |
| 09:56:19 | sean-k-mooney | it does not look like UPGRADE_PROJECTS is correct https://github.com/openstack/grenade/blob/stable/ussuri/grenade.sh#L361-L365 | |
| 09:56:38 | sean-k-mooney | or rather its not upgrading all the porjects unless i missed something | |
| 09:56:51 | sean-k-mooney | its just upgrading ironic | |
| 09:56:59 | sean-k-mooney | then it upgrades tempest | |
| 09:58:12 | sean-k-mooney | https://zuul.opendev.org/t/openstack/build/90de727c38d54f4b80f6998ff3b19c17/log/job-output.txt#47896-47897 | |
| 10:00:31 | sean-k-mooney | it looks like plugins are registered for upgrade | |
| 10:00:33 | sean-k-mooney | https://github.com/openstack/grenade/blob/08199218df99ead17d7c34256ed0969b8ae3b3af/inc/upgrade#L101-L111 | |
| 10:00:45 | elod | ok, so it works a bit different then in nova | |
| 10:01:00 | rpittau | sean-k-mooney, elod, bauzas, it seems fixed by https://review.opendev.org/q/If5f14654ab9aee2a140bbfb869b50d63cb289fdf | |
| 10:01:02 | sean-k-mooney | in that it does not run test before upgrading yes | |
| 10:01:07 | rpittau | please check #openstack-qa | |
| 10:01:16 | bauzas | ack | |
| 10:01:40 | sean-k-mooney | rpittau: that is not the fix | |
| 10:01:47 | rpittau | mmm wrong link? | |
| 10:01:49 | sean-k-mooney | at least i dont think so | |
| 10:02:10 | sean-k-mooney | i think the package is being installed in the ironic upgrade | |
| 10:02:12 | rpittau | sean-k-mooney: according to yoctozepto it is | |
| 10:02:16 | sean-k-mooney | not in the tempest one | |
| 10:02:19 | rpittau | ok | |
| 10:03:08 | sean-k-mooney | ya its form 2021-04-09 22:14:21.761580 | controller | 2021-04-09 22:14:21.761 | + inc/python:setup_package:445 : pip_install -e /opt/stack/new/ironic | |
| 10:03:26 | sean-k-mooney | that pulls in Collecting oslo.messaging===12.5.2 | |
| 10:04:10 | sean-k-mooney | rpittau: the issue here is the ironic job is only upgrading ironic and tempest | |
| 10:04:18 | sean-k-mooney | rpittau: it need to upgrade all installed services | |
| 10:04:33 | sean-k-mooney | because they all share the same version of oslo.messaging | |
| 10:04:48 | yoctozepto | rpittau, sean-k-mooney: that thing is fixing, e.g., ironic-grenade-dsvm job | |
| 10:04:50 | rpittau | sean-k-mooney: mmm ok I see, I think this was intended for some reason, need to double-check with dtantsur and TheJulia | |
| 10:05:04 | yoctozepto | (on train) | |
| 10:05:05 | sean-k-mooney | and the one form victorica that ironic is installing when its upgraded breaks ussuri | |
| 10:05:38 | yoctozepto | the failure was in smoke tests BEFORE upgrade | |
| 10:05:41 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/ussuri: Add regression test for bug #1908075 https://review.opendev.org/c/openstack/nova/+/783655 | |
| 10:05:44 | openstack | bug 1908075 in OpenStack Compute (nova) "Nova allows a non-multiattach volume to be attached to multiple instances *if* its volume state is reset by an admin" [Low,In progress] https://launchpad.net/bugs/1908075 - Assigned to Lee Yarwood (lyarwood) | |
| 10:05:45 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/ussuri: api: Reject volume attach requests when an active bdm exists https://review.opendev.org/c/openstack/nova/+/783656 | |
| 10:05:54 | sean-k-mooney | yoctozepto: the smoke tests are disabled | |
| 10:06:01 | sean-k-mooney | yoctozepto: before and after upgrade in this job | |
| 10:06:28 | sean-k-mooney | yoctozepto: this job is failing because the victoria version of oslo messagin is been install when ironic is upgraded | |
| 10:06:37 | sean-k-mooney | yoctozepto: but the rest of openstack is not upgraded | |
| 10:06:48 | yoctozepto | what job are we talking about? :D I am only about ironic-grenade-dsvm on train | |
| 10:07:05 | sean-k-mooney | yoctozepto: ironic-grenade | |
| 10:07:14 | sean-k-mooney | no -dvsm | |
| 10:07:18 | yoctozepto | oh, the new one; then it's irrelevant as you said | |
| 10:07:22 | yoctozepto | must be other issue | |
| 10:07:32 | sean-k-mooney | yep the issue is as i said | |
| 10:07:50 | sean-k-mooney | the ironic plugin was the only thing run in the upgrade | |
| 10:08:18 | sean-k-mooney | so its currently trying to run victoria ironic with ussuri everything else but oslo move or removed a decorator | |
| 10:08:39 | sean-k-mooney | so the ussuri code cant run with the new verion of the oslo package form victoria | |
| 10:08:52 | sean-k-mooney | the fix is just to upgrade all the porjects | |
| 10:09:40 | sean-k-mooney | yoctozepto: https://zuul.opendev.org/t/openstack/build/90de727c38d54f4b80f6998ff3b19c17/log/job-output.txt#47896-47897 see it just ran ironic | |
| 10:12:04 | yoctozepto | yes, I confirm what sean-k-mooney is saying | |
| 10:12:17 | yoctozepto | sorry for confusion, I misread the question | |
| 10:12:53 | rpittau | sean-k-mooney, yoctozepto, thanks, I will need to verify with other ironic cores | |
| 10:13:31 | sean-k-mooney | rpittau: unless ironic is installed in a venv its unsafe to upgrade it in isolation | |
| 10:13:44 | yoctozepto | ++ | |
| 10:13:44 | sean-k-mooney | rpittau: since we share depencies | |
| 10:13:57 | rpittau | ofc, makes total sense | |
| 10:13:58 | sean-k-mooney | rpittau: you could try to do that if you use the old UC | |
| 10:14:07 | yoctozepto | (psst, you can always base your gate on kolla, just saying ;-) ) | |
| 10:14:11 | rpittau | :) | |
| 10:14:34 | rpittau | I guess I have a topic for today meeting :) | |
| 10:14:37 | sean-k-mooney | yoctozepto: do you have full tempest/upgrade jobs in kolla yet? | |
| 10:15:11 | sean-k-mooney | i know it was a goal a long time ago just never followed up to see if ye completed it | |
| 10:16:51 | yoctozepto | sean-k-mooney: tempest not :-( but last cycle I made it able to track branches of other projects with zuul so that depends-on works etc. | |
| 10:17:00 | yoctozepto | I got bifrost to check against kolla-ansible | |
| 10:17:08 | yoctozepto | because they often broke us :D | |
| 10:17:43 | sean-k-mooney | yoctozepto: i still feel a little bad that i never had time to do phase 2 of the bifrost work | |
| 10:18:35 | sean-k-mooney | when i implemeted the inital work i had planned to decomose it into multiple contaier runnign the differen services instead of using systemd. | |
| 10:18:57 | sean-k-mooney | oh the other had it worked so :) | |
| 10:19:00 | yoctozepto | sean-k-mooney: https://github.com/markgoddard/bikolla might interest you then | |
| 10:19:40 | sean-k-mooney | haha why not just use kolla ironic support | |
| 10:19:57 | sean-k-mooney | e.g. just enable the minium service for ironic | |
| 10:20:11 | sean-k-mooney | i assume that is what that does | |
| 10:20:38 | sean-k-mooney | hum yes it looks like that is correct more or less https://github.com/markgoddard/bikolla/blob/master/etc/kolla/globals.yml | |
| 10:21:32 | yoctozepto | yes ;-) | |
| 10:21:41 | yoctozepto | just need to orchestrate the thing | |
| 10:22:11 | sean-k-mooney | im debating if im going to keep ironic or not when i redeploy my home cluster | |
| 10:22:32 | sean-k-mooney | part of me wants too but i use it rarely | |
| 10:22:47 | sean-k-mooney | and it seams to have issue with ubunutu | |
| 10:22:54 | sean-k-mooney | my centos image work fine | |
| 10:23:23 | sean-k-mooney | but i dont think the ipa image or cloud init fully work with ubunuts netplan conrigurtion | |
| 10:23:55 | sean-k-mooney | so i end up installing the os with a USB when i need anything other then centos anyway | |
| 10:34:10 | sean-k-mooney | stephenfin: im getting pinged for https://review.opendev.org/q/topic:%22bug%252F1910466%22+(status:open%20OR%20status:merged) downstream in bz. when you have time can you rereview those | |
| 10:34:16 | dtantsur | rpittau, sean-k-mooney, grenade should update all services.. breakage in infra jobs? | |
| 10:34:25 | stephenfin | sean-k-mooney: sure | |
| 10:34:31 | dtantsur | maybe we did something wrong when migrating to zuul v3? | |
| 10:34:52 | sean-k-mooney | dtantsur: perhaps only the plugins seam to be listed | |
| 10:37:33 | elod | just for the record I've looked into the last successful run of ironic-grenade on stable/victoria and it also upgraded to oslo.messaging===12.5.2 | |