| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-04-12 | |||
| 09:13:06 | elod | bauzas: no, not in victoria. stable/train is broken though (nova-grenade-multinode), but hopefully fixed by this patch when it merges: https://review.opendev.org/c/openstack/grenade/+/785831 | |
| 09:13:50 | sean-k-mooney | https://zuul.opendev.org/t/openstack/build/90de727c38d54f4b80f6998ff3b19c17/log/controller/logs/pip3-freeze.txt#101 | |
| 09:13:53 | bauzas | elod: looks like it could be related with rpittau | |
| 09:13:56 | sean-k-mooney | oslo.messaging==12.5.2 | |
| 09:14:08 | bauzas | they don't correctly pin the versions IMHO | |
| 09:14:16 | bauzas | sean-k-mooney: yup | |
| 09:14:58 | sean-k-mooney | the version would be pinned in UC | |
| 09:15:02 | sean-k-mooney | right? | |
| 09:15:38 | sean-k-mooney | https://github.com/openstack/requirements/blob/stable/ussuri/upper-constraints.txt#L274-L277 | |
| 09:15:55 | sean-k-mooney | so that is above the UC constrait for ussuri | |
| 09:16:27 | sean-k-mooney | it is the correct one for victoria | |
| 09:16:38 | sean-k-mooney | but grenade has not run yes | |
| 09:16:42 | sean-k-mooney | *yet | |
| 09:17:49 | sean-k-mooney | maybe it has. th ejob logs look weird | |
| 09:17:58 | sean-k-mooney | i dont see the devstack or grenade log | |
| 09:18:19 | sean-k-mooney | well the devstack log is in old | |
| 09:18:25 | bauzas | I'm looking at https://zuul.opendev.org/t/openstack/build/90de727c38d54f4b80f6998ff3b19c17/log/job-output.txt | |
| 09:18:35 | bauzas | and I can see the upgrade | |
| 09:18:36 | elod | bauzas rpittau : at a first glance the failure looks different. anyway, there was a change in grenade on last Thursday, maybe related: https://review.opendev.org/q/topic:"standardize-tempest-tox-constraints"+status:merged | |
| 09:18:58 | sean-k-mooney | actully the devstack log has the grenade output in it? | |
| 09:19:05 | sean-k-mooney | https://zuul.opendev.org/t/openstack/build/90de727c38d54f4b80f6998ff3b19c17/log/controller/logs/old/devstacklog.txt | |
| 09:21:18 | sean-k-mooney | if this is the new zuul v3 grenade job it looks like its overriding the devstack log with the greneade log | |
| 09:25:38 | sean-k-mooney | it looks like LOGFILE="/opt/stack/old/logs/devstacklog.txt" is not working properly | |
| 09:25:57 | sean-k-mooney | the devstack logs are just in the main output file | |
| 09:33:50 | bauzas | honestly, I'm damn puzzled | |
| 09:34:41 | elod | this looks also interesting: https://zuul.opendev.org/t/openstack/build/90de727c38d54f4b80f6998ff3b19c17/log/job-output.txt#48860-48864 | |
| 09:35:32 | elod | that shows some error, but wierdly the end it succeeds | |
| 09:35:57 | sean-k-mooney | this is the old resolver | |
| 09:36:12 | sean-k-mooney | and its waring about the new one right | |
| 09:37:36 | rpittau | elod: thans for pointing to that change, maybe asking gmann could be worth it | |
| 09:37:36 | sean-k-mooney | https://zuul.opendev.org/t/openstack/build/90de727c38d54f4b80f6998ff3b19c17/log/job-output.txt#48455 | |
| 09:37:53 | sean-k-mooney | interesting so it install the correct verion and here it upgrade beyond the correct version | |
| 09:38:27 | sean-k-mooney | well its the victoria version | |
| 09:40:41 | sean-k-mooney | hum i guess that is after devstack completed initally | |
| 09:41:13 | sean-k-mooney | elod: bauzas is grenade ment to be enabled in the first run? | |
| 09:41:43 | sean-k-mooney | it look like devstack finsihedn and then it imediatly started runnign grenade without any tempest test in between | |
| 09:45:09 | sean-k-mooney | elod: bauzas what happening is its doing a normal devstack install and then its upgrading ironic only and then running tempest | |
| 09:45:10 | elod | there should be a smoke test. at least in nova's grenade jobs there is | |
| 09:45:29 | sean-k-mooney | right that is what i expect | |
| 09:45:57 | sean-k-mooney | but for some reason this is going strait to the upgrade of ironic and does not upgrade other services and then runs tempest | |
| 09:46:28 | sean-k-mooney | so upgrading ironic pulls in the victoria verion of oslo.messaging which is incompatible with ussui of everything else | |
| 09:47:30 | sean-k-mooney | actully its upgrading ironic then tempest nou tnot the rest | |
| 09:55:21 | sean-k-mooney | elod: the smoke tests are disabled https://zuul.opendev.org/t/openstack/build/90de727c38d54f4b80f6998ff3b19c17/log/controller/logs/grenade_localrc.txt#16 | |
| 09:55:43 | sean-k-mooney | for both the pre and post upgrade | |
| 09:56:19 | sean-k-mooney | it does not look like UPGRADE_PROJECTS is correct https://github.com/openstack/grenade/blob/stable/ussuri/grenade.sh#L361-L365 | |
| 09:56:38 | sean-k-mooney | or rather its not upgrading all the porjects unless i missed something | |
| 09:56:51 | sean-k-mooney | its just upgrading ironic | |
| 09:56:59 | sean-k-mooney | then it upgrades tempest | |
| 09:58:12 | sean-k-mooney | https://zuul.opendev.org/t/openstack/build/90de727c38d54f4b80f6998ff3b19c17/log/job-output.txt#47896-47897 | |
| 10:00:31 | sean-k-mooney | it looks like plugins are registered for upgrade | |
| 10:00:33 | sean-k-mooney | https://github.com/openstack/grenade/blob/08199218df99ead17d7c34256ed0969b8ae3b3af/inc/upgrade#L101-L111 | |
| 10:00:45 | elod | ok, so it works a bit different then in nova | |
| 10:01:00 | rpittau | sean-k-mooney, elod, bauzas, it seems fixed by https://review.opendev.org/q/If5f14654ab9aee2a140bbfb869b50d63cb289fdf | |
| 10:01:02 | sean-k-mooney | in that it does not run test before upgrading yes | |
| 10:01:07 | rpittau | please check #openstack-qa | |
| 10:01:16 | bauzas | ack | |
| 10:01:40 | sean-k-mooney | rpittau: that is not the fix | |
| 10:01:47 | rpittau | mmm wrong link? | |
| 10:01:49 | sean-k-mooney | at least i dont think so | |
| 10:02:10 | sean-k-mooney | i think the package is being installed in the ironic upgrade | |
| 10:02:12 | rpittau | sean-k-mooney: according to yoctozepto it is | |
| 10:02:16 | sean-k-mooney | not in the tempest one | |
| 10:02:19 | rpittau | ok | |
| 10:03:08 | sean-k-mooney | ya its form 2021-04-09 22:14:21.761580 | controller | 2021-04-09 22:14:21.761 | + inc/python:setup_package:445 : pip_install -e /opt/stack/new/ironic | |
| 10:03:26 | sean-k-mooney | that pulls in Collecting oslo.messaging===12.5.2 | |
| 10:04:10 | sean-k-mooney | rpittau: the issue here is the ironic job is only upgrading ironic and tempest | |
| 10:04:18 | sean-k-mooney | rpittau: it need to upgrade all installed services | |
| 10:04:33 | sean-k-mooney | because they all share the same version of oslo.messaging | |
| 10:04:48 | yoctozepto | rpittau, sean-k-mooney: that thing is fixing, e.g., ironic-grenade-dsvm job | |
| 10:04:50 | rpittau | sean-k-mooney: mmm ok I see, I think this was intended for some reason, need to double-check with dtantsur and TheJulia | |
| 10:05:04 | yoctozepto | (on train) | |
| 10:05:05 | sean-k-mooney | and the one form victorica that ironic is installing when its upgraded breaks ussuri | |
| 10:05:38 | yoctozepto | the failure was in smoke tests BEFORE upgrade | |
| 10:05:41 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/ussuri: Add regression test for bug #1908075 https://review.opendev.org/c/openstack/nova/+/783655 | |
| 10:05:44 | openstack | bug 1908075 in OpenStack Compute (nova) "Nova allows a non-multiattach volume to be attached to multiple instances *if* its volume state is reset by an admin" [Low,In progress] https://launchpad.net/bugs/1908075 - Assigned to Lee Yarwood (lyarwood) | |
| 10:05:45 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/ussuri: api: Reject volume attach requests when an active bdm exists https://review.opendev.org/c/openstack/nova/+/783656 | |
| 10:05:54 | sean-k-mooney | yoctozepto: the smoke tests are disabled | |
| 10:06:01 | sean-k-mooney | yoctozepto: before and after upgrade in this job | |
| 10:06:28 | sean-k-mooney | yoctozepto: this job is failing because the victoria version of oslo messagin is been install when ironic is upgraded | |
| 10:06:37 | sean-k-mooney | yoctozepto: but the rest of openstack is not upgraded | |
| 10:06:48 | yoctozepto | what job are we talking about? :D I am only about ironic-grenade-dsvm on train | |
| 10:07:05 | sean-k-mooney | yoctozepto: ironic-grenade | |
| 10:07:14 | sean-k-mooney | no -dvsm | |
| 10:07:18 | yoctozepto | oh, the new one; then it's irrelevant as you said | |
| 10:07:22 | yoctozepto | must be other issue | |
| 10:07:32 | sean-k-mooney | yep the issue is as i said | |
| 10:07:50 | sean-k-mooney | the ironic plugin was the only thing run in the upgrade | |
| 10:08:18 | sean-k-mooney | so its currently trying to run victoria ironic with ussuri everything else but oslo move or removed a decorator | |
| 10:08:39 | sean-k-mooney | so the ussuri code cant run with the new verion of the oslo package form victoria | |
| 10:08:52 | sean-k-mooney | the fix is just to upgrade all the porjects | |
| 10:09:40 | sean-k-mooney | yoctozepto: https://zuul.opendev.org/t/openstack/build/90de727c38d54f4b80f6998ff3b19c17/log/job-output.txt#47896-47897 see it just ran ironic | |
| 10:12:04 | yoctozepto | yes, I confirm what sean-k-mooney is saying | |
| 10:12:17 | yoctozepto | sorry for confusion, I misread the question | |
| 10:12:53 | rpittau | sean-k-mooney, yoctozepto, thanks, I will need to verify with other ironic cores | |
| 10:13:31 | sean-k-mooney | rpittau: unless ironic is installed in a venv its unsafe to upgrade it in isolation | |
| 10:13:44 | yoctozepto | ++ | |
| 10:13:44 | sean-k-mooney | rpittau: since we share depencies | |
| 10:13:57 | rpittau | ofc, makes total sense | |
| 10:13:58 | sean-k-mooney | rpittau: you could try to do that if you use the old UC | |
| 10:14:07 | yoctozepto | (psst, you can always base your gate on kolla, just saying ;-) ) | |