| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-04-11 | |||
| 15:35:32 | openstackgerrit | Merged openstack/nova stable/ussuri: Raise InstanceMappingNotFound if StaleDataError is encountered https://review.opendev.org/c/openstack/nova/+/777253 | |
| #openstack-nova - 2021-04-12 | |||
| 02:41:19 | openstackgerrit | Qiu Fossen proposed openstack/nova-specs master: Repropose copying pmem devices data spec https://review.opendev.org/c/openstack/nova-specs/+/785563 | |
| 07:47:18 | bauzas | good morning Nova | |
| 08:29:55 | gibi | bauzas: good morning | |
| 08:31:19 | bauzas | eventually freed from PTOs and kids | |
| 08:33:44 | yonglihe | good morning | |
| 08:34:53 | yonglihe | bauzas, free from kids, really, tell me how | |
| 08:35:14 | bauzas | yonglihe: by passing them to the grandparents during the vacations ;) | |
| 08:35:23 | yonglihe | -:) | |
| 08:35:34 | bauzas | (during the *school* vacations tbc= | |
| 08:37:28 | openstackgerrit | Lucas Alvares Gomes proposed openstack/nova master: [OVN] Explicitly set nova-next job to ML2/OVS https://review.opendev.org/c/openstack/nova/+/776944 | |
| 08:40:47 | xinranwang | gibi: Hi gibi, I have reproposed the smartnic spec, please review it when you got time. thanks! https://review.opendev.org/c/openstack/nova-specs/+/783632 | |
| 08:40:56 | openstackgerrit | Lucas Alvares Gomes proposed openstack/nova master: [OVN] Explicitly set grenade job to ML2/OVS https://review.opendev.org/c/openstack/nova/+/776934 | |
| 08:40:56 | openstackgerrit | Lucas Alvares Gomes proposed openstack/nova master: [OVN] Explicitly set nova-next job to ML2/OVS https://review.opendev.org/c/openstack/nova/+/776944 | |
| 08:46:15 | rpittau | good morning everyone! Not sure if it's a known issue, when running grenaed jobs in stable/victoria in ironic, we're seeing this error in nova-api loga: http://paste.openstack.org/show/804380/ | |
| 08:47:16 | rpittau | this is an example of failing job: https://zuul.opendev.org/t/openstack/build/90de727c38d54f4b80f6998ff3b19c17 | |
| 08:51:29 | bauzas | rpittau: looking | |
| 08:51:37 | rpittau | thanks! | |
| 08:53:04 | bauzas | looks like an oslo messaging version issue | |
| 08:53:16 | bauzas | eg. https://zuul.opendev.org/t/openstack/build/90de727c38d54f4b80f6998ff3b19c17/log/controller/logs/screen-n-api.txt#2034 | |
| 08:54:52 | bauzas | but that's on the old ussuri API | |
| 08:54:57 | rpittau | interesting, we started noticing the failure just last friday | |
| 08:55:03 | openstackgerrit | Qiu Fossen proposed openstack/nova-specs master: Allow migrating PMEM's data https://review.opendev.org/c/openstack/nova-specs/+/785563 | |
| 08:56:26 | openstackgerrit | Elod Illes proposed openstack/nova stable/train: DNM: nova-grenade-multinode test https://review.opendev.org/c/openstack/nova/+/785731 | |
| 08:57:10 | bauzas | rpittau: list of installed packages https://zuul.opendev.org/t/openstack/build/90de727c38d54f4b80f6998ff3b19c17/log/job-output.txt#51286 | |
| 08:57:30 | openstackgerrit | Qiu Fossen proposed openstack/nova-specs master: Allow migrating PMEM's data https://review.opendev.org/c/openstack/nova-specs/+/785563 | |
| 08:58:30 | bauzas | I'm not a grenade specialist, but it looks like it happens with the original install, before upgrading, while testing the original deployment | |
| 08:58:34 | openstackgerrit | Elod Illes proposed openstack/nova stable/train: DNM: nova-grenade-multinode test https://review.opendev.org/c/openstack/nova/+/785731 | |
| 08:58:55 | rpittau | bauzas: yeah, I can confirm as the job fails immediately | |
| 08:59:08 | bauzas | elod: have you seen any issues with grenade on stable/victoria jobs ? | |
| 08:59:15 | rpittau | so nova doesn't like the version of oslo messaging installed in ussuri | |
| 08:59:30 | bauzas | rpittau: that's what I'm trying to see | |
| 08:59:36 | rpittau | ok, thanks | |
| 09:13:06 | elod | bauzas: no, not in victoria. stable/train is broken though (nova-grenade-multinode), but hopefully fixed by this patch when it merges: https://review.opendev.org/c/openstack/grenade/+/785831 | |
| 09:13:50 | sean-k-mooney | https://zuul.opendev.org/t/openstack/build/90de727c38d54f4b80f6998ff3b19c17/log/controller/logs/pip3-freeze.txt#101 | |
| 09:13:53 | bauzas | elod: looks like it could be related with rpittau | |
| 09:13:56 | sean-k-mooney | oslo.messaging==12.5.2 | |
| 09:14:08 | bauzas | they don't correctly pin the versions IMHO | |
| 09:14:16 | bauzas | sean-k-mooney: yup | |
| 09:14:58 | sean-k-mooney | the version would be pinned in UC | |
| 09:15:02 | sean-k-mooney | right? | |
| 09:15:38 | sean-k-mooney | https://github.com/openstack/requirements/blob/stable/ussuri/upper-constraints.txt#L274-L277 | |
| 09:15:55 | sean-k-mooney | so that is above the UC constrait for ussuri | |
| 09:16:27 | sean-k-mooney | it is the correct one for victoria | |
| 09:16:38 | sean-k-mooney | but grenade has not run yes | |
| 09:16:42 | sean-k-mooney | *yet | |
| 09:17:49 | sean-k-mooney | maybe it has. th ejob logs look weird | |
| 09:17:58 | sean-k-mooney | i dont see the devstack or grenade log | |
| 09:18:19 | sean-k-mooney | well the devstack log is in old | |
| 09:18:25 | bauzas | I'm looking at https://zuul.opendev.org/t/openstack/build/90de727c38d54f4b80f6998ff3b19c17/log/job-output.txt | |
| 09:18:35 | bauzas | and I can see the upgrade | |
| 09:18:36 | elod | bauzas rpittau : at a first glance the failure looks different. anyway, there was a change in grenade on last Thursday, maybe related: https://review.opendev.org/q/topic:"standardize-tempest-tox-constraints"+status:merged | |
| 09:18:58 | sean-k-mooney | actully the devstack log has the grenade output in it? | |
| 09:19:05 | sean-k-mooney | https://zuul.opendev.org/t/openstack/build/90de727c38d54f4b80f6998ff3b19c17/log/controller/logs/old/devstacklog.txt | |
| 09:21:18 | sean-k-mooney | if this is the new zuul v3 grenade job it looks like its overriding the devstack log with the greneade log | |
| 09:25:38 | sean-k-mooney | it looks like LOGFILE="/opt/stack/old/logs/devstacklog.txt" is not working properly | |
| 09:25:57 | sean-k-mooney | the devstack logs are just in the main output file | |
| 09:33:50 | bauzas | honestly, I'm damn puzzled | |
| 09:34:41 | elod | this looks also interesting: https://zuul.opendev.org/t/openstack/build/90de727c38d54f4b80f6998ff3b19c17/log/job-output.txt#48860-48864 | |
| 09:35:32 | elod | that shows some error, but wierdly the end it succeeds | |
| 09:35:57 | sean-k-mooney | this is the old resolver | |
| 09:36:12 | sean-k-mooney | and its waring about the new one right | |
| 09:37:36 | rpittau | elod: thans for pointing to that change, maybe asking gmann could be worth it | |
| 09:37:36 | sean-k-mooney | https://zuul.opendev.org/t/openstack/build/90de727c38d54f4b80f6998ff3b19c17/log/job-output.txt#48455 | |
| 09:37:53 | sean-k-mooney | interesting so it install the correct verion and here it upgrade beyond the correct version | |
| 09:38:27 | sean-k-mooney | well its the victoria version | |
| 09:40:41 | sean-k-mooney | hum i guess that is after devstack completed initally | |
| 09:41:13 | sean-k-mooney | elod: bauzas is grenade ment to be enabled in the first run? | |
| 09:41:43 | sean-k-mooney | it look like devstack finsihedn and then it imediatly started runnign grenade without any tempest test in between | |
| 09:45:09 | sean-k-mooney | elod: bauzas what happening is its doing a normal devstack install and then its upgrading ironic only and then running tempest | |
| 09:45:10 | elod | there should be a smoke test. at least in nova's grenade jobs there is | |
| 09:45:29 | sean-k-mooney | right that is what i expect | |
| 09:45:57 | sean-k-mooney | but for some reason this is going strait to the upgrade of ironic and does not upgrade other services and then runs tempest | |
| 09:46:28 | sean-k-mooney | so upgrading ironic pulls in the victoria verion of oslo.messaging which is incompatible with ussui of everything else | |
| 09:47:30 | sean-k-mooney | actully its upgrading ironic then tempest nou tnot the rest | |
| 09:55:21 | sean-k-mooney | elod: the smoke tests are disabled https://zuul.opendev.org/t/openstack/build/90de727c38d54f4b80f6998ff3b19c17/log/controller/logs/grenade_localrc.txt#16 | |
| 09:55:43 | sean-k-mooney | for both the pre and post upgrade | |
| 09:56:19 | sean-k-mooney | it does not look like UPGRADE_PROJECTS is correct https://github.com/openstack/grenade/blob/stable/ussuri/grenade.sh#L361-L365 | |
| 09:56:38 | sean-k-mooney | or rather its not upgrading all the porjects unless i missed something | |
| 09:56:51 | sean-k-mooney | its just upgrading ironic | |
| 09:56:59 | sean-k-mooney | then it upgrades tempest | |
| 09:58:12 | sean-k-mooney | https://zuul.opendev.org/t/openstack/build/90de727c38d54f4b80f6998ff3b19c17/log/job-output.txt#47896-47897 | |
| 10:00:31 | sean-k-mooney | it looks like plugins are registered for upgrade | |
| 10:00:33 | sean-k-mooney | https://github.com/openstack/grenade/blob/08199218df99ead17d7c34256ed0969b8ae3b3af/inc/upgrade#L101-L111 | |
| 10:00:45 | elod | ok, so it works a bit different then in nova | |
| 10:01:00 | rpittau | sean-k-mooney, elod, bauzas, it seems fixed by https://review.opendev.org/q/If5f14654ab9aee2a140bbfb869b50d63cb289fdf | |
| 10:01:02 | sean-k-mooney | in that it does not run test before upgrading yes | |
| 10:01:07 | rpittau | please check #openstack-qa | |
| 10:01:16 | bauzas | ack | |
| 10:01:40 | sean-k-mooney | rpittau: that is not the fix | |
| 10:01:47 | rpittau | mmm wrong link? | |
| 10:01:49 | sean-k-mooney | at least i dont think so | |
| 10:02:10 | sean-k-mooney | i think the package is being installed in the ironic upgrade | |
| 10:02:12 | rpittau | sean-k-mooney: according to yoctozepto it is | |
| 10:02:16 | sean-k-mooney | not in the tempest one | |
| 10:02:19 | rpittau | ok | |
| 10:03:08 | sean-k-mooney | ya its form 2021-04-09 22:14:21.761580 | controller | 2021-04-09 22:14:21.761 | + inc/python:setup_package:445 : pip_install -e /opt/stack/new/ironic | |
| 10:03:26 | sean-k-mooney | that pulls in Collecting oslo.messaging===12.5.2 | |
| 10:04:10 | sean-k-mooney | rpittau: the issue here is the ironic job is only upgrading ironic and tempest | |
| 10:04:18 | sean-k-mooney | rpittau: it need to upgrade all installed services | |