| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-09-30 | |||
| 17:48:22 | melwitt | I see that the job began collecting jsonschema==2.6.0 but ended up installing 4.0.0 | |
| 17:48:54 | melwitt | but why... | |
| 17:49:37 | melwitt | or how. it's pinned to 3.0.2 as you said earlier https://github.com/openstack/requirements/blob/stable/train/upper-constraints.txt#L634 | |
| 17:49:53 | melwitt | collecting: https://zuul.opendev.org/t/openstack/build/08e374906e2844ef865d4c698b3b78e5/log/job-output.txt#38834 | |
| 17:51:41 | melwitt | ok but then it uninstalls 4.0.0 and installs 2.6.0 here https://zuul.opendev.org/t/openstack/build/08e374906e2844ef865d4c698b3b78e5/log/job-output.txt#38886-38895 | |
| 17:52:17 | melwitt | and then tempest installs 4.0.0 https://zuul.opendev.org/t/openstack/build/08e374906e2844ef865d4c698b3b78e5/log/job-output.txt#55512 | |
| 17:52:53 | melwitt | gmann: does tempest running on stable branch not use upper-constraints from stable? ^ | |
| 17:55:23 | elodilles | actually, since this is a grenade job, it should be stein | |
| 17:55:44 | elodilles | where jsonschema is constrained 2.6.0 | |
| 17:56:09 | melwitt | oh right | |
| 17:56:40 | elodilles | but tempest @ stein-last requires: jsonschema>=3.2.0 # MIT | |
| 17:57:08 | elodilles | so it is strange how it works | |
| 18:00:42 | elodilles | (just for the record, currently jsonschema===3.2.0 is in the upper-constraints.txt @ master) | |
| 18:12:47 | melwitt | yeah, I don't understand it | |
| 18:23:18 | opendevreview | Lee Yarwood proposed openstack/nova master: trivial: Rename LibvirtConfigGuestPCIeRootPortController https://review.opendev.org/c/openstack/nova/+/812002 | |
| 20:07:25 | opendevreview | Merged openstack/nova master: Store old_flavor already on source host during resize https://review.opendev.org/c/openstack/nova/+/810909 | |
| 21:32:01 | gmann | melwitt:elodilles: in train, we still using tempest master which I need to move to train-last soon | |
| 21:32:49 | melwitt | gmann: lmk if I can help. stable/train currently broken due to new jsonschema 4.0.0 from master | |
| 21:32:56 | gmann | in stein yes tempest 26.0.0 is used | |
| 21:33:39 | gmann | melwitt: ok. I need to go out to pick up my wife. I will check after coming back | |
| 21:35:20 | melwitt | gmann: np thanks, I can work on it, just I would need a hint where to make a change | |
| 21:39:32 | gmann | melwitt: at first glance, it seems it is using stein branch constraint only https://zuul.opendev.org/t/openstack/build/08e374906e2844ef865d4c698b3b78e5/log/job-output.txt#12290 | |
| 21:41:10 | melwitt | hm ok | |
| 22:16:11 | gmann | melwitt: I think tempest job on stable/stein also should fail on this. | |
| 22:17:15 | melwitt | gmann: yeah.. I've been looking at how it says it's using the stein upper-constraints file, but then after it does "tox -r -notest -efull" the 4.0.0 jsonschema gets installed. I don't understand that | |
| 22:17:56 | gmann | melwitt: it is pinned to used stable/stein u-c in stable/stein with tempest 26.0.0 | |
| 22:18:04 | melwitt | so I was looking at https://github.com/openstack/tempest/blob/5d7e46f5689040ddaff798662f7a44fc71758d2e/tox.ini#L14 and I didn't know what the -c does but it looks like this is supposed to be use the env var upper-constraints file if present else use master? | |
| 22:18:45 | gmann | but it should not install latest jsonschema | |
| 22:19:22 | melwitt | here it says "full installed" ... jsonschema==4.0.0 https://zuul.opendev.org/t/openstack/build/08e374906e2844ef865d4c698b3b78e5/log/job-output.txt#12300 | |
| 22:19:27 | melwitt | I took that to mean the tox -efull env installed 4.0.0 | |
| 22:22:46 | gmann | but later on it uninstall 4.0.0 and install 2.6.0 https://zuul.opendev.org/t/openstack/build/08e374906e2844ef865d4c698b3b78e5/log/job-output.txt#12423-12427 | |
| 22:23:21 | melwitt | oh, again? oh sorry | |
| 22:24:20 | melwitt | ok here's the last time it installs 4.0.0 before the error https://zuul.opendev.org/t/openstack/build/08e374906e2844ef865d4c698b3b78e5/log/job-output.txt#28518 | |
| 22:24:25 | gmann | but in final installed version in venv I always see 4.0.0 so not sure who & when it is installed/upgraded | |
| 22:24:41 | melwitt | I get confused by how it installs and uninstalls 4.0.0 repeatedly | |
| 22:25:31 | melwitt | venv installdeps: -c/opt/stack/old/requirements/upper-constraints.txt, -r/opt/stack/old/tempest/requirements.txt, -r/opt/stack/old/tempest/doc/requirements.txt | |
| 22:32:47 | gmann | as it fallback to latest bcz u-c not matched, we should use the compatible constraint while we cut the tempest 26.0.0 instead of stable/stein by default. | |
| 22:37:15 | gmann | at least that is tested combination tempest we released tempest 26.0.0 | |
| 22:38:28 | melwitt | what does it mean u-c didn't match? | |
| 22:40:00 | gmann | tempest 26.0.0 requirement has jsonschema>=3.2.0 and u-c jsonschema== 2.6.0 - https://github.com/openstack/tempest/blob/26.0.0/requirements.txt#L6 | |
| 22:40:16 | melwitt | ohhh I see. ok | |
| 22:40:17 | gmann | that is why it try to go for latest jsonschema | |
| 22:41:25 | gmann | complex thing here is to make tempest venv work on single (non-master) constraints, we have to take care in many place otherwise tox will recreate the tempest venv. many place I mean in devstack, run-tempest role, tox.ini etc | |
| 22:41:39 | melwitt | I see now. I didn't know it would ignore u-c if the requirement was higher than it ... I thought it would have failed like "can't install the tempest that you want" | |
| 22:42:12 | clarkb | melwitt: it should always use the constraint | |
| 22:42:34 | clarkb | or fail I guess, but ya if it doesn't do that I would consider that a bug in pip | |
| 22:42:57 | gmann | it actuall try to lowerdown the version as per -c but we have verify-config script from tempest running tempest in between so it fail | |
| 22:43:09 | gmann | otherwise the final version could be as per u-c only | |
| 22:43:21 | melwitt | hm O.o | |
| 22:43:23 | gmann | like it happen in master testing, it install 3.2.0 as per u-c not latest | |
| 22:44:25 | melwitt | sorry I'm getting even more confused. it's the verify-config script from tempest that is installing > u-c? | |
| 22:45:14 | gmann | melwitt: no in between somehow latest jsonschema is installed and verify-config run fail. why it install latest jsonschema and not failing on version conflict might be bug as clarkb mentioned | |
| 22:46:00 | gmann | basically in both case either failed with conflict or latest jsonschema, our constraint mismatch is issue | |
| 22:46:26 | melwitt | yeah. ok | |
| 22:47:14 | clarkb | Looking at the log pip says "tempest 26.0.0 has requirement jsonschema>=3.2.0, but you'll have jsonschema 2.6.0 which is incompatible." then it installs jsonschema 2.6.0 | |
| 22:47:20 | clarkb | I think constraitns are working properly as a result | |
| 22:47:31 | clarkb | (that was my understanding of how they work, constraints always wins) | |
| 22:47:41 | gmann | we have two way to solve it 1. use tempest tag (<26.0.0) which is compatible with stable/stein constraints 2. use compatible constraints (higher than of stable/stein) with tempest 26.0.0 | |
| 22:47:54 | clarkb | your requirements and constraints are at odds with each other | |
| 22:47:58 | gmann | yeah | |
| 22:48:27 | melwitt | yeah, that's why I wondered if https://github.com/openstack/tempest/blob/5d7e46f5689040ddaff798662f7a44fc71758d2e/tox.ini#L14 is doing something wrong, the log says that "tempest full" is what installed 4.0.0 | |
| 22:48:30 | gmann | requirement is from tempest 26.0.0 which is cut at the time of stable/vicrotia | |
| 22:49:30 | gmann | clarkb: but it does not say who installed 4.0.0 but we see that version as final version in installed version in venv | |
| 22:50:03 | clarkb | are you sure 4.0.0 is the final version? | |
| 22:50:32 | clarkb | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_08e/791807/5/check/neutron-grenade-multinode/08e3749/controller/logs/old/devstacklog.txt shows it is failing because jsonschema.compat isn't present which I assume is in >=3.2.0 | |
| 22:50:34 | gmann | clarkb: https://zuul.opendev.org/t/openstack/build/08e374906e2844ef865d4c698b3b78e5/log/job-output.txt#28518 | |
| 22:51:49 | gmann | they seems removed in 4.0.0 that is why installing 4.0.0. end up started failing due to this constraints and requirement mismatch | |
| 22:52:29 | gmann | I checked before 4.0.0 release we were istalling latest and it was juts working because of no incompatible change there | |
| 22:52:58 | clarkb | ok I think I see it | |
| 22:53:02 | clarkb | there are two different venvs | |
| 22:53:07 | clarkb | 'venv' and 'venv-tempest | |
| 22:53:47 | clarkb | they both install jsonschema 4.0.0 using venv-tempest installdeps: -c/opt/stack/old/requirements/upper-constraints.txt, -r/opt/stack/old/tempest/requirements.txt and venv installdeps: -c/opt/stack/old/requirements/upper-constraints.txt, -r/opt/stack/old/tempest/requirements.txt, -r/opt/stack/old/tempest/doc/requirements.txt | |
| 22:54:05 | clarkb | But venv-tempest has an extra step where it does venv-tempest run-test: commands[0] | pip install -c /tmp/tempest_u_c_m.03mOFK3yxm -r requirements.txt | |
| 22:54:31 | clarkb | It is exceptionally odd to me to run pip in the run-test portion of a tox target | |
| 22:54:36 | clarkb | but there you have it | |
| 22:55:51 | clarkb | looks like venv-tempest does command = {posargs} so I'm not sure what calls it with that to downgrade | |
| 22:56:23 | clarkb | but that modifies .tox/tempest and not .tox/venv | |
| 22:56:42 | clarkb | then you try ot run something in .tox/venv and it breaks due to the newer version of jsonschema | |
| 22:57:09 | melwitt | my head is spinning 😆 | |
| 22:57:59 | melwitt | but I'm very glad to know what is going on. I'll need time to process it | |
| 22:58:36 | clarkb | https://opendev.org/openstack/devstack/src/branch/master/lib/tempest#L627 that is where the pip install that puts jsonschema 2.6.0 into .tox/tempest via the venv-tempest target comes from | |
| 22:58:59 | clarkb | melwitt: my head is spinning too :) there is a lot of back and forth in here and I'm not really sure I understand why as much as now I at least understand how that exception arises | |
| 22:59:51 | melwitt | yeah... I hope there's a way to fix the root cause, having it be able to do this seems too confusing and error prone | |
| 23:00:12 | clarkb | then we run https://opendev.org/openstack/devstack/src/branch/master/lib/tempest#L649 and it explodes because the version is too new | |
| 23:00:55 | clarkb | gmann: I think that means you need a newer version of tempest that works against newer jsonschema as constrained there? | |
| 23:01:05 | clarkb | rather than older | |
| 23:01:33 | gmann | older | |
| 23:02:24 | clarkb | hrm wait -c/opt/stack/old/requirements/upper-constraints.txt that should be using stable/stein requirements in old/ right? | |
| 23:02:30 | clarkb | because this job ran against train | |
| 23:02:44 | clarkb | stable/stein requirements u-c says jsonschema should be 2.6.0 | |
| 23:03:20 | melwitt | right | |
| 23:04:15 | clarkb | the job-output.txt for that build has made my browser very sad | |
| 23:04:30 | melwitt | my browser has a sad as well | |
| 23:06:12 | gmann | heh, i opened three and it stuck | |
| 23:06:33 | gmann | clarkb: melwitt https://review.opendev.org/c/openstack/devstack/+/812092 | |
| 23:06:36 | gmann | let' see | |
| 23:06:58 | gmann | I am going with 1st solution of lowering the tempest as per stable/stein constraints | |
| 23:07:03 | clarkb | gmann: if that fixes it it doesn't explain why old/requirements somehow became newer requirements | |
| 23:07:24 | clarkb | I do see that old/requirements is initially checked out to 3a3ed8a7 which is correct | |
| 23:07:24 | gmann | clarkb: did not get? | |