| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-09-30 | |||
| 22:19:27 | melwitt | I took that to mean the tox -efull env installed 4.0.0 | |
| 22:22:46 | gmann | but later on it uninstall 4.0.0 and install 2.6.0 https://zuul.opendev.org/t/openstack/build/08e374906e2844ef865d4c698b3b78e5/log/job-output.txt#12423-12427 | |
| 22:23:21 | melwitt | oh, again? oh sorry | |
| 22:24:20 | melwitt | ok here's the last time it installs 4.0.0 before the error https://zuul.opendev.org/t/openstack/build/08e374906e2844ef865d4c698b3b78e5/log/job-output.txt#28518 | |
| 22:24:25 | gmann | but in final installed version in venv I always see 4.0.0 so not sure who & when it is installed/upgraded | |
| 22:24:41 | melwitt | I get confused by how it installs and uninstalls 4.0.0 repeatedly | |
| 22:25:31 | melwitt | venv installdeps: -c/opt/stack/old/requirements/upper-constraints.txt, -r/opt/stack/old/tempest/requirements.txt, -r/opt/stack/old/tempest/doc/requirements.txt | |
| 22:32:47 | gmann | as it fallback to latest bcz u-c not matched, we should use the compatible constraint while we cut the tempest 26.0.0 instead of stable/stein by default. | |
| 22:37:15 | gmann | at least that is tested combination tempest we released tempest 26.0.0 | |
| 22:38:28 | melwitt | what does it mean u-c didn't match? | |
| 22:40:00 | gmann | tempest 26.0.0 requirement has jsonschema>=3.2.0 and u-c jsonschema== 2.6.0 - https://github.com/openstack/tempest/blob/26.0.0/requirements.txt#L6 | |
| 22:40:16 | melwitt | ohhh I see. ok | |
| 22:40:17 | gmann | that is why it try to go for latest jsonschema | |
| 22:41:25 | gmann | complex thing here is to make tempest venv work on single (non-master) constraints, we have to take care in many place otherwise tox will recreate the tempest venv. many place I mean in devstack, run-tempest role, tox.ini etc | |
| 22:41:39 | melwitt | I see now. I didn't know it would ignore u-c if the requirement was higher than it ... I thought it would have failed like "can't install the tempest that you want" | |
| 22:42:12 | clarkb | melwitt: it should always use the constraint | |
| 22:42:34 | clarkb | or fail I guess, but ya if it doesn't do that I would consider that a bug in pip | |
| 22:42:57 | gmann | it actuall try to lowerdown the version as per -c but we have verify-config script from tempest running tempest in between so it fail | |
| 22:43:09 | gmann | otherwise the final version could be as per u-c only | |
| 22:43:21 | melwitt | hm O.o | |
| 22:43:23 | gmann | like it happen in master testing, it install 3.2.0 as per u-c not latest | |
| 22:44:25 | melwitt | sorry I'm getting even more confused. it's the verify-config script from tempest that is installing > u-c? | |
| 22:45:14 | gmann | melwitt: no in between somehow latest jsonschema is installed and verify-config run fail. why it install latest jsonschema and not failing on version conflict might be bug as clarkb mentioned | |
| 22:46:00 | gmann | basically in both case either failed with conflict or latest jsonschema, our constraint mismatch is issue | |
| 22:46:26 | melwitt | yeah. ok | |
| 22:47:14 | clarkb | Looking at the log pip says "tempest 26.0.0 has requirement jsonschema>=3.2.0, but you'll have jsonschema 2.6.0 which is incompatible." then it installs jsonschema 2.6.0 | |
| 22:47:20 | clarkb | I think constraitns are working properly as a result | |
| 22:47:31 | clarkb | (that was my understanding of how they work, constraints always wins) | |
| 22:47:41 | gmann | we have two way to solve it 1. use tempest tag (<26.0.0) which is compatible with stable/stein constraints 2. use compatible constraints (higher than of stable/stein) with tempest 26.0.0 | |
| 22:47:54 | clarkb | your requirements and constraints are at odds with each other | |
| 22:47:58 | gmann | yeah | |
| 22:48:27 | melwitt | yeah, that's why I wondered if https://github.com/openstack/tempest/blob/5d7e46f5689040ddaff798662f7a44fc71758d2e/tox.ini#L14 is doing something wrong, the log says that "tempest full" is what installed 4.0.0 | |
| 22:48:30 | gmann | requirement is from tempest 26.0.0 which is cut at the time of stable/vicrotia | |
| 22:49:30 | gmann | clarkb: but it does not say who installed 4.0.0 but we see that version as final version in installed version in venv | |
| 22:50:03 | clarkb | are you sure 4.0.0 is the final version? | |
| 22:50:32 | clarkb | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_08e/791807/5/check/neutron-grenade-multinode/08e3749/controller/logs/old/devstacklog.txt shows it is failing because jsonschema.compat isn't present which I assume is in >=3.2.0 | |
| 22:50:34 | gmann | clarkb: https://zuul.opendev.org/t/openstack/build/08e374906e2844ef865d4c698b3b78e5/log/job-output.txt#28518 | |
| 22:51:49 | gmann | they seems removed in 4.0.0 that is why installing 4.0.0. end up started failing due to this constraints and requirement mismatch | |
| 22:52:29 | gmann | I checked before 4.0.0 release we were istalling latest and it was juts working because of no incompatible change there | |
| 22:52:58 | clarkb | ok I think I see it | |
| 22:53:02 | clarkb | there are two different venvs | |
| 22:53:07 | clarkb | 'venv' and 'venv-tempest | |
| 22:53:47 | clarkb | they both install jsonschema 4.0.0 using venv-tempest installdeps: -c/opt/stack/old/requirements/upper-constraints.txt, -r/opt/stack/old/tempest/requirements.txt and venv installdeps: -c/opt/stack/old/requirements/upper-constraints.txt, -r/opt/stack/old/tempest/requirements.txt, -r/opt/stack/old/tempest/doc/requirements.txt | |
| 22:54:05 | clarkb | But venv-tempest has an extra step where it does venv-tempest run-test: commands[0] | pip install -c /tmp/tempest_u_c_m.03mOFK3yxm -r requirements.txt | |
| 22:54:31 | clarkb | It is exceptionally odd to me to run pip in the run-test portion of a tox target | |
| 22:54:36 | clarkb | but there you have it | |
| 22:55:51 | clarkb | looks like venv-tempest does command = {posargs} so I'm not sure what calls it with that to downgrade | |
| 22:56:23 | clarkb | but that modifies .tox/tempest and not .tox/venv | |
| 22:56:42 | clarkb | then you try ot run something in .tox/venv and it breaks due to the newer version of jsonschema | |
| 22:57:09 | melwitt | my head is spinning 😆 | |
| 22:57:59 | melwitt | but I'm very glad to know what is going on. I'll need time to process it | |
| 22:58:36 | clarkb | https://opendev.org/openstack/devstack/src/branch/master/lib/tempest#L627 that is where the pip install that puts jsonschema 2.6.0 into .tox/tempest via the venv-tempest target comes from | |
| 22:58:59 | clarkb | melwitt: my head is spinning too :) there is a lot of back and forth in here and I'm not really sure I understand why as much as now I at least understand how that exception arises | |
| 22:59:51 | melwitt | yeah... I hope there's a way to fix the root cause, having it be able to do this seems too confusing and error prone | |
| 23:00:12 | clarkb | then we run https://opendev.org/openstack/devstack/src/branch/master/lib/tempest#L649 and it explodes because the version is too new | |
| 23:00:55 | clarkb | gmann: I think that means you need a newer version of tempest that works against newer jsonschema as constrained there? | |
| 23:01:05 | clarkb | rather than older | |
| 23:01:33 | gmann | older | |
| 23:02:24 | clarkb | hrm wait -c/opt/stack/old/requirements/upper-constraints.txt that should be using stable/stein requirements in old/ right? | |
| 23:02:30 | clarkb | because this job ran against train | |
| 23:02:44 | clarkb | stable/stein requirements u-c says jsonschema should be 2.6.0 | |
| 23:03:20 | melwitt | right | |
| 23:04:15 | clarkb | the job-output.txt for that build has made my browser very sad | |
| 23:04:30 | melwitt | my browser has a sad as well | |
| 23:06:12 | gmann | heh, i opened three and it stuck | |
| 23:06:33 | gmann | clarkb: melwitt https://review.opendev.org/c/openstack/devstack/+/812092 | |
| 23:06:36 | gmann | let' see | |
| 23:06:58 | gmann | I am going with 1st solution of lowering the tempest as per stable/stein constraints | |
| 23:07:03 | clarkb | gmann: if that fixes it it doesn't explain why old/requirements somehow became newer requirements | |
| 23:07:24 | clarkb | I do see that old/requirements is initially checked out to 3a3ed8a7 which is correct | |
| 23:07:24 | gmann | clarkb: did not get? | |
| 23:07:43 | clarkb | gmann: requirements u-c for stable/stein has jsonschema===2.6.0 in it | |
| 23:07:50 | clarkb | gmann: that means we should never have installed 4.0.0 | |
| 23:07:59 | gmann | tempest 26.0.0 requirement has jsonschema>=3.2.0 and u-c jsonschema== 2.6.0 - https://github.com/openstack/tempest/blob/26.0.0/requirements.txt#L6 | |
| 23:08:00 | clarkb | that implies to me that the u-c file we are using is not for stable/stein | |
| 23:08:12 | clarkb | gmann: yes but we install 4.0.0 | |
| 23:08:19 | clarkb | which is from a newer u-c | |
| 23:08:27 | melwitt | yeah.. I don't know that there's a way to see the file | |
| 23:08:36 | melwitt | bc I was wondering the same thing earlier | |
| 23:08:42 | gmann | it try to check 2.6.0 as per u-c | |
| 23:09:08 | gmann | 4.0.0 is coming from somewhere, it is not yet in master u-c | |
| 23:09:34 | gmann | master u-c is also 3.2.0 | |
| 23:09:37 | clarkb | hrm | |
| 23:10:32 | gmann | how 4.0.0 is installed and u-c complain did not fail is not known to me | |
| 23:11:26 | clarkb | earlier in the job we see Requirement already satisfied: jsonschema===2.6.0 in /usr/local/lib/python3.6/dist-packages (from -c /opt/stack/old/requirements/upper-constraints.txt (line 232)) | |
| 23:11:37 | clarkb | that shows us that /opt/stack/old/requirements/upper-constraints.txt (line 232) is correct until that point | |
| 23:13:41 | melwitt | line 12300 is the first time I see jsonschema==4.0.0 | |
| 23:16:24 | melwitt | it does "lib/tempest:install_tempest:731: set_tempest_venv_constraints /tmp/tempest_u_c_m.IxpNU7ab69" before that but afaict TEMPEST_VENV_UPPER_CONSTRAINTS=/opt/stack/old/requirements/upper-constraints.txt always | |
| 23:16:48 | clarkb | melwitt: ya and a little later it uses /tmp/tempest_u_c_m.IxpNU7ab69 in a pip install in that venv which downgrades jsonschema | |
| 23:17:04 | clarkb | I wonder if this issue was known and that is why there is the explicit step after the fact of reinstalling with constraints | |
| 23:17:15 | melwitt | oh.... seems likely | |
| 23:17:26 | clarkb | basically for $some reason the initial tox setup doesn't do the right thing. So someone hacked in the extra pip install to devstack to fix things | |
| 23:17:40 | clarkb | except we don't do the extra pip install for the .tox/venv venv target later and it breaks | |
| 23:19:22 | melwitt | there's this note that I don't understand https://github.com/openstack/devstack/blob/57a868dd874922a0caed8ace0dc0426f29129277/lib/tempest#L712 | |
| 23:19:47 | clarkb | ok I think I've got it | |
| 23:20:14 | clarkb | the tox venv setup happens in two passes. The first pass installs all requirements as listed in the requirements file with the constraints | |
| 23:20:18 | clarkb | then it pip install tempest | |
| 23:20:48 | clarkb | The expectation is that all of tempest's deps be previously installed properly but jsonschema isn't because tempest says it wants jsonschema>=3.2.0 | |
| 23:20:55 | clarkb | gmann: ^ maybe this is what you were saying. | |
| 23:21:04 | clarkb | I guess normally you don't have this mismatch because because tempest doesn't branch it gets weird | |