Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-09-30
23:08:36 melwitt bc I was wondering the same thing earlier
23:08:42 gmann it try to check 2.6.0 as per u-c
23:09:08 gmann 4.0.0 is coming from somewhere, it is not yet in master u-c
23:09:34 gmann master u-c is also 3.2.0
23:09:37 clarkb hrm
23:10:32 gmann how 4.0.0 is installed and u-c complain did not fail is not known to me
23:11:26 clarkb earlier in the job we see Requirement already satisfied: jsonschema===2.6.0 in /usr/local/lib/python3.6/dist-packages (from -c /opt/stack/old/requirements/upper-constraints.txt (line 232))
23:11:37 clarkb that shows us that /opt/stack/old/requirements/upper-constraints.txt (line 232) is correct until that point
23:13:41 melwitt line 12300 is the first time I see jsonschema==4.0.0
23:16:24 melwitt it does "lib/tempest:install_tempest:731: set_tempest_venv_constraints /tmp/tempest_u_c_m.IxpNU7ab69" before that but afaict TEMPEST_VENV_UPPER_CONSTRAINTS=/opt/stack/old/requirements/upper-constraints.txt always
23:16:48 clarkb melwitt: ya and a little later it uses /tmp/tempest_u_c_m.IxpNU7ab69 in a pip install in that venv which downgrades jsonschema
23:17:04 clarkb I wonder if this issue was known and that is why there is the explicit step after the fact of reinstalling with constraints
23:17:15 melwitt oh.... seems likely
23:17:26 clarkb basically for $some reason the initial tox setup doesn't do the right thing. So someone hacked in the extra pip install to devstack to fix things
23:17:40 clarkb except we don't do the extra pip install for the .tox/venv venv target later and it breaks
23:19:22 melwitt there's this note that I don't understand https://github.com/openstack/devstack/blob/57a868dd874922a0caed8ace0dc0426f29129277/lib/tempest#L712
23:19:47 clarkb ok I think I've got it
23:20:14 clarkb the tox venv setup happens in two passes. The first pass installs all requirements as listed in the requirements file with the constraints
23:20:18 clarkb then it pip install tempest
23:20:48 clarkb The expectation is that all of tempest's deps be previously installed properly but jsonschema isn't because tempest says it wants jsonschema>=3.2.0
23:20:55 clarkb gmann: ^ maybe this is what you were saying.
23:21:04 clarkb I guess normally you don't have this mismatch because because tempest doesn't branch it gets weird
23:21:47 melwitt ok so when it installs tempest it doesn't use u-c? every command I saw seemed to be passing a u-c file though, that's what I don't get
23:22:15 clarkb melwitt: right tox does a two step install. It installs everything listed in the deps directive. Then it installs the actual repo it is in
23:22:23 clarkb The second step install isn't suepr configurable iirc
23:22:36 clarkb gmann: https://opendev.org/openstack/tempest/src/tag/24.0.0/requirements.txt#L6 is the newest tempest with jsonschema 2.6.0
23:24:10 gmann clarkb: yes, that is last tag with 2.6.0
23:25:13 gmann and on the tox step: yes and before u-c things setup the right version verify-config run make it fail
23:25:38 gmann which is run directly with venv not 'going inside the venv and then run '
23:26:23 gmann 'directly with vnev' i mean it create venv first and then run before u-c things happen
23:27:04 gmann tox -r --notest -efull is also very first creation of vnev but as it does not run anything so it just does not complain on 4.0.0
23:27:33 clarkb gmann: well full also fixes the constraints after the fact
23:28:37 gmann yes after it fix but in verify-config case it run tempest things before constraints fixes. that is what i am suspecting. 'installing with req and then later at some step tox handle the constraints'
23:29:58 gmann like, tox installing all as per req -> running verify-config -> will take care of constraints(but it fail before verify-config itself)
23:30:08 clarkb https://tox.wiki/en/latest/config.html#conf-install_command I think that can be used to make this better
23:30:23 gmann that is sequence I am suspecting which causing failure. thoguh we had bad compatiblity on req and constraint
23:30:37 clarkb basically create an install command that has -c /path/to/constraints/ then the same constraints application will be used on deps and the actual package
23:30:51 gmann clarkb: yeah, we should do that way
23:33:09 gmann clarkb: and it was like that before jsonschema 4.0.0 which could have been caught if installtion fail on req-constraints mismatch
23:33:45 clarkb well constraints never cause a failure at install time. The constraint just wins always
23:33:49 clarkb but you get the warning about it at least
23:35:29 gmann ah right.
23:35:47 gmann clarkb: melwitt let's see if this work (it should) https://review.opendev.org/c/openstack/devstack/+/812092
23:36:22 gmann and I will say we always use first compatible stable version of tempest insetad of last and figure out the compatible newer constraints
23:36:30 gmann for EM stable
23:36:31 clarkb gmann: you might try to use 24.0.0 instead
23:36:38 clarkb and then treat that as the last compatible version
23:36:59 gmann clarkb: 24.0.0 might create issue in future if anything else newer comes as it is not tested on stable/stein constraints
23:37:08 gmann for jsonschema might work
23:38:18 gmann I was too optimistic on using the last compatible(newer) tempest with old constraints and assumed it will work :)
23:39:29 clarkb gmann: it will work if you use the tox install command thing. But I guess the ship has sailed for old tags
23:39:41 clarkb but if you update the tox.ini before the next tag it would be corrected for the future
23:40:22 gmann clarkb: yes that I will do for next tag, agree on that
23:40:35 gmann clarkb: and another things with constraints is, as we touch tempest venv using tox command in many place like running test in grenade/run-tempest role ect if constraints mismatch at any place it will re-create the tox which is another issue
23:41:15 gmann so simple things for EM stable is use tested compatible version and constraints and do not assume the things :).
23:41:47 clarkb ya if 20.0.0 is the most aligned with stein's constraints that seems like the one to use
23:42:03 gmann yes, that is version released while openstack stein release.
23:43:11 gmann for tox recreation things I was searching for any flag where we can stop tox to recreate and force-live with what is installed but there is no flag seems
23:43:41 gmann that could have solved our problem with tempest venv testing case
23:44:21 gmann clarkb: melwitt 20.0.0 seems working. https://zuul.openstack.org/stream/9234f33acf1247cdb995416ca9389240?logfile=console.log
23:45:17 gmann let me put nova train DNM patch also as grenade also toush tox env while running test
23:47:25 melwitt thank you gmann
23:47:27 opendevreview Ghanshyam proposed openstack/nova stable/train: DNM: testing nova grenade job https://review.opendev.org/c/openstack/nova/+/812095
23:47:29 gmann ^^
23:47:34 melwitt and clarkb++
23:48:22 gmann I will add note in devstack/tempest about it while pinning tempest and its constraints for future EM stable
23:48:46 gmann and install-command thing
23:49:41 melwitt yes please \o/
23:51:35 gmann melwitt: sorry about issue, its tempest constraints things on EM stable always cause new issue over time. and honestly saying I am not 100% confident if that happen in future too :) but at least testing with version released duing that stable branch will make things simple
23:52:22 gmann * I am not 100% confident that it will not happen in future too :
23:53:46 melwitt gmann: I'm not complaining as I barely understand all this :) just happy you'll be able to do something to fix or make it less likely to happen in the future
23:54:48 gmann sure, will do those doc things tomorrow if it works fine in grenade too.
23:56:06 gmann melwitt: ah lower constraints seem broken on stable/train. is it new ? https://zuul.opendev.org/t/openstack/build/2cdc193e00974abcb67b58ffe566f371
23:56:25 melwitt gmann: no it's not. sec
23:56:55 gmann melwitt: it seems failing in other old patches too https://review.opendev.org/c/openstack/nova/+/811824
23:58:12 melwitt gmann: yeah. there's two options, one is make l-c non-voting https://review.opendev.org/c/openstack/nova/+/811762 and the other is pin setuptools https://review.opendev.org/c/openstack/nova/+/810461
23:58:33 gmann ah that one. i got it
23:58:42 gmann I voted for first even remove testing l-c
23:59:21 melwitt it sounded like consensus favors the latter. I was just not sure if it needs an update before approving, based on discussion on the patch
23:59:23 melwitt ah, heh. yeah. I am neutral about it
23:59:55 melwitt since we have a fix, we can fix this time ... until next break :)
#openstack-nova - 2021-10-01
00:00:41 gmann sure. but I am sure we will continue spending time every time and always find solution :P just joking
00:00:52 gmann we means except me :)
00:00:53 melwitt hehe
05:49:43 opendevreview melanie witt proposed openstack/nova master: Add logic to enforce local api and db limits https://review.opendev.org/c/openstack/nova/+/712139
05:49:43 opendevreview melanie witt proposed openstack/nova master: Enforce api and db limits https://review.opendev.org/c/openstack/nova/+/712142
05:49:44 opendevreview melanie witt proposed openstack/nova master: Update quota_class APIs for db and api limits https://review.opendev.org/c/openstack/nova/+/712143
05:49:44 opendevreview melanie witt proposed openstack/nova master: Update limit APIs https://review.opendev.org/c/openstack/nova/+/712707
05:49:45 opendevreview melanie witt proposed openstack/nova master: Update quota sets APIs https://review.opendev.org/c/openstack/nova/+/712749
05:49:45 opendevreview melanie witt proposed openstack/nova master: Tell oslo.limit how to count nova resources https://review.opendev.org/c/openstack/nova/+/713301
05:49:46 opendevreview melanie witt proposed openstack/nova master: Enforce resource limits using oslo.limit https://review.opendev.org/c/openstack/nova/+/615180
05:49:46 opendevreview melanie witt proposed openstack/nova master: Add legacy limits and usage to placement unified limits https://review.opendev.org/c/openstack/nova/+/713498
05:49:47 opendevreview melanie witt proposed openstack/nova master: Update quota apis with keystone limits and usage https://review.opendev.org/c/openstack/nova/+/713499
05:49:47 opendevreview melanie witt proposed openstack/nova master: Add reno for unified limits https://review.opendev.org/c/openstack/nova/+/715271
06:53:45 opendevreview melanie witt proposed openstack/nova master: Enforce api and db limits https://review.opendev.org/c/openstack/nova/+/712142
06:53:46 opendevreview melanie witt proposed openstack/nova master: Update quota_class APIs for db and api limits https://review.opendev.org/c/openstack/nova/+/712143
06:53:46 opendevreview melanie witt proposed openstack/nova master: Update limit APIs https://review.opendev.org/c/openstack/nova/+/712707
06:53:47 opendevreview melanie witt proposed openstack/nova master: Update quota sets APIs https://review.opendev.org/c/openstack/nova/+/712749
06:53:47 opendevreview melanie witt proposed openstack/nova master: Tell oslo.limit how to count nova resources https://review.opendev.org/c/openstack/nova/+/713301
06:53:48 opendevreview melanie witt proposed openstack/nova master: Enforce resource limits using oslo.limit https://review.opendev.org/c/openstack/nova/+/615180

Earlier   Later