Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-09-30
15:58:11 gibi unpatched and patched
15:59:46 melwitt elodilles: hi, what is the other patch you think might be better for fixing l-c on stable/train? re: https://review.opendev.org/c/openstack/nova/+/811762/1#message-d28da6263de9241f21a437daf6a0b7273eec47e4
16:00:38 melwitt I'm not opinionated how to fix it, so if there is another way, I'm happy to rebase the patches onto that
16:01:58 opendevreview Merged openstack/nova master: Update contributor guide for Yoga https://review.opendev.org/c/openstack/nova/+/809936
16:12:31 opendevreview Merged openstack/nova master: Add regression test for bug #1943431 https://review.opendev.org/c/openstack/nova/+/810755
16:40:50 elodilles melwitt: I though about this one: https://review.opendev.org/c/openstack/nova/+/810461
16:41:25 elodilles I missed that it is not yet merged o:)
16:41:48 elodilles so anyway, both ways work for me
17:01:18 melwitt elodilles: ah ok, thanks for the link. does it need an update based on the latest comments about min version? or would that be a separate patch?
17:14:13 melwitt elodilles: also meant to tell you I saw grenade jobs on stable/train also failing with "AttributeError: module 'jsonschema' has no attribute 'compat'" https://zuul.opendev.org/t/openstack/build/08e374906e2844ef865d4c698b3b78e5/log/job-output.txt#28564 is this also related to the setuptools issue?
17:34:40 elodilles melwitt: oh, I haven't seen that yet :( that most probably the very same jsonschema 4.0.0 release from yesterday that we faced in openstack-releases' validator :(
17:37:56 elodilles though jsonschema should be upper constrained in stable O.o
17:39:18 melwitt hm yeah
17:41:02 melwitt elodilles: oh but the error was raised from tempest, which would be coming from master
17:43:43 elodilles actually tempest should be used from train-last tag and still, should be installed with upper constraints. or am I wrong?
17:44:52 elodilles seemingly I am wrong at some point otherwise it wouldn't fail :)
17:46:34 melwitt well.. not sure. because if this fails here it seems like it would fail in the tempest repo itself but it's not
17:48:22 melwitt I see that the job began collecting jsonschema==2.6.0 but ended up installing 4.0.0
17:48:54 melwitt but why...
17:49:37 melwitt or how. it's pinned to 3.0.2 as you said earlier https://github.com/openstack/requirements/blob/stable/train/upper-constraints.txt#L634
17:49:53 melwitt collecting: https://zuul.opendev.org/t/openstack/build/08e374906e2844ef865d4c698b3b78e5/log/job-output.txt#38834
17:51:41 melwitt ok but then it uninstalls 4.0.0 and installs 2.6.0 here https://zuul.opendev.org/t/openstack/build/08e374906e2844ef865d4c698b3b78e5/log/job-output.txt#38886-38895
17:52:17 melwitt and then tempest installs 4.0.0 https://zuul.opendev.org/t/openstack/build/08e374906e2844ef865d4c698b3b78e5/log/job-output.txt#55512
17:52:53 melwitt gmann: does tempest running on stable branch not use upper-constraints from stable? ^
17:55:23 elodilles actually, since this is a grenade job, it should be stein
17:55:44 elodilles where jsonschema is constrained 2.6.0
17:56:09 melwitt oh right
17:56:40 elodilles but tempest @ stein-last requires: jsonschema>=3.2.0 # MIT
17:57:08 elodilles so it is strange how it works
18:00:42 elodilles (just for the record, currently jsonschema===3.2.0 is in the upper-constraints.txt @ master)
18:12:47 melwitt yeah, I don't understand it
18:23:18 opendevreview Lee Yarwood proposed openstack/nova master: trivial: Rename LibvirtConfigGuestPCIeRootPortController https://review.opendev.org/c/openstack/nova/+/812002
20:07:25 opendevreview Merged openstack/nova master: Store old_flavor already on source host during resize https://review.opendev.org/c/openstack/nova/+/810909
21:32:01 gmann melwitt:elodilles: in train, we still using tempest master which I need to move to train-last soon
21:32:49 melwitt gmann: lmk if I can help. stable/train currently broken due to new jsonschema 4.0.0 from master
21:32:56 gmann in stein yes tempest 26.0.0 is used
21:33:39 gmann melwitt: ok. I need to go out to pick up my wife. I will check after coming back
21:35:20 melwitt gmann: np thanks, I can work on it, just I would need a hint where to make a change
21:39:32 gmann melwitt: at first glance, it seems it is using stein branch constraint only https://zuul.opendev.org/t/openstack/build/08e374906e2844ef865d4c698b3b78e5/log/job-output.txt#12290
21:41:10 melwitt hm ok
22:16:11 gmann melwitt: I think tempest job on stable/stein also should fail on this.
22:17:15 melwitt gmann: yeah.. I've been looking at how it says it's using the stein upper-constraints file, but then after it does "tox -r -notest -efull" the 4.0.0 jsonschema gets installed. I don't understand that
22:17:56 gmann melwitt: it is pinned to used stable/stein u-c in stable/stein with tempest 26.0.0
22:18:04 melwitt so I was looking at https://github.com/openstack/tempest/blob/5d7e46f5689040ddaff798662f7a44fc71758d2e/tox.ini#L14 and I didn't know what the -c does but it looks like this is supposed to be use the env var upper-constraints file if present else use master?
22:18:45 gmann but it should not install latest jsonschema
22:19:22 melwitt here it says "full installed" ... jsonschema==4.0.0 https://zuul.opendev.org/t/openstack/build/08e374906e2844ef865d4c698b3b78e5/log/job-output.txt#12300
22:19:27 melwitt I took that to mean the tox -efull env installed 4.0.0
22:22:46 gmann but later on it uninstall 4.0.0 and install 2.6.0 https://zuul.opendev.org/t/openstack/build/08e374906e2844ef865d4c698b3b78e5/log/job-output.txt#12423-12427
22:23:21 melwitt oh, again? oh sorry
22:24:20 melwitt ok here's the last time it installs 4.0.0 before the error https://zuul.opendev.org/t/openstack/build/08e374906e2844ef865d4c698b3b78e5/log/job-output.txt#28518
22:24:25 gmann but in final installed version in venv I always see 4.0.0 so not sure who & when it is installed/upgraded
22:24:41 melwitt I get confused by how it installs and uninstalls 4.0.0 repeatedly
22:25:31 melwitt venv installdeps: -c/opt/stack/old/requirements/upper-constraints.txt, -r/opt/stack/old/tempest/requirements.txt, -r/opt/stack/old/tempest/doc/requirements.txt
22:32:47 gmann as it fallback to latest bcz u-c not matched, we should use the compatible constraint while we cut the tempest 26.0.0 instead of stable/stein by default.
22:37:15 gmann at least that is tested combination tempest we released tempest 26.0.0
22:38:28 melwitt what does it mean u-c didn't match?
22:40:00 gmann tempest 26.0.0 requirement has jsonschema>=3.2.0 and u-c jsonschema== 2.6.0 - https://github.com/openstack/tempest/blob/26.0.0/requirements.txt#L6
22:40:16 melwitt ohhh I see. ok
22:40:17 gmann that is why it try to go for latest jsonschema
22:41:25 gmann complex thing here is to make tempest venv work on single (non-master) constraints, we have to take care in many place otherwise tox will recreate the tempest venv. many place I mean in devstack, run-tempest role, tox.ini etc
22:41:39 melwitt I see now. I didn't know it would ignore u-c if the requirement was higher than it ... I thought it would have failed like "can't install the tempest that you want"
22:42:12 clarkb melwitt: it should always use the constraint
22:42:34 clarkb or fail I guess, but ya if it doesn't do that I would consider that a bug in pip
22:42:57 gmann it actuall try to lowerdown the version as per -c but we have verify-config script from tempest running tempest in between so it fail
22:43:09 gmann otherwise the final version could be as per u-c only
22:43:21 melwitt hm O.o
22:43:23 gmann like it happen in master testing, it install 3.2.0 as per u-c not latest
22:44:25 melwitt sorry I'm getting even more confused. it's the verify-config script from tempest that is installing > u-c?
22:45:14 gmann melwitt: no in between somehow latest jsonschema is installed and verify-config run fail. why it install latest jsonschema and not failing on version conflict might be bug as clarkb mentioned
22:46:00 gmann basically in both case either failed with conflict or latest jsonschema, our constraint mismatch is issue
22:46:26 melwitt yeah. ok
22:47:14 clarkb Looking at the log pip says "tempest 26.0.0 has requirement jsonschema>=3.2.0, but you'll have jsonschema 2.6.0 which is incompatible." then it installs jsonschema 2.6.0
22:47:20 clarkb I think constraitns are working properly as a result
22:47:31 clarkb (that was my understanding of how they work, constraints always wins)
22:47:41 gmann we have two way to solve it 1. use tempest tag (<26.0.0) which is compatible with stable/stein constraints 2. use compatible constraints (higher than of stable/stein) with tempest 26.0.0
22:47:54 clarkb your requirements and constraints are at odds with each other
22:47:58 gmann yeah
22:48:27 melwitt yeah, that's why I wondered if https://github.com/openstack/tempest/blob/5d7e46f5689040ddaff798662f7a44fc71758d2e/tox.ini#L14 is doing something wrong, the log says that "tempest full" is what installed 4.0.0
22:48:30 gmann requirement is from tempest 26.0.0 which is cut at the time of stable/vicrotia
22:49:30 gmann clarkb: but it does not say who installed 4.0.0 but we see that version as final version in installed version in venv
22:50:03 clarkb are you sure 4.0.0 is the final version?
22:50:32 clarkb https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_08e/791807/5/check/neutron-grenade-multinode/08e3749/controller/logs/old/devstacklog.txt shows it is failing because jsonschema.compat isn't present which I assume is in >=3.2.0
22:50:34 gmann clarkb: https://zuul.opendev.org/t/openstack/build/08e374906e2844ef865d4c698b3b78e5/log/job-output.txt#28518
22:51:49 gmann they seems removed in 4.0.0 that is why installing 4.0.0. end up started failing due to this constraints and requirement mismatch
22:52:29 gmann I checked before 4.0.0 release we were istalling latest and it was juts working because of no incompatible change there
22:52:58 clarkb ok I think I see it
22:53:02 clarkb there are two different venvs
22:53:07 clarkb 'venv' and 'venv-tempest
22:53:47 clarkb they both install jsonschema 4.0.0 using venv-tempest installdeps: -c/opt/stack/old/requirements/upper-constraints.txt, -r/opt/stack/old/tempest/requirements.txt and venv installdeps: -c/opt/stack/old/requirements/upper-constraints.txt, -r/opt/stack/old/tempest/requirements.txt, -r/opt/stack/old/tempest/doc/requirements.txt
22:54:05 clarkb But venv-tempest has an extra step where it does venv-tempest run-test: commands[0] | pip install -c /tmp/tempest_u_c_m.03mOFK3yxm -r requirements.txt
22:54:31 clarkb It is exceptionally odd to me to run pip in the run-test portion of a tox target
22:54:36 clarkb but there you have it
22:55:51 clarkb looks like venv-tempest does command = {posargs} so I'm not sure what calls it with that to downgrade
22:56:23 clarkb but that modifies .tox/tempest and not .tox/venv
22:56:42 clarkb then you try ot run something in .tox/venv and it breaks due to the newer version of jsonschema
22:57:09 melwitt my head is spinning 😆
22:57:59 melwitt but I'm very glad to know what is going on. I'll need time to process it
22:58:36 clarkb https://opendev.org/openstack/devstack/src/branch/master/lib/tempest#L627 that is where the pip install that puts jsonschema 2.6.0 into .tox/tempest via the venv-tempest target comes from
22:58:59 clarkb melwitt: my head is spinning too :) there is a lot of back and forth in here and I'm not really sure I understand why as much as now I at least understand how that exception arises
22:59:51 melwitt yeah... I hope there's a way to fix the root cause, having it be able to do this seems too confusing and error prone

Earlier   Later