Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-12-09
10:14:32 stephenfin Hmm, I recall seeing something from mordred about this a while ago on openstack-discuss. Wonder if I can find it...
10:16:28 lyarwood actually let me write up a bug for this
10:16:29 stephenfin elod, lyarwood: Okay, this is what I was thinking of http://lists.openstack.org/pipermail/openstack-discuss/2020-March/013642.html
10:17:05 elod just found it, too, but have to re-read :)
10:17:27 stephenfin Not exactly it, but it does describe the issue I was seeing with OSC. Specifically "That means we uninstall and reinstall flake8 at different versions over and over again - and the final state is not one that is completely consistent."
10:17:39 stephenfin pip went nuts when those were included
10:18:24 stephenfin With that said, we have out own hacking plugins and tests for same, which means we do actually depend on those things to run unit tests. I don't know what the answer for that is :-\
10:18:41 stephenfin Maybe it's just "Not A Problem" (TM)? :)
10:20:20 openstackgerrit Lee Yarwood proposed openstack/nova stable/train: Cap bandit at 1.6.2 when using py2 https://review.opendev.org/c/openstack/nova/+/766171
10:20:21 openstackgerrit Lee Yarwood proposed openstack/nova stable/train: libvirt: Skip encryption metadata lookups if secret already exists on host https://review.opendev.org/c/openstack/nova/+/765771
10:20:27 lyarwood oh wait, so adding the cap in test-reqs isn't going to work?
10:20:48 elod the uninstall and reinstall is mainly a problem in devstack. but test-req installation is removed in devstack, so that's not a problem anymore.
10:21:23 elod lyarwood: i think it will work
10:22:10 lyarwood okay something is still installing bandit in the grenade jobs as well on stable/train FWIW
10:22:16 lyarwood brb
10:22:36 elod the question is whether there are some benefit if we move linters to tox.ini instead. which I don't see yet, as that would add another place where we should look for dependencies... but maybe I'm wrong :X
10:24:33 stephenfin elod: Is it actually a dependency? You don't need it to run the main code nor the tests (for anything that doesn't have tests for custom linters, that is). It's a dependency but only in the same way tox is a dependency
10:25:13 lyarwood stephenfin: there's a bandit env in tox
10:25:30 lyarwood stephenfin: I assume that's why it's there?
10:25:47 stephenfin I don't get you. wdym?
10:26:05 lyarwood [testenv:bandit]
10:26:07 lyarwood # NOTE(browne): This is required for the integration test job of the bandit
10:26:09 lyarwood # project. Please do not remove.
10:26:11 lyarwood envdir = {toxworkdir}/shared
10:26:13 lyarwood commands = bandit -r nova -x tests -n 5 -ll
10:26:15 lyarwood ^ in tox.ini on stable/train
10:27:06 stephenfin oh, okay, I'm not saying we don't need to specify bandit somewhere. I'm saying we don't need to do it in test-requirements.txt because it doesn't need to be subject to e.g. lower-constraints checks
10:27:14 stephenfin we can do it in tox.ini instead
10:27:18 lyarwood ah right sorry
10:27:49 lyarwood yeah well I get elod's point that it's just another place to look for deps but if it isn't needed outside of that tox env then I'd be okay with just listing it there in tox.ini
10:28:11 stephenfin In case it helps, the way I was diagnosing those lower-constraints jobs yesterday was to create a new virtualenv, update pip and run the same command as the lower-constraints tox target
10:28:20 lyarwood do you want to push a change on master?
10:28:35 stephenfin I had to do that because on Fedora 33, I get Python 3.9 in my virtualenv which isn't compatible with a few of the dependencies
10:28:53 lyarwood yeah I just hacked the base python version when working on this the other day
10:29:08 lyarwood upgraded pip and reproduced the issue
10:29:14 lyarwood but that's different to this issue again
10:29:23 stephenfin Fair
10:29:24 lyarwood this was just an uncapped dep dropping py2 support
10:29:53 lyarwood tbh I think we might need to do that until py39 is actually supported
10:30:07 lyarwood otherwise some of us on modern distros get stung all the damn time
10:30:17 stephenfin yuuup
10:30:49 stephenfin fwiw, you can also do this
10:31:02 lyarwood oh cool there's a LC failure now on stable/train as well
10:31:03 stephenfin TOX_CONSTRAINTS_FILE=lower-constraints.txt tox -e py36
10:31:22 stephenfin sub UPPER_ for TOX_ on pre-victoria iirc
10:31:31 stephenfin \o/
10:31:41 elod wasn't there another discussion in mailing list that the usage of basepython is discouraged? o:)
10:32:03 stephenfin unless this was recently, I fixed that
10:32:16 lyarwood ERROR: No matching distribution found for hacking<1.2.0,>=1.1.0
10:32:23 lyarwood ^ stephenfin was that the LC issue you were working on?
10:33:36 elod this 'no matching distro' issue seems more like some mirror thing to me. (and again, I might be wrong :X)
10:33:50 stephenfin Not that exact, but it looks familiar. That happens because it can't match the dependencies of that hacking version with those required by other dependencies
10:33:54 stephenfin It's very misleading
10:34:18 lyarwood yeah I thought that the other day but talking to fungi we found https://review.opendev.org/c/openstack/nova/+/765824
10:34:42 lyarwood stephenfin: yeah indeed it's an awful error message
10:35:40 elod :S
10:38:48 lpetrut hi, I have a quick question about the lower-constraints file: it's supposed to contain only direct dependencies, right? for example, if we need package x, which in turn requires package y, would package y need to be in lower-constraints.txt?
10:39:12 stephenfin lpetrut: it would, yes
10:39:47 stephenfin However, we haven't been very good around managing that since the tooling situation is quite poor
10:40:09 stephenfin So I don't think anyone is going to hold it against you in a review
10:41:18 lpetrut stephenfin: thanks for clearing it out. yep, it's really difficult to maintain, I was hoping to be able to limit lower-constraints to direct dependencies
10:41:57 lpetrut but I guess that would affect its usefulness
10:43:44 lyarwood stephenfin: which version of py36 are you using btw?
10:44:52 lyarwood stephenfin: everything is borked for me with 3.6.12 with setuptools 49.1.3 /o\
10:45:21 lyarwood and https://github.com/pypa/setuptools/issues/2017
10:45:55 stephenfin I was using 3.6.12, but that was with OSC, not nova
10:46:28 lpetrut lywarwood: looks like I'm not the only one having a hard time chasing Python dependencies :)
10:46:30 lyarwood yeah it's failing to install MarkupSafe==1.0
10:46:58 lyarwood lpetrut: yup don't you just love python some days? :)
10:51:07 sean-k-mooney lowerconstriat has more then direct depencies. it was auto generated using pip freeze
10:51:20 sean-k-mooney we have removed some of the indirect deps but not all of them
10:52:40 sean-k-mooney lyarwood: just be glad your not dealing with nodejs
11:05:53 stephenfin lyarwood: you've probably figured this out already but https://github.com/openstack/nova/blob/stable/train/lower-constraints.txt#L37
11:06:05 stephenfin https://github.com/openstack/nova/blob/stable/train/test-requirements.txt#L5
11:06:43 lyarwood yeah I've been trying to prove that was the issue but I can't build a venv
11:06:51 lyarwood let me squash this into my other change
11:07:06 stephenfin I'm doing
11:07:08 stephenfin deactivate; rm -rf .venv; virtualenv .venv --python=python3.6; source .venv/bin/activate; pip install --upgrade pip; pip install -vvv -c lower-constraints.txt 'hacking>=1.1.0,<1.2.0'
11:07:10 stephenfin on repeat
11:07:20 stephenfin while playing with lower-constraints.txt
11:09:55 openstackgerrit Lee Yarwood proposed openstack/nova stable/train: Cap bandit and raise hacking lower-constraint https://review.opendev.org/c/openstack/nova/+/766171
11:09:56 openstackgerrit Lee Yarwood proposed openstack/nova stable/train: libvirt: Skip encryption metadata lookups if secret already exists on host https://review.opendev.org/c/openstack/nova/+/765771
11:10:16 lyarwood oh right because that moves flake etc
11:10:18 lyarwood gah
11:10:58 lyarwood so broken
11:11:54 stephenfin I wonder if we can start using train-era virtualenv and pip?
11:13:38 openstackgerrit Lee Yarwood proposed openstack/nova stable/train: Cap bandit while also raising hacking and flake lower-constraints https://review.opendev.org/c/openstack/nova/+/766171
11:13:39 openstackgerrit Lee Yarwood proposed openstack/nova stable/train: libvirt: Skip encryption metadata lookups if secret already exists on host https://review.opendev.org/c/openstack/nova/+/765771
11:14:08 lyarwood my issue was more with the version of setuptools being pulled in by default by virtualenv, I don't think that's tied to the version of virtualenv itself right?
11:14:19 stephenfin I think it is
11:14:22 lyarwood ah
11:14:30 lyarwood I thouight that was a python version thing
11:14:34 stephenfin I saw something from fungi...somewhere this morning
11:14:59 sean-k-mooney stephenfin: im not sure about that you can tell virtualenv to download and decompress setuptools
11:15:08 sean-k-mooney i think by default it uses your host copy
11:15:58 sean-k-mooney there is a --setuptools<version> flag
11:16:46 stephenfin if that was the case, why is the gate failing? They (Canonical) are hardly releasing new versions of pip on 18.04 still
11:16:47 sean-k-mooney --download might also be needed
11:16:53 sean-k-mooney but i think that is for latest
11:17:01 openstackgerrit Lee Yarwood proposed openstack/nova stable/train: [stable-only] Cap bandit while also raising hacking and flake lower-constraints https://review.opendev.org/c/openstack/nova/+/766171

Earlier   Later