| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-09-21 | |||
| 16:34:16 | sean-k-mooney | gibi: technially i dont think we are allowe to bump miniums on stable on the other hand we already did it a while ago for the inital lower constraitnts fix | |
| 16:34:26 | gibi | on placement we are impacted on master but the fix has been landed | |
| 16:34:40 | gibi | on stable/xena we could release RC2 with the bump | |
| 16:34:47 | elodilles | I also like option 2, and actually found some (abandoned) trial from the past that was similar https://review.opendev.org/q/topic:constrain-tox-install | |
| 16:34:50 | gibi | but on stable/wallaby and back the same quertion applies | |
| 16:35:12 | sean-k-mooney | from a disto point of view 2 is not great | |
| 16:35:22 | sean-k-mooney | since they cannot pin them the same way | |
| 16:35:25 | bauzas | option 1 has a question I can't answer | |
| 16:35:32 | sean-k-mooney | but most distros also dont use lower constratints | |
| 16:35:33 | gibi | does LTS distros use unconstrained setuptools as well? | |
| 16:35:54 | bauzas | can we deliver a .y release by bumping the decorator dependency without breaking the semver rules ? | |
| 16:36:07 | sean-k-mooney | rhel ignored the upper and lower constraitnts | |
| 16:36:14 | sean-k-mooney | debian used to look at lower | |
| 16:36:25 | sean-k-mooney | ubunut used to look at upper | |
| 16:36:46 | sean-k-mooney | we can try 2 but i feel like we are going to end up with 3 | |
| 16:37:00 | gibi | sean-k-mooney: sure we could end up with 3 for different reasons :) | |
| 16:37:12 | artom | So 2 for distros means "maintain this old version of virtualenv in your repos", right? | |
| 16:37:22 | elodilles | bauzas: I think we can, but it's probably not fortunate. semver says that req changes needs MINOR / .y version bump | |
| 16:37:28 | artom | But... don't we already have upper-constraints that does the exact same thing? | |
| 16:37:36 | gibi | artom: yes, as virtualenv bundles setuptools | |
| 16:37:56 | bauzas | elodilles: so option 1 doesn't break stable policy per se | |
| 16:38:00 | gibi | artom: this is tricky as the tox install in our CI does not use constraint file | |
| 16:38:14 | gibi | artom: we install tox, then with tox we install deps | |
| 16:38:25 | gibi | artom: but when we install tox it pulls in setuptools | |
| 16:38:59 | artom | Oh, so this is before any -constraints is applied | |
| 16:39:15 | bauzas | the problem is with the venv, not the dependencies | |
| 16:39:20 | bauzas | artom: right | |
| 16:39:30 | elodilles | bauzas: I also haven't found that written, but I remember that we tried to avoid req changes | |
| 16:39:30 | bauzas | because of the bundle | |
| 16:39:48 | bauzas | but, there are ways to create venvs without bundling setuptools, right? | |
| 16:40:05 | elodilles | bauzas: and the problem is that we will face the same issue whenever something new things comes in | |
| 16:40:12 | artom | Wait, I though the decorator==3.4.0 install happened *inside* the venv, as a dependency? | |
| 16:40:29 | bauzas | elodilles: agreed, option 1 only fixes the decorator issue and doesn't resolve any other issue | |
| 16:40:49 | bauzas | I'm also in favor of option 2 | |
| 16:41:05 | gibi | artom: to have a venv where you can install deps, you have to have the venv package installed | |
| 16:41:08 | bauzas | but I wonder whether we could just tell tox to create venvs without bundling setuptools | |
| 16:41:35 | gibi | bauzas: I think if you have a venv without setuptools then you dont have pip in the venv either | |
| 16:41:35 | stephenfin | just weighing in, 3 isn't an ideal solution either | |
| 16:41:48 | gibi | bauzas: so you cannot install additional things with pip | |
| 16:41:56 | bauzas | gibi: are you sure ? | |
| 16:42:02 | gibi | bauzas: 75% | |
| 16:42:05 | bauzas | I think it does install pip | |
| 16:42:14 | gibi | I think pip depends on setuptools | |
| 16:42:18 | stephenfin | in this case it's the lower-constraint that has broken, however, it's conceivable that the upper constraint for a particularly old release could also become incompatible with setuptools | |
| 16:42:20 | gibi | but I could be wrong | |
| 16:42:32 | gibi | stephenfin: good point | |
| 16:42:35 | artom | gibi, right, so we install tox on the "host", which pulls in unconstrained setuptools, which then break installing decorator==3.4.0, my question is, aren't we installing decorator==3.4.0 inside the venv? | |
| 16:42:47 | gibi | artom: hm, | |
| 16:42:48 | artom | So it should be independent of what's on the "host"? | |
| 16:42:53 | gibi | artom: you have a point | |
| 16:43:13 | gibi | artom: for some reason our ci install tox already in a venv but I lost following that track | |
| 16:43:40 | stephenfin | artom: it installs setuptools in the venv also, right? | |
| 16:44:05 | artom | Don't know... but then it'd be a separate venv for the actual nova install that pulls in decorator... | |
| 16:44:40 | gibi | I agree that something is missing from our understanding about venvs and tox install | |
| 16:45:00 | stephenfin | >>> setuptools.__file__ | |
| 16:45:06 | stephenfin | '/home/stephenfin/Development/openstack/nova/.tox/py36/lib/python3.6/site-packages/setuptools/__init__.py' | |
| 16:45:16 | sean-k-mooney | https://tox.readthedocs.io/en/latest/config.html#conf-requires | |
| 16:45:33 | sean-k-mooney | if we need to pin it we can pin the setuptools and pip version with requires | |
| 16:45:45 | bauzas | I think it's doable | |
| 16:46:03 | gibi | sean-k-mooney: good finding | |
| 16:46:03 | clarkb | tox pulls in virtualenv, virtualenv bundles setuptools for the new virtualenvs it makes | |
| 16:46:09 | sean-k-mooney | it is but fungi and clarkb advised against doing that | |
| 16:46:14 | elodilles | in my abandoned trial in the past i tried to use upper-constraints.txt for tox install, it probably could solve the issue if that works (i don't remember whether it worked) | |
| 16:46:16 | clarkb | virtualenv has an escape hatch for that but I'm not sure how tox exposes that (if at all) | |
| 16:46:24 | bauzas | https://paste.opendev.org/show/809477/ | |
| 16:46:27 | clarkb | you need to pin virtualenv is the tldr if you are going to pin something | |
| 16:46:34 | clarkb | but you have to do it when installing tox | |
| 16:46:46 | bauzas | you can create a venv without setuptools and pip it later | |
| 16:46:58 | clarkb | right but does tox support any of that? | |
| 16:47:04 | clarkb | as it is tox making the virtualenv automatically | |
| 16:47:15 | bauzas | tox uses the venv module, right?N | |
| 16:47:23 | clarkb | not by default I think it can | |
| 16:47:32 | artom | clarkb, gibi, ah, that's the missing understanding, when virtualenv creates a venv, it automatically stuffs the setuptools version it came with in that venv... right? | |
| 16:47:35 | clarkb | basically oyu have to see how much of this is configurable on the tox side to do what you want | |
| 16:47:41 | clarkb | artom: yes exactly | |
| 16:47:44 | opendevreview | Stephen Finucane proposed openstack/nova master: db: Add migration to resolve shadow table discrepancies https://review.opendev.org/c/openstack/nova/+/805738 | |
| 16:47:45 | opendevreview | Stephen Finucane proposed openstack/nova master: tests: Walk database migrations in correct order https://review.opendev.org/c/openstack/nova/+/810291 | |
| 16:48:14 | bauzas | clarkb: you're right, I need to dig into tox | |
| 16:48:20 | stephenfin | This looks like good background reading https://tox.readthedocs.io/en/latest/example/package.html | |
| 16:48:21 | gibi | clarkb: thanks | |
| 16:48:25 | artom | Sounds like we have no choice but to pin virtualenv then | |
| 16:48:30 | stephenfin | Sounds like the escape hatches we need might be there | |
| 16:48:42 | artom | Unless there's a way to tell virtualenv which setuptools version to install in the venvs in creates | |
| 16:48:53 | fungi | if you want tox to use venv as the virtualenv backend, these days the trick is to export VIRTUALENV_CREATOR = venv in the inherited setenv in your tox.ini | |
| 16:49:21 | fungi | there used to be a tox-venv plugin, but that was deprecated/abandoned once virtualenv grew the option to call out to the venv module itself | |
| 16:49:30 | bauzas | can we use https://tox.readthedocs.io/en/latest/config.html#conf-requires ? | |
| 16:49:53 | fungi | so it's not tox calling venv directly, but rather tox calling virtualenv and then virtualenv being told to use the venv module to create the env rather than its internal implementation | |
| 16:50:13 | sean-k-mooney | i feel like the complexity of maintianing this pining outways the usefullness of the job | |
| 16:50:29 | bauzas | sean-k-mooney: that's the problem with option 2 | |
| 16:50:45 | bauzas | ideally, I'd like us to maitain setuptools in our reqs | |
| 16:50:52 | bauzas | maintain* even | |
| 16:50:59 | bauzas | but this doesn't sound easy | |
| 16:51:03 | artom | So the value of the job is what - guarantee that will still work with the oldest feasible versions of things? | |
| 16:51:13 | artom | (lower-constraints I mean) | |
| 16:51:25 | stephenfin | sean-k-mooney: We will always have pinning in the form of upper-constraints though | |
| 16:51:26 | sean-k-mooney | yes basically to make sure when we backport we dont raise the requiremetns as a result | |
| 16:51:41 | clarkb | the correct way to maintain setuptools in your requirements is via the pyproject file | |
| 16:51:44 | sean-k-mooney | stephenfin: right but for stable we are not allowed to raise miniums | |
| 16:51:47 | clarkb | not requirements | |
| 16:52:14 | stephenfin | and? we can't raise maximums either | |