| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-09-16 | |||
| 17:57:00 | sean-k-mooney | i know a lot of project just deleted there old lower constriants jobs | |
| 17:58:52 | clarkb | the governance repo had problems with pydot2 which hasn't been maintained for years | |
| 17:59:21 | clarkb | I don't think we want ot downgrade setuptools if we can avoid it. We can't really control what version of setuptools others use effectively and wide compatibility is desireable | |
| 17:59:39 | clarkb | (note with the whole pyproject.toml stuff you do get a bit more control but openstack hasn'tdone any of that) | |
| 18:00:32 | sean-k-mooney | ya its more we have 3 options delete the job, update the dep or hack around to make it work | |
| 18:00:47 | sean-k-mooney | pinnign setup tools on an em branch is just that | |
| 18:00:48 | clarkb | can you bump the dep version up such that it works? | |
| 18:00:53 | sean-k-mooney | a hackaround to make it work | |
| 18:00:58 | clarkb | that is all lower constraints is supposed to track iirc. The oldest version that works | |
| 18:01:18 | sean-k-mooney | clarkb: well we are technialy not allow to bump min verison on stable right | |
| 18:01:28 | sean-k-mooney | but pratically speaking yes we could | |
| 18:01:36 | sean-k-mooney | and have to keeep it working already | |
| 18:02:19 | clarkb | ya for stable maybe removing the job entirely makes sense | |
| 18:02:52 | fungi | note the most recent setuptools release was a week ago, not sure if that lines up with when these failures started | |
| 18:03:22 | fungi | however tox updated today | |
| 18:03:32 | fungi | so it may have started vendoring a newer setuptools | |
| 18:03:47 | fungi | this is one of the reasons "pinning" setuptools is complicated | |
| 18:05:05 | sean-k-mooney | ya | |
| 18:05:13 | sean-k-mooney | looking at the releases https://pypi.org/project/decorator/#history | |
| 18:05:20 | sean-k-mooney | we are using 3.4.0 now | |
| 18:05:30 | sean-k-mooney | from october 2012 | |
| 18:05:35 | fungi | we, actually it's virtualenv which vendors setuptools, and that was also released today | |
| 18:05:39 | sean-k-mooney | the next release 3.4.1 is 2015 | |
| 18:06:16 | fungi | v20.8.0 (2021-09-16): upgrade embedded setuptools to 58.0.4 from 57.4.0 and pip to 21.2.4 from 21.2.3 | |
| 18:06:45 | fungi | so my guess is this is setuptools 58.x.x behavior brought in by tox 20.8 | |
| 18:07:30 | fungi | you could try manually downgrading setuptools<58 in the created env to find out, but i don't recommend that as a fix | |
| 18:09:14 | sean-k-mooney | ya im just trying to repodcue it localy now | |
| 18:09:22 | fungi | v58.0.0 Breaking Changes: Removed support for 2to3 during builds. Projects should port to a unified codebase or pin to an older version of Setuptools using PEP 518 build-requires. | |
| 18:09:31 | sean-k-mooney | GLOB sdist-make: /home/sean/repos/nova/setup.py | |
| 18:09:36 | sean-k-mooney | that new to me ^ | |
| 18:09:54 | sean-k-mooney | i have not seen a GLOB sdist-make line before | |
| 18:10:06 | sean-k-mooney | The conflict is caused by: | |
| 18:10:07 | sean-k-mooney | jinja2 2.10 depends on MarkupSafe>=0.23 | |
| 18:10:09 | sean-k-mooney | The user requested (constraint) markupsafe==1.0 | |
| 18:10:16 | sean-k-mooney | so we have other conflict too aparently | |
| 18:10:22 | fungi | and earlier i said tox 20.8 but meant virtualenv 20.8 | |
| 18:11:08 | sean-k-mooney | ImportError: cannot import name 'Feature' from 'setuptools' | |
| 18:11:15 | fungi | sean-k-mooney: yes, this is one of the reasons i contended calculating and maintaining a lower constraints list would be unworkable long term | |
| 18:12:36 | sean-k-mooney | the import error was from markupsafe with tox 3.20.1 | |
| 18:12:48 | sean-k-mooney | althoguh ok i ghet the same on ewith latest tox | |
| 18:12:59 | sean-k-mooney | 3.24.4 | |
| 18:13:24 | fungi | yeah, downgrading tox probably won't help. you more likely need to downgrade setuptools, maybe virtualenv, and possibly pip | |
| 18:13:44 | clarkb | and rebuild the venv | |
| 18:14:00 | sean-k-mooney | yeah am elodilles melwitt how would you feel about makeing it non voting of killing it on stable/train | |
| 18:14:23 | sean-k-mooney | clarkb: yep used -r to rebuild the venv | |
| 18:14:35 | fungi | right. the chain of problems is that these days tox *always* wants to use the latest available version of virtualenv, virtualenv regularly vendors in its own copy of latest setuptools, and setuptools occasionally drops features used by very old packages | |
| 18:14:46 | sean-k-mooney | i could try messing with the other packages but realisticlly we agree that pinning is not what we want to do | |
| 18:15:06 | melwitt | sean-k-mooney: this sounds like a case for removing the l-c job. also I saw this on stable/ussuri (I guess that means it's also happening on stable/train?) | |
| 18:15:32 | sean-k-mooney | oh i assume train train is broke with a diffent error | |
| 18:15:33 | fungi | almost certainly | |
| 18:15:37 | sean-k-mooney | but still broken | |
| 18:15:47 | dansmith | I'm just jumping in here, but yes, whatever $reason is a good case to drop the l-c job :P | |
| 18:15:53 | fungi | er, train almost certainly would suffer from this if it also has a l-c job i mean | |
| 18:16:48 | sean-k-mooney | fungi: https://github.com/openstack/nova/blob/stable/train/.zuul.yaml#L375 | |
| 18:16:53 | sean-k-mooney | so yes | |
| 18:17:00 | sean-k-mooney | we still have the template | |
| 18:17:06 | melwitt | tomorrow we can ask elodilles what he thinks. I think he's likely done for today | |
| 18:17:17 | sean-k-mooney | we fixed it the last time https://github.com/openstack/nova/commit/b2037fc4e356b55949339a1358c16431a9ab8930 | |
| 18:17:24 | fungi | right, i personally never had high hopes for the l-c experiment (mainly because of the forward flow of time and packages from a decade ago not being able to reasonably predict modern systems), but especially not for stable branches. i would cut my losses on l-c if it were me, but it's not my call | |
| 18:17:46 | dansmith | we had some resistance to removing it in glance, | |
| 18:18:03 | dansmith | but then we got to a point where we couldn't add a new dep version or pip would run off into lala land, which helped seal the deal :) | |
| 18:18:20 | sean-k-mooney | it would resolve the disccion im haveing with stephenfin too | |
| 18:18:39 | sean-k-mooney | where he wanted to have it pinend to the oldest python we support for a release | |
| 18:18:47 | sean-k-mooney | which i disagreed with | |
| 18:19:03 | sean-k-mooney | since that was not realistic to how distors worked | |
| 18:19:37 | fungi | well, basically if you head down that road, using old dependencies is not isolated to the python deps you have, those don't exist in a vaccuum and they make assumptions about contemporary systems as a whole | |
| 18:20:35 | sean-k-mooney | speaking of being done for today i likely should finish up soon | |
| 18:20:48 | fungi | i agree it's not a very good way to try to mimic what stable distros experience, because 1. they pin a lot more about the entire environment than we reasonably can, and 2. they backport fixes to things which wouldn't be reflected with this test methodology anyway | |
| 18:21:00 | sean-k-mooney | since this is now efffectily blocking the backport of a cve fix however i think we likely should move forward with droping it so we can do the release | |
| 18:21:37 | fungi | another alternative is early transition out of maintenance mode for stable/ussuri, but that seems like the most drastic choice | |
| 18:22:02 | sean-k-mooney | fungi: well it was more the version of the packate and the version of python are often uncoupled | |
| 18:22:16 | fungi | yes, that too | |
| 18:22:42 | fungi | anyway, lower constraints jobs are not mandated by the pti, so dropping them doesn't affect the maintenance state for the branch | |
| 18:22:54 | fungi | at least not in any official sense | |
| 19:08:02 | opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: [PoC][yoga] Off-path Networking Backends Support https://review.opendev.org/c/openstack/nova/+/808199 | |
| 19:48:16 | gibi | bauzas: I've update the nova rc1 patch with the prelude hash https://review.opendev.org/c/openstack/releases/+/808706 I think it is good to go now | |
| 20:34:52 | admin1 | hi @all .. how does a nova snapshot work when both glance and nova are backed by ceph ? does it still create a local disk on the hypervisor and uploads ? | |
| 20:35:03 | admin1 | or does everything happen entirely on the ceph side ? | |
| 20:39:10 | admin1 | my snapshot is kind of stuck .. even for a small instance .. and i don't see anything in the hypervisor ( files, or uploads or disk waiting ) so i think its entirely on the ceph side .. just wanted to validate | |
| 20:52:01 | dansmith | admin1: if you have them share a pool, nova just asks ceph to do the snapshot in-place and then nova "tells" glance about it.. no bits fly around like normal | |
| 20:52:35 | dansmith | ("them" meaning glance and nova) | |
| 20:56:12 | opendevreview | melanie witt proposed openstack/nova master: Add section for 'nova-manage placement audit' tool https://review.opendev.org/c/openstack/nova/+/809479 | |
| 20:56:32 | admin1 | nova is on vms pool and glance is on images pool .. so 2 diff pools | |
| 20:58:15 | dansmith | admin1: okay so I think that means it's going to download and then re-upload it, but not totally positive | |
| 20:59:05 | melwitt | admin1: if you have configured your deployment like this https://docs.ceph.com/en/mimic/rbd/rbd-openstack/#configure-openstack-to-use-ceph it should be able to do the fast snapshot | |
| 21:00:09 | admin1 | melwitt , refering to enabling copy-on-write cloning of images ? | |
| 21:00:35 | melwitt | yes | |
| 21:01:17 | admin1 | its not enabled .. | |
| 21:01:22 | admin1 | what does exactly "Note that this exposes the back end location via Glance’s API, so the endpoint with this option enabled should not be publicly accessible." mean ? | |
| 21:01:54 | admin1 | i will try to find more | |
| 21:02:07 | admin1 | melwitt, in absense of this setting, how does the nova snapshot work ? | |
| 21:02:08 | melwitt | another thing that can cause it to not do the CoW is if the image is not in RAW format, but I think glance might automatically convert to RAW now, depending on what release you have | |
| 21:02:19 | admin1 | all images are in raw format | |
| 21:02:31 | admin1 | i converted them before uploading to glance | |
| 21:02:58 | melwitt | that is a warning that the backend url will be shown to users querying glance API so it should not be a url that is publicly accessible (for security reason) | |
| 21:03:45 | melwitt | ok cool. if you don't set up as described in that ceph doc, nova will download the image from glance first and then boot the instance | |
| 21:04:40 | melwitt | if the image is large, it can be slow | |
| 21:08:37 | melwitt | there's some glance setting around chunk size which you might want to adjust to get better throughput, I think it's this? https://docs.openstack.org/glance/latest/configuration/glance_api.html#glance.store.rbd.store.rbd_store_chunk_size | |
| 21:09:03 | melwitt | *settings | |
| 21:18:58 | opendevreview | melanie witt proposed openstack/nova master: Add section for 'nova-manage placement audit' tool https://review.opendev.org/c/openstack/nova/+/809479 | |