| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-09-16 | |||
| 16:21:13 | fungi | trying to make sure this ossa update doesn't fall through the cracks | |
| 16:23:39 | melwitt | ++ thanks fungi | |
| 16:32:34 | melwitt | sean-k-mooney: I've got it | |
| 16:32:57 | opendevreview | Merged openstack/placement master: Update master for stable/xena https://review.opendev.org/c/openstack/placement/+/809365 | |
| 16:34:40 | opendevreview | melanie witt proposed openstack/nova stable/train: address open redirect with 3 forward slashes https://review.opendev.org/c/openstack/nova/+/806629 | |
| 16:34:46 | clarkb | Hello, I've got a cloud (the inmotion cloud you might see some of your CI jobs running on) that nodepool is failing to boot instances on. | |
| 16:35:11 | clarkb | the compute log says things like Instance f98ce366-90b1-43ba-8513-bf2ea559c931 has allocations against this compute host but is not found in the database. | |
| 16:35:27 | clarkb | and the conductor log says things like nova.exception_Remote.NoValidHost_Remote: No valid host was found. | |
| 16:35:48 | clarkb | `openstackclient limits show --absolute` shows that we aren't using any quota. | |
| 16:36:07 | clarkb | But it appears that nova is essentially saying the compute hosts are full up and it can't schedule more work? | |
| 16:36:37 | melwitt | clarkb: orphaned allocations in placement :( are making the hosts appear to be using resources when they shouldn't be | |
| 16:37:09 | clarkb | Is there a way to list those without digging into a database? osc server list is empty too | |
| 16:37:43 | melwitt | this is the doc for dealing with that https://docs.openstack.org/nova/latest/admin/troubleshooting/orphaned-allocations.html probably just need to run 'nova-manage placement heal_allocations' tool | |
| 16:37:49 | clarkb | thank you | |
| 16:39:44 | fungi | remember to lay your hands on the server when you chant "heal_allocations" too | |
| 16:40:22 | fungi | it doesn't really make the command work any better, but it looks cool if anyone happens to be walking by | |
| 16:40:37 | melwitt | 😂 | |
| 16:42:56 | melwitt | this is just another reminder to prioritize the healing service (that would run heal_allocations periodically on its own, among other things) that's been in the backlog that I haven't had time to work on | |
| 16:58:26 | opendevreview | Merged openstack/nova master: Add the Xena prelude section https://review.opendev.org/c/openstack/nova/+/807786 | |
| 17:40:34 | melwitt | elodilles: oh no, l-c job failure again https://review.opendev.org/c/openstack/nova/+/806628 | |
| 17:47:33 | sean-k-mooney | The user requested decorator>=3.4.0 | |
| 17:47:35 | sean-k-mooney | The user requested (constraint) decorator==3.4.0 | |
| 17:47:47 | sean-k-mooney | .... | |
| 17:47:54 | sean-k-mooney | that does not feel like it should be a conflict | |
| 17:48:25 | sean-k-mooney | since 3.4.0==3.4.0 is true and 3.4.0>=3.4.0 is also true | |
| 17:48:37 | melwitt | yeah | |
| 17:48:40 | melwitt | there's also this error "error in decorator setup command: use_2to3 is invalid." | |
| 17:48:41 | clarkb | I think fungi said those issues may come up when pypi serves us stale indexes | |
| 17:48:44 | melwitt | I don't know what that means | |
| 17:48:53 | sean-k-mooney | oh error in decorator setup command: use_2to3 is invalid. | |
| 17:48:53 | clarkb | the use_2to3 error is due to a new setuptools I think they removed that flag | |
| 17:49:03 | sean-k-mooney | so this is not deps related | |
| 17:49:09 | sean-k-mooney | ya | |
| 17:49:17 | melwitt | https://zuul.opendev.org/t/openstack/build/4290861d5a464d099ad38165c99b647d/log/job-output.txt#792 | |
| 17:49:23 | sean-k-mooney | the deps are fine but its unhappy with setuptools | |
| 17:49:26 | clarkb | basically python software with modern pypa tools are expected to be python3 and not converted | |
| 17:50:01 | sean-k-mooney | clarkb: is there a compat flag we can enable | |
| 17:50:07 | sean-k-mooney | or jus tmove this to python 3 | |
| 17:50:47 | melwitt | aside: the jump to link doesn't seem to be working for me lately when I link to a line number in a zuul output | |
| 17:51:08 | sean-k-mooney | ya that only worked for me if the file is small | |
| 17:51:12 | sean-k-mooney | it highlights it | |
| 17:51:21 | sean-k-mooney | but does not move to it if its not loaded quick enough | |
| 17:51:39 | melwitt | yeah. hrm. I wonder if something changed. the above ^ link doesn't jump me to the line and it loads really fast | |
| 17:52:27 | clarkb | sean-k-mooney: I think pypa isn't interested in having compat flags. Updating or replacing deps is probably necessary | |
| 17:52:56 | sean-k-mooney | melwitt: if i open it https://zuul.opendev.org/t/openstack/build/4290861d5a464d099ad38165c99b647d/log/job-output.txt#792 then change it to https://zuul.opendev.org/t/openstack/build/4290861d5a464d099ad38165c99b647d/log/job-output.txt#791 | |
| 17:52:59 | sean-k-mooney | it works fine | |
| 17:53:51 | sean-k-mooney | clarkb: ya or droping lower constraints. i just checked and its on py3 | |
| 17:54:04 | sean-k-mooney | we have https://github.com/openstack/nova/blob/stable/train/tox.ini#L10 | |
| 17:54:20 | sean-k-mooney | and we do not override it for lower constraints | |
| 17:55:13 | sean-k-mooney | clarkb: the other option we have woudl be to downgrade setuptools | |
| 17:55:27 | sean-k-mooney | pin it in the tox env to one that works | |
| 17:55:48 | sean-k-mooney | perhaps usign requires https://tox.readthedocs.io/en/latest/config.html#conf-requires | |
| 17:56:25 | sean-k-mooney | that would be a stable only change i guess if we did that. | |
| 17:56:35 | sean-k-mooney | clarkb: do you know if this is affecting anyone else | |
| 17:57:00 | sean-k-mooney | i know a lot of project just deleted there old lower constriants jobs | |
| 17:58:52 | clarkb | the governance repo had problems with pydot2 which hasn't been maintained for years | |
| 17:59:21 | clarkb | I don't think we want ot downgrade setuptools if we can avoid it. We can't really control what version of setuptools others use effectively and wide compatibility is desireable | |
| 17:59:39 | clarkb | (note with the whole pyproject.toml stuff you do get a bit more control but openstack hasn'tdone any of that) | |
| 18:00:32 | sean-k-mooney | ya its more we have 3 options delete the job, update the dep or hack around to make it work | |
| 18:00:47 | sean-k-mooney | pinnign setup tools on an em branch is just that | |
| 18:00:48 | clarkb | can you bump the dep version up such that it works? | |
| 18:00:53 | sean-k-mooney | a hackaround to make it work | |
| 18:00:58 | clarkb | that is all lower constraints is supposed to track iirc. The oldest version that works | |
| 18:01:18 | sean-k-mooney | clarkb: well we are technialy not allow to bump min verison on stable right | |
| 18:01:28 | sean-k-mooney | but pratically speaking yes we could | |
| 18:01:36 | sean-k-mooney | and have to keeep it working already | |
| 18:02:19 | clarkb | ya for stable maybe removing the job entirely makes sense | |
| 18:02:52 | fungi | note the most recent setuptools release was a week ago, not sure if that lines up with when these failures started | |
| 18:03:22 | fungi | however tox updated today | |
| 18:03:32 | fungi | so it may have started vendoring a newer setuptools | |
| 18:03:47 | fungi | this is one of the reasons "pinning" setuptools is complicated | |
| 18:05:05 | sean-k-mooney | ya | |
| 18:05:13 | sean-k-mooney | looking at the releases https://pypi.org/project/decorator/#history | |
| 18:05:20 | sean-k-mooney | we are using 3.4.0 now | |
| 18:05:30 | sean-k-mooney | from october 2012 | |
| 18:05:35 | fungi | we, actually it's virtualenv which vendors setuptools, and that was also released today | |
| 18:05:39 | sean-k-mooney | the next release 3.4.1 is 2015 | |
| 18:06:16 | fungi | v20.8.0 (2021-09-16): upgrade embedded setuptools to 58.0.4 from 57.4.0 and pip to 21.2.4 from 21.2.3 | |
| 18:06:45 | fungi | so my guess is this is setuptools 58.x.x behavior brought in by tox 20.8 | |
| 18:07:30 | fungi | you could try manually downgrading setuptools<58 in the created env to find out, but i don't recommend that as a fix | |
| 18:09:14 | sean-k-mooney | ya im just trying to repodcue it localy now | |
| 18:09:22 | fungi | v58.0.0 Breaking Changes: Removed support for 2to3 during builds. Projects should port to a unified codebase or pin to an older version of Setuptools using PEP 518 build-requires. | |
| 18:09:31 | sean-k-mooney | GLOB sdist-make: /home/sean/repos/nova/setup.py | |
| 18:09:36 | sean-k-mooney | that new to me ^ | |
| 18:09:54 | sean-k-mooney | i have not seen a GLOB sdist-make line before | |
| 18:10:06 | sean-k-mooney | The conflict is caused by: | |
| 18:10:07 | sean-k-mooney | jinja2 2.10 depends on MarkupSafe>=0.23 | |
| 18:10:09 | sean-k-mooney | The user requested (constraint) markupsafe==1.0 | |
| 18:10:16 | sean-k-mooney | so we have other conflict too aparently | |
| 18:10:22 | fungi | and earlier i said tox 20.8 but meant virtualenv 20.8 | |
| 18:11:08 | sean-k-mooney | ImportError: cannot import name 'Feature' from 'setuptools' | |
| 18:11:15 | fungi | sean-k-mooney: yes, this is one of the reasons i contended calculating and maintaining a lower constraints list would be unworkable long term | |
| 18:12:36 | sean-k-mooney | the import error was from markupsafe with tox 3.20.1 | |
| 18:12:48 | sean-k-mooney | althoguh ok i ghet the same on ewith latest tox | |
| 18:12:59 | sean-k-mooney | 3.24.4 | |
| 18:13:24 | fungi | yeah, downgrading tox probably won't help. you more likely need to downgrade setuptools, maybe virtualenv, and possibly pip | |
| 18:13:44 | clarkb | and rebuild the venv | |
| 18:14:00 | sean-k-mooney | yeah am elodilles melwitt how would you feel about makeing it non voting of killing it on stable/train | |
| 18:14:23 | sean-k-mooney | clarkb: yep used -r to rebuild the venv | |
| 18:14:35 | fungi | right. the chain of problems is that these days tox *always* wants to use the latest available version of virtualenv, virtualenv regularly vendors in its own copy of latest setuptools, and setuptools occasionally drops features used by very old packages | |