Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2024-06-20
08:43:08 opendevreview Sylvain Bauza proposed openstack/nova master: WIP : Enable an experimental job for testing mtty https://review.opendev.org/c/openstack/nova/+/922140
08:43:09 opendevreview Sylvain Bauza proposed openstack/nova master: WIP : mtty: add devstack compilation for fedora builds https://review.opendev.org/c/openstack/nova/+/918420
11:03:01 opendevreview Sylvain Bauza proposed openstack/nova master: WIP : Enable an experimental job for testing mtty https://review.opendev.org/c/openstack/nova/+/922140
11:03:02 opendevreview Sylvain Bauza proposed openstack/nova master: WIP : mtty: add devstack compilation for fedora builds https://review.opendev.org/c/openstack/nova/+/918420
13:59:42 frickler bauzas: did you see the "3 RLock(s) were not greened" messages e.g. at the start of https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_42a/922140/13/experimental/nova-mtty/42a251f/controller/logs/screen-n-super-cond.txt ? JayF mentioned something similar yesterday in the ironic channel, not sure if that may actually be relevant?
14:00:14 frickler seems to be related to running on py3.12 at least
14:04:18 JayF I asked my eventlet expert downstream about it. I haven't gotten any response yet. I suspect it maybe that the way they are detecting those locks are getting false positives.
14:25:40 frickler ack, otherwise one may want to insert more debugging output into that warning
14:48:48 JayF There isn't much more to give. Basically it's just inquiring from the garbage collector if it's tracking any rlock objects. It can't really reliably /do/ anything with those objects if detected.
14:49:16 JayF I printed the information on the objects, and nonsurprisingly, it wasn't useful. I suspect it might be possible to get deeper with a debugger + a lower level knowledge of python than I have at hand
16:28:48 opendevreview Rafael Weingartner proposed openstack/nova master: Block VM instance resize to flavor with different aggregates https://review.opendev.org/c/openstack/nova/+/922421
16:51:47 melwitt frickler, JayF: I looked up that log message in eventlet and there's a code comment about how in python < 3.10 the warning won't be logged, so that at least explains why we start seeing it in newer python https://github.com/eventlet/eventlet/blob/ec6c0fffccc129bea468a5badb03034d83f8cc5e/eventlet/patcher.py#L406
16:52:10 JayF melwitt: well, I think there's a behavior change in 3.12
16:52:29 JayF melwitt: because chasing down + eliminating those warnings was part of getting eventlet happier when the changes started
16:53:11 melwitt yeah, the code comment explains the behavior change
16:53:38 JayF it explains behavior change from 3.9 -> 3.10, not >=3.10 changing
16:54:28 JayF I'm going to test on 3.10 with my ironic reproducer and see
16:54:29 melwitt "On older Pythons (< 3.10), gc.get_objects() won't return any RLock instances, so this warning won't get logged on older Pythons." that isn't explaining what happens different in >= 3.10?
16:55:02 JayF I'm saying that at no point have I said *3.10* is the point where behavior changed. I'm seeing it in 3.12 and absolutely chased down + elminated many of these dangling locks in Ironic on =>3.10 at some point
16:55:51 JayF so either: 1) some non-python-version behavior changed to give us ungreened locks (really just locks that were made before we monkey patched) or 2) python version is the culprit
16:56:04 JayF I don't know which yet, or even if it matters in reality, but it's not the kinda thing I wanna ignore
16:56:57 melwitt ok, well, my response was to "seems to be related to running on py3.12" and I found the code comment so I thought that was the relation
16:57:27 JayF Confirmed: in Ironic python 3.10 does not give the warning, in python 3.12 it gives the warning with 2 locks
16:57:37 JayF I'll test with 3.11 to ensure it's 3.11 -> 3.12 behavior change as hypothesized
17:14:49 JayF melwitt: frickler: FWIW I've confirmed with some certainty the behavior change is python-version-related and not anything else. With Ironic as the example; python 3.11: no ungreened locks, python 3.12: 2 ungreened locks. Only library difference is pyayncore and setuptools installed on py3.12 and installing those under py3.11 doesn't cause the issue.
17:15:21 JayF if nova is seeing something similar, I may file an upstream eventlet issue
17:31:30 frickler JayF: well the log I linked to above was from bauzas testing nova on Noble, so running with py3.12 too. I'd need to double check that it doesn't happen on py3.11
17:34:03 frickler hmm, opensearch does find some hits in toy-py311 jobs, too, like https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_ce6/922421/1/check/openstack-tox-py311/ce64a27/job-output.txt
17:34:18 JayF they almost always appear in unit tests
17:34:22 sean-k-mooney frickler: i was able to stack on noble fine yestereday
17:34:22 JayF they are only relevant for real runs
17:34:28 sean-k-mooney i have that runnign in a vm currently
17:34:36 sean-k-mooney but i didn try ot use the deploy openstack
17:38:29 sean-k-mooney frickler: i can boot a vm and do a console log show
17:39:21 frickler JayF: ah, o.k., the number found in those unit tests seems to go from 1 to 4 though from 3.11 to 3.12. also not sure neutron-functional job falls https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_cec/917430/8/check/neutron-fullstack-with-uwsgi/cec5565/controller/logs/services.txt
17:39:37 frickler other than that the only hit is the new noble job
17:41:04 sean-k-mooney im sure there are proably bugs but at first glance things appear to work fin on noble and python 3.12
17:41:04 frickler sean-k-mooney: yes, what triggered me to look at the logs were the tempest failures in the mtty job. but they seem to be related to the actual mtty stuff afaict
17:41:26 JayF My current working hypothesis is it's a reporting issue, not a functional one.
17:41:46 sean-k-mooney frickler: ah ok well i htink there is a bug in sylvains nova patch
17:41:54 sean-k-mooney or somehting related
17:42:19 frickler JayF: yes, at least I have no data that would say otherwise, but would be great if we could be sure of that
17:42:22 sean-k-mooney the tempet test are failign because nova is not reporting an resouce provider inventireos for the mtty devices
17:42:57 sean-k-mooney frickler: the only thing i had to do was pip install -U setuptools in the requiremend and global venv
17:43:14 sean-k-mooney i started on a patch to do that in devstack yesterday but didnt get back to it
17:43:41 sean-k-mooney has anyoen proposed a noble job to the devstack repo yet?
17:43:58 sean-k-mooney if not i can maybe do that this week
17:45:17 frickler ah, yes, I still wanted to add some unit test for edit-constraints to the reqs fix
17:49:25 opendevreview sean mooney proposed openstack/nova master: [WIP] split monkey_patching form import https://review.opendev.org/c/openstack/nova/+/922425
18:08:54 sean-k-mooney frickler: JayF https://review.opendev.org/c/openstack/devstack/+/922427 should work?
18:09:21 JayF nice, I can hook that up to ironic once it shows it's working elsewhere
18:09:25 JayF would be interesting to see what's busted
18:09:44 sean-k-mooney maybe
18:09:59 sean-k-mooney i just got back to the eventlet stuff after 4 weeks or so
18:10:15 sean-k-mooney so right now im trying to load back up the context on what i was doing
18:12:47 sean-k-mooney JayF: ill generally be moving most of my dev envs over to noble or c10s over the next few weeks, both will be using python 3.12
18:13:13 JayF Honestly, it's just a fluke I caught this. 3.12 is my default python, I was testing a local dev thing for Ironic, and just saw it
18:13:48 sean-k-mooney i feel like that how we catch most things
18:14:09 sean-k-mooney one of use runs a ill advicsed setup (i.e. the latest and gratest of whatever) and fine odd bugs
18:15:12 sean-k-mooney in anycase that should give use a full tempest-full run on noble without any other chagnes if it stacks properly
18:15:27 frickler sean-k-mooney: while just testing you may also want to drop all the other jobs in the check pipeline
18:15:29 JayF I was about to make some snarky comment about being offended you find my gentoo laptop ill advised ... then I realized you're *probably* actually right :D
18:15:31 sean-k-mooney whihc shoudl be antoher datapoint to if this is an actul problem or jsut a logging issue
18:15:47 sean-k-mooney frickler: yep i remmebered that after i pushed
18:15:55 sean-k-mooney frickler: ill do it if i respin
18:16:04 frickler ack
18:19:11 sean-k-mooney pbr still depends on distutils by the way
18:19:12 sean-k-mooney https://opendev.org/openstack/pbr/src/branch/master/pbr/core.py#L51
18:20:34 sean-k-mooney so until we drop that or we start adding seutuptools > whatever
18:20:49 sean-k-mooney to the relevent project that depend on distutils
18:20:57 sean-k-mooney we will need to keep the workarouncx in devstack
18:33:20 opendevreview Rafael Weingartner proposed openstack/nova master: Block VM instance resize to flavor with different aggregates https://review.opendev.org/c/openstack/nova/+/922421
18:35:37 opendevreview Rafael Weingartner proposed openstack/nova master: Block VM instance resize to flavor with different aggregates https://review.opendev.org/c/openstack/nova/+/922421
18:38:45 clarkb sean-k-mooney: PBR also depends on setuptools (which caries distutils these days) so you're depending on both really
18:39:11 sean-k-mooney clarkb: i didnt see the requrieemtns file for pbr in the repo
18:39:20 sean-k-mooney did i miss it or is that depenciy specified elsewher
18:39:47 clarkb sean-k-mooney: its implicit as pbr is a setuptools setup_requires hook or whatever its called
18:39:50 sean-k-mooney there are no deps in https://github.com/openstack/pbr/blob/d03d617c09e7ba8ddf62d1e53d71685cd708e2da/setup.cfg or https://github.com/openstack/pbr/blob/d03d617c09e7ba8ddf62d1e53d71685cd708e2da/setup.py
18:40:04 clarkb you can't list it as a requirment because of chicke and egg issues. The pyproject.toml documentation does explicitly list setuptools though
18:40:15 sean-k-mooney clarkb: ok but without an explict depency we can specify a min version
18:40:36 sean-k-mooney right here https://github.com/openstack/pbr/blob/d03d617c09e7ba8ddf62d1e53d71685cd708e2da/pyproject.toml.future#L7
18:40:44 clarkb pbr is expected to run with a large range of setuptools versions
18:40:54 clarkb there probably is some min version but its going to be old
18:41:29 sean-k-mooney we basically need to say if python 3.12 reuqire setuptools >= the one with distutils
18:41:43 sean-k-mooney which we would normaly do in a requriemtes.txt
18:41:53 sean-k-mooney with the version spericifc marker
18:42:31 clarkb right but you can't because of the chicken and egg problem. The only solution to that is to use pyproject.toml instead
18:42:39 clarkb but no projects are doing that yet as far as I know
18:42:39 sean-k-mooney i.e. setuptools>=70;python_version='3.12'
18:43:09 sean-k-mooney clarkb: well i started adding pyproject.toml support
18:43:25 sean-k-mooney but it need the wsgi_srcipts to be replaced
18:43:32 sean-k-mooney so that work kind of stalled
18:43:36 clarkb basically the dependency may exist but you can't encode that in a requirements file in a way that works/makes sense. You can encode it in project.toml build requires though
18:44:27 sean-k-mooney well for now we can just have devstack install it
18:44:56 sean-k-mooney for porject other then pbr if they have a dep on distutils they can sepciry the requireemnt in the normal way
18:45:31 clarkb yes, the workaround there is appropriate. I just wanted to call out that it isn't just for distutils. You also need setuptools proper which isn't included by default in modern python either
18:45:55 sean-k-mooney ack ya thats also true
18:47:29 sean-k-mooney looking at codesearch/hound there are more project then i would like importing distutils
18:48:02 sean-k-mooney .... inlcuing oslo.versionedobjects
18:48:38 sean-k-mooney we proably should look to remove that dep and replace it with somethign else

Earlier   Later