| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-02-07 | |||
| 16:55:04 | bauzas | huzzah | |
| 16:55:05 | elodilles | stable/wallaby gate is passing (failing openstacksdk-functional-devstack job was removed from wallaby) | |
| 16:55:13 | bauzas | thanks gmann for the hard work on stable/wallaby | |
| 16:55:18 | elodilles | yepp | |
| 16:55:19 | elodilles | ++ | |
| 16:55:21 | elodilles | #info stable/victoria gate is affected by the same failing openstacksdk-functional-devstack job | |
| 16:55:30 | elodilles | #info ussuri and train gates are broken broken ('Could not build wheels for bcrypt, cryptography' errors) | |
| 16:55:38 | elodilles | #info stable branch status / gate failures tracking etherpad: https://etherpad.opendev.org/p/nova-stable-branch-ci | |
| 16:55:41 | elodilles | EOM | |
| 16:55:54 | bauzas | elodilles: I'll pay attention to the ussuri branch with the CVE backport | |
| 16:56:01 | elodilles | bauzas: ack | |
| 16:56:10 | bauzas | ... when I have time :) | |
| 16:56:18 | elodilles | bauzas: i have a proposed potential workaround | |
| 16:56:26 | elodilles | for ussuri | |
| 16:56:42 | bauzas | elodilles: cool, let's figure that out after the meeting, tomorrow per say | |
| 16:56:51 | elodilles | bauzas: ++ | |
| 16:56:54 | bauzas | fwiw, I'm planning to deliver the cve fix down to ussuri | |
| 16:57:07 | bauzas | but not provide any backport to train | |
| 16:57:08 | elodilles | why not train? :) | |
| 16:57:16 | bauzas | due to the oslo.utils versioning | |
| 16:57:26 | bauzas | most of the distros now made the backports | |
| 16:57:36 | bauzas | so it's upstream support | |
| 16:57:42 | bauzas | and Train is on EM | |
| 16:57:50 | elodilles | well, Wallaby is EM | |
| 16:57:56 | bauzas | and ussuri too | |
| 16:57:57 | elodilles | (and Xena soon, too) | |
| 16:58:12 | bauzas | but it was simple to backport the fix down to ussuri | |
| 16:58:19 | bauzas | it was cheap, so we proposed it | |
| 16:58:28 | elodilles | ok :) thanks for that! | |
| 16:58:28 | bauzas | backporting it to train is a totally different story | |
| 16:58:54 | bauzas | it requires some oslo.utils backport too (and then a janga puzzle with dependency management) | |
| 16:59:06 | elodilles | :S | |
| 16:59:24 | bauzas | so, things are said, crystal clear. | |
| 16:59:39 | elodilles | thanks, i see | |
| 16:59:41 | bauzas | elodilles: thanks elodilles for the stable report | |
| 16:59:46 | elodilles | np | |
| 16:59:50 | bauzas | last point for the 20 secs left | |
| 16:59:54 | bauzas | #topic Open discussion | |
| 16:59:59 | bauzas | nothing on the agenda | |
| 17:00:04 | bauzas | so I'll close the meeting | |
| 17:00:16 | bauzas | feel free to add your items for next week | |
| 17:00:19 | bauzas | thnaks all | |
| 17:00:21 | opendevmeet | Log: https://meetings.opendev.org/meetings/nova/2023/nova.2023-02-07-16.01.log.html | |
| 17:00:21 | opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/nova/2023/nova.2023-02-07-16.01.txt | |
| 17:00:21 | opendevmeet | Minutes: https://meetings.opendev.org/meetings/nova/2023/nova.2023-02-07-16.01.html | |
| 17:00:21 | opendevmeet | Meeting ended Tue Feb 7 17:00:21 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | |
| 17:00:21 | bauzas | #endmeeting | |
| 17:02:33 | elodilles | thanks o/ | |
| 17:02:37 | sean-k-mooney | elodilles: we can chat after the meeting but is the crytography issue beign adressed in train. | |
| 17:04:07 | gibi | bauzas: I left feedback in https://review.opendev.org/c/openstack/nova/+/872975 | |
| 17:04:29 | elodilles | sean-k-mooney[m]: ensure-trust workaround seems to be working for stable/ussuri (i proposed it to master last time accidentally): https://review.opendev.org/c/openstack/grenade/+/872969/ | |
| 17:05:01 | bauzas | gibi: appreciated. I can speak French, English and a bit of German and Spanish, but my eventlet is definitely small | |
| 17:05:24 | elodilles | sean-k-mooney[m]: and ykarel in neutron added it to tempest (devstack, actually) as well for train, so hopefully something like that should do the trick in train | |
| 17:05:53 | gibi | it is not a big codebase (eventlet + greenlet) but it is not straitforward either. And greenlet has parts of it implemented as a C extension to python :) | |
| 17:06:46 | bauzas | gibi: elodilles: thanks for the reviews | |
| 17:07:00 | bauzas | to clarify, 1/ this is hard to tell which tests are in cause | |
| 17:07:00 | elodilles | np | |
| 17:07:16 | sean-k-mooney | elodilles: ack i was asking as its currenlty blocking a ceilometer/devstack fix i was working on | |
| 17:07:43 | sean-k-mooney | elodilles: changes to the telemetry tempest plugin check for compatiablity with train | |
| 17:07:44 | bauzas | elodilles: you missed today's conversation but I basically grepped the occurrences of such problem trying to intersect the failing tests and basically there were no suspects | |
| 17:07:58 | bauzas | elodilles: because the failing test is not responsible | |
| 17:08:21 | bauzas | elodilles: this is just an unfortunate test that runs at the same time than the thread is throwing the exception | |
| 17:08:23 | sean-k-mooney | elodilles: i tought you siad orginaly ensure-rust would not work | |
| 17:08:28 | bauzas | bite by the bullet | |
| 17:08:57 | bauzas | gibi: about your comment, not sure I fully understand | |
| 17:09:09 | bauzas | gibi: I can remove the DNM: prefix in the log | |
| 17:09:14 | elodilles | sean-k-mooney: yepp, and i was wrong :/ (pushed it to the wrong branch) | |
| 17:09:41 | gibi | bauzas: I'm not happy to merge this as it will log things that is misleading I would rather keep a DNM patch that we recheck until hits the issue | |
| 17:09:45 | sean-k-mooney | elodilles: ah ok :) then cool i can recheck https://review.opendev.org/c/openstack/telemetry-tempest-plugin/+/872350 after its merged | |
| 17:09:54 | bauzas | gibi: but for case #1 you mentioned, this seems to me OK to have this log | |
| 17:10:17 | bauzas | gibi: I mean | |
| 17:10:19 | elodilles | bauzas: ack. i got your intention (i think): to catch the test with the 'DNM' log whenever we see the failing test | |
| 17:10:40 | bauzas | I'm a developer, I'm writing a functest and I forget to poison libvirt | |
| 17:10:53 | sean-k-mooney | we have a fixture to poision it | |
| 17:11:01 | sean-k-mooney | and we dont install libvirt so the import should also fail | |
| 17:11:02 | bauzas | then I'd see my gate saying -1 if myself I'm not brave enough to run the functest locally | |
| 17:11:16 | sean-k-mooney | in teh fucntional tests the libvirt python package should not be there | |
| 17:11:36 | sean-k-mooney | unless it has been baked into the ci image | |
| 17:11:56 | clarkb | it shouldn't be | |
| 17:11:58 | bauzas | sean-k-mooney: context is, the poison disappears when the thread is executed | |
| 17:12:04 | sean-k-mooney | we intentionally od not list libvirt in test-requirements.txt or requirements.txt | |
| 17:12:14 | bauzas | so the threads gets a None attribute for the import | |
| 17:12:40 | sean-k-mooney | ok so it raises an import error as we expect | |
| 17:12:49 | bauzas | to quote gibi "an existing test case that is properly poisoned and mocked libvirt. But an eventlet is leaked out from the test, the test finished and removed the mock. Then the leaked eventlet wakes up while a later test case runs and because the mock was removed in when the original test finished the leaked eventlet now imports libvirt and hits the poison set up by the current test." | |
| 17:12:53 | sean-k-mooney | that should fail the test bug im guessign we are using spawn_n | |
| 17:13:15 | gibi | bauzas: the code you injected does not help catching such case where the poison was not added | |
| 17:13:47 | gibi | bauzas: and our goal here now is to know what test leaked the eventlet | |
| 17:14:04 | gibi | to be able to reproduce the leak locally and fix it | |
| 17:14:06 | sean-k-mooney | it will just log an error with the orgianl evently id to help identigy the test that was not poisoned | |
| 17:14:09 | bauzas | gibi: yup, that's why I'm trying where to patch | |
| 17:14:43 | sean-k-mooney | gibi: is it a libvirt import in all cases or just some | |
| 17:15:04 | bauzas | sean-k-mooney: no the test that runs when the greenthread wakes up then turns into a failure | |
| 17:15:21 | bauzas | https://4dca9d38a541907e85e1-0253beca39d73a6e7192d5b32ed5edc2.ssl.cf2.rackcdn.com/860282/2/check/nova-tox-functional-py310/466e0d7/testr_results.html | |
| 17:15:28 | bauzas | (one of the many occurences) | |
| 17:15:34 | bauzas | or https://4dca9d38a541907e85e1-0253beca39d73a6e7192d5b32ed5edc2.ssl.cf2.rackcdn.com/860282/2/check/nova-tox-functional-py310/466e0d7/testr_results.html | |
| 17:15:46 | bauzas | or https://4dca9d38a541907e85e1-0253beca39d73a6e7192d5b32ed5edc2.ssl.cf2.rackcdn.com/860282/2/check/nova-tox-functional-py310/466e0d7/testr_results.html | |
| 17:16:06 | gibi | sean-k-mooney: depending on when the leaked eventlet weaks up it either hits the libvirt poison and fails the test, or just logs the stack traces and let the test passes if no poison is in place | |
| 17:16:11 | bauzas | or https://7ffaea22ff93fca2f0ea-bf433abff5f8b85f7f80257b72ac6f67.ssl.cf5.rackcdn.com/869900/7/gate/nova-tox-functional-py38/3b10d8a/testr_results.html (sorry) | |
| 17:16:28 | sean-k-mooney | gibi: ack | |
| 17:16:30 | bauzas | gibi: yup, I found some run | |