Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2023-02-07
16:56:26 elodilles for ussuri
16:56:42 bauzas elodilles: cool, let's figure that out after the meeting, tomorrow per say
16:56:51 elodilles bauzas: ++
16:56:54 bauzas fwiw, I'm planning to deliver the cve fix down to ussuri
16:57:07 bauzas but not provide any backport to train
16:57:08 elodilles why not train? :)
16:57:16 bauzas due to the oslo.utils versioning
16:57:26 bauzas most of the distros now made the backports
16:57:36 bauzas so it's upstream support
16:57:42 bauzas and Train is on EM
16:57:50 elodilles well, Wallaby is EM
16:57:56 bauzas and ussuri too
16:57:57 elodilles (and Xena soon, too)
16:58:12 bauzas but it was simple to backport the fix down to ussuri
16:58:19 bauzas it was cheap, so we proposed it
16:58:28 elodilles ok :) thanks for that!
16:58:28 bauzas backporting it to train is a totally different story
16:58:54 bauzas it requires some oslo.utils backport too (and then a janga puzzle with dependency management)
16:59:06 elodilles :S
16:59:24 bauzas so, things are said, crystal clear.
16:59:39 elodilles thanks, i see
16:59:41 bauzas elodilles: thanks elodilles for the stable report
16:59:46 elodilles np
16:59:50 bauzas last point for the 20 secs left
16:59:54 bauzas #topic Open discussion
16:59:59 bauzas nothing on the agenda
17:00:04 bauzas so I'll close the meeting
17:00:16 bauzas feel free to add your items for next week
17:00:19 bauzas thnaks all
17:00:21 opendevmeet Log: https://meetings.opendev.org/meetings/nova/2023/nova.2023-02-07-16.01.log.html
17:00:21 opendevmeet Minutes (text): https://meetings.opendev.org/meetings/nova/2023/nova.2023-02-07-16.01.txt
17:00:21 opendevmeet Minutes: https://meetings.opendev.org/meetings/nova/2023/nova.2023-02-07-16.01.html
17:00:21 opendevmeet Meeting ended Tue Feb 7 17:00:21 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
17:00:21 bauzas #endmeeting
17:02:33 elodilles thanks o/
17:02:37 sean-k-mooney elodilles: we can chat after the meeting but is the crytography issue beign adressed in train.
17:04:07 gibi bauzas: I left feedback in https://review.opendev.org/c/openstack/nova/+/872975
17:04:29 elodilles sean-k-mooney[m]: ensure-trust workaround seems to be working for stable/ussuri (i proposed it to master last time accidentally): https://review.opendev.org/c/openstack/grenade/+/872969/
17:05:01 bauzas gibi: appreciated. I can speak French, English and a bit of German and Spanish, but my eventlet is definitely small
17:05:24 elodilles sean-k-mooney[m]: and ykarel in neutron added it to tempest (devstack, actually) as well for train, so hopefully something like that should do the trick in train
17:05:53 gibi it is not a big codebase (eventlet + greenlet) but it is not straitforward either. And greenlet has parts of it implemented as a C extension to python :)
17:06:46 bauzas gibi: elodilles: thanks for the reviews
17:07:00 bauzas to clarify, 1/ this is hard to tell which tests are in cause
17:07:00 elodilles np
17:07:16 sean-k-mooney elodilles: ack i was asking as its currenlty blocking a ceilometer/devstack fix i was working on
17:07:43 sean-k-mooney elodilles: changes to the telemetry tempest plugin check for compatiablity with train
17:07:44 bauzas elodilles: you missed today's conversation but I basically grepped the occurrences of such problem trying to intersect the failing tests and basically there were no suspects
17:07:58 bauzas elodilles: because the failing test is not responsible
17:08:21 bauzas elodilles: this is just an unfortunate test that runs at the same time than the thread is throwing the exception
17:08:23 sean-k-mooney elodilles: i tought you siad orginaly ensure-rust would not work
17:08:28 bauzas bite by the bullet
17:08:57 bauzas gibi: about your comment, not sure I fully understand
17:09:09 bauzas gibi: I can remove the DNM: prefix in the log
17:09:14 elodilles sean-k-mooney: yepp, and i was wrong :/ (pushed it to the wrong branch)
17:09:41 gibi bauzas: I'm not happy to merge this as it will log things that is misleading I would rather keep a DNM patch that we recheck until hits the issue
17:09:45 sean-k-mooney elodilles: ah ok :) then cool i can recheck https://review.opendev.org/c/openstack/telemetry-tempest-plugin/+/872350 after its merged
17:09:54 bauzas gibi: but for case #1 you mentioned, this seems to me OK to have this log
17:10:17 bauzas gibi: I mean
17:10:19 elodilles bauzas: ack. i got your intention (i think): to catch the test with the 'DNM' log whenever we see the failing test
17:10:40 bauzas I'm a developer, I'm writing a functest and I forget to poison libvirt
17:10:53 sean-k-mooney we have a fixture to poision it
17:11:01 sean-k-mooney and we dont install libvirt so the import should also fail
17:11:02 bauzas then I'd see my gate saying -1 if myself I'm not brave enough to run the functest locally
17:11:16 sean-k-mooney in teh fucntional tests the libvirt python package should not be there
17:11:36 sean-k-mooney unless it has been baked into the ci image
17:11:56 clarkb it shouldn't be
17:11:58 bauzas sean-k-mooney: context is, the poison disappears when the thread is executed
17:12:04 sean-k-mooney we intentionally od not list libvirt in test-requirements.txt or requirements.txt
17:12:14 bauzas so the threads gets a None attribute for the import
17:12:40 sean-k-mooney ok so it raises an import error as we expect
17:12:49 bauzas to quote gibi "an existing test case that is properly poisoned and mocked libvirt. But an eventlet is leaked out from the test, the test finished and removed the mock. Then the leaked eventlet wakes up while a later test case runs and because the mock was removed in when the original test finished the leaked eventlet now imports libvirt and hits the poison set up by the current test."
17:12:53 sean-k-mooney that should fail the test bug im guessign we are using spawn_n
17:13:15 gibi bauzas: the code you injected does not help catching such case where the poison was not added
17:13:47 gibi bauzas: and our goal here now is to know what test leaked the eventlet
17:14:04 gibi to be able to reproduce the leak locally and fix it
17:14:06 sean-k-mooney it will just log an error with the orgianl evently id to help identigy the test that was not poisoned
17:14:09 bauzas gibi: yup, that's why I'm trying where to patch
17:14:43 sean-k-mooney gibi: is it a libvirt import in all cases or just some
17:15:04 bauzas sean-k-mooney: no the test that runs when the greenthread wakes up then turns into a failure
17:15:21 bauzas https://4dca9d38a541907e85e1-0253beca39d73a6e7192d5b32ed5edc2.ssl.cf2.rackcdn.com/860282/2/check/nova-tox-functional-py310/466e0d7/testr_results.html
17:15:28 bauzas (one of the many occurences)
17:15:34 bauzas or https://4dca9d38a541907e85e1-0253beca39d73a6e7192d5b32ed5edc2.ssl.cf2.rackcdn.com/860282/2/check/nova-tox-functional-py310/466e0d7/testr_results.html
17:15:46 bauzas or https://4dca9d38a541907e85e1-0253beca39d73a6e7192d5b32ed5edc2.ssl.cf2.rackcdn.com/860282/2/check/nova-tox-functional-py310/466e0d7/testr_results.html
17:16:06 gibi sean-k-mooney: depending on when the leaked eventlet weaks up it either hits the libvirt poison and fails the test, or just logs the stack traces and let the test passes if no poison is in place
17:16:11 bauzas or https://7ffaea22ff93fca2f0ea-bf433abff5f8b85f7f80257b72ac6f67.ssl.cf5.rackcdn.com/869900/7/gate/nova-tox-functional-py38/3b10d8a/testr_results.html (sorry)
17:16:28 sean-k-mooney gibi: ack
17:16:30 bauzas gibi: yup, I found some run
17:16:40 gibi sean-k-mooney: the poision acts like the global state the lets the leaked eventlet manipulate the running test case
17:16:51 sean-k-mooney gibi: an dis it spawn_n in all cases
17:17:17 sean-k-mooney gibi:yes but this is not a reulst of the poision its just highlighign an exisitng issue
17:17:32 gibi sean-k-mooney: yes the poison is good
17:17:42 sean-k-mooney we did have an existing thing like this related to noticiation i think in the past right
17:17:48 gibi sean-k-mooney: yes
17:17:50 sean-k-mooney and we checked the eventlet id
17:18:06 gibi sean-k-mooney: that embeds the testcase id to the eventlet
17:18:12 sean-k-mooney yep
17:18:14 gibi and checks it during the notification code path
17:18:17 sean-k-mooney which is what bauzas is logging now
17:18:18 gibi and that path is fixed
17:18:38 gibi sean-k-mooney: yes, we try to log that now for this poison / live_migration_abort() codepath

Earlier   Later