Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2023-02-07
16:18:31 dansmith sorry, the same test helper
16:18:48 bauzas and probably the rootcause
16:18:52 sean-k-mooney ya so likely the same cause
16:18:53 bauzas same rootcause
16:19:03 bauzas which is a flakey httpservice
16:19:48 bauzas either way, seems we have a path forward with the github image repo then ?
16:21:09 bauzas looks like people are gone
16:21:33 dansmith bauzas: no, it's already using that via redirect
16:21:36 bauzas there is another CI failure I'd like to talk about
16:21:40 sean-k-mooney from the name i would not expect either to depned on downloading an image over http but i have not looked at the detail of the test. i was expecting tempest to upload the image form disk.
16:21:41 dansmith bauzas: I'll take it and work something out
16:21:54 bauzas dansmith: very much appreciated, trust me.
16:22:10 bauzas dansmith: fwiw, the hits number seems low compared to other bits
16:22:29 bauzas bites*
16:22:41 bauzas so, about https://bugs.launchpad.net/nova/+bug/1946339
16:22:42 dansmith yeah, but if we have no other obvious ones to work on, at least I can make some progress on this :)
16:22:50 bauzas dansmith: heh
16:23:15 bauzas so, after a day of co-investigation with my CSI partner gibi on https://bugs.launchpad.net/nova/+bug/1946339
16:23:30 bauzas we identified this may come from a non-poisoned libvirt
16:23:56 bauzas the funny part is that we hit this in a thread, not in the main test
16:24:04 bauzas hence why we missed it before
16:24:21 bauzas I have a question
16:24:40 bauzas do people agree with merging https://review.opendev.org/c/openstack/nova/+/872975 even if it says it's a dnm ?
16:24:54 bauzas (tbc, I can make an update and remove the dnm title)
16:25:04 dansmith we should remove the dnm for sure
16:25:18 sean-k-mooney bauzas: melwitt had a patch to poison importing libvrt that should catuch this by the way
16:25:29 opendevreview Sylvain Bauza proposed openstack/nova master: Add logging for leaking out the non-poisoned libvirt testcase https://review.opendev.org/c/openstack/nova/+/872975
16:25:42 dansmith bauzas: do you know about the thing you can do to add additional test payload report sections?
16:25:46 bauzas dansmith: acked ^
16:25:57 dansmith depending on what you're trying to do, that can be more useful than logging sometimes
16:26:09 bauzas dansmith: nope, hence my sending the bottle to the sea, asking for advices
16:26:15 sean-k-mooney bauzas: can you put a sleep in that busy loop too
16:26:29 dansmith it's not a busy loop is it?
16:26:33 bauzas nope
16:26:44 gibi it is walking a tree up
16:26:55 bauzas we're trying to find an attribute from an eventlet object and if we can't find it, we walk the ascendance
16:26:57 sean-k-mooney it will loop until the test_case_id is not None
16:27:10 gibi it walks along the eventlet.parent link
16:27:12 sean-k-mooney i guess its proably fine
16:27:31 gibi so while it busy it is bounded
16:27:38 sean-k-mooney oh sorry your right it is doing that
16:27:41 sean-k-mooney ok
16:27:42 bauzas dansmith: so, about the payload reporting, you gained my interest
16:27:52 dansmith bauzas: https://github.com/openstack/glance/blob/master/glance/tests/functional/__init__.py#L1129-L1130
16:28:13 dansmith bauzas: that adds another section of the test failure reporting, like "here's the stdout I captured" and "here are the log lines I captured"
16:28:23 bauzas ffff
16:28:37 bauzas dansmith: ++
16:28:43 dansmith helps to separate nova-logging from something specifically to be reported by the test case
16:28:52 dansmith especially if debug logging isn't captured, or is being mocked out, etc
16:29:18 dansmith in glance I found it useful because their functional workers run outside the main process, but also in some cases where I needed to debug failures
16:29:28 dansmith (failures that happen infrequently)
16:29:39 dansmith anyway, just FYI, might be helpful
16:29:48 bauzas it could be
16:29:48 gibi dansmith: ohh that is good to know :)
16:30:22 sean-k-mooney oh addDetail
16:30:31 bauzas dansmith: the problem is that we get an exception from a test which is actually not due by this test but rather by a leaked eventlet thread that blows up at that point in time
16:30:40 sean-k-mooney i have seen that before but never looked into it ya look useful
16:31:18 dansmith gibi: yeah, it's kinda nice :)
16:31:39 bauzas ideally I would like to trace the whole parenting stack that triggered the leaky thread
16:32:32 gibi bauzas: we will hopefuly get the name of the leaky test case and then we can create a local reproduction
16:33:11 bauzas gibi: a stack would have been better but yeah
16:33:20 gibi you have a stack
16:33:25 gibi but it start when the thread starts
16:33:40 bauzas that's the parent stack I want :)
16:33:43 gibi yeah
16:33:45 gibi that is hard
16:35:09 bauzas yup
16:35:25 bauzas anyway, reviews appreciated on https://review.opendev.org/c/openstack/nova/+/872975
16:35:40 bauzas moving on ?
16:35:52 sean-k-mooney sure
16:36:39 bauzas #link https://zuul.openstack.org/builds?project=openstack%2Fnova&project=openstack%2Fplacement&pipeline=periodic-weekly Nova&Placement periodic jobs status
16:36:41 gibi I'm on it
16:36:51 bauzas #info Please look at the gate failures and file a bug report with the gate-failure tag.
16:36:54 bauzas #info Please look at the gate failures and file a bug report with the gate-failure tag.
16:37:01 bauzas #info STOP DOING BLIND RECHECKS aka. 'recheck' https://docs.openstack.org/project-team-guide/testing.html#how-to-handle-test-failures
16:37:12 bauzas done with this topic, phew.
16:37:18 bauzas #topic Release Planning
16:37:23 bauzas #link https://releases.openstack.org/antelope/schedule.html
16:37:28 bauzas #info Antelope-3 is in 1.2 weeks
16:37:39 bauzas I said 1.2 because it will be on Thursday next week
16:37:47 bauzas #link https://blueprints.launchpad.net/nova/antelope All accepted blueprints for 2023.1
16:37:52 bauzas #link https://etherpad.opendev.org/p/nova-antelope-blueprint-status Blueprint status for 2023.1
16:37:59 bauzas feel free to comment it as much as you want ^
16:38:28 bauzas I was originally planning to do a full reviews set by today, but due to the former topic, I abandoned my promise
16:39:00 elodilles a bit related to release: 'Release final os-vif for 2023.1 Antelope' https://review.opendev.org/c/openstack/releases/+/872779
16:39:06 bauzas (yet again saying, don't expect the reviews to magically happen, be present and interact with us)
16:39:19 bauzas elodilles: good catch I forgot to add it the agenda
16:39:23 bauzas Important :
16:40:05 bauzas #info Thursday is the non-client libs feature freeze, which means we can only accept features changes for os-vif, os-traits and os-rc up until Thursday
16:40:25 bauzas later changes will be on hold until next release
16:40:40 elodilles ++
16:41:17 bauzas I haven't looked at os-vif, os-traits and os-resourceclasses master branches, but I think we have open changes on them
16:42:16 bauzas so, if anyone wants some addition to those libraries, I'd recommend them to ping me or anyone else for reviews
16:43:05 bauzas last point
16:43:18 bauzas FeatureFreeze is on next Thursday
16:43:26 bauzas we'll see how the gate goes by that time
16:43:59 bauzas but as for the older releases, the most important for having your series accepted for Antelope is to get a +W before Thursday EOB
16:44:20 bauzas we'll manage the rechecks if needed
16:44:41 bauzas don't freak out by the gate stability, but please continue to ensure your patches are ready for reviews
16:44:55 sean-k-mooney elodilles: i was planning to propose an os-vif release to include rodolfos patches

Earlier   Later