Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2023-02-07
16:08:05 bauzas I think the hardest one is at the bottom of the document
16:08:05 dansmith oof yeah
16:08:23 bauzas I had a lovely morning and a half-afternoon spent on that one
16:08:42 bauzas in the context of a soonish feature freeze, more hands are more than welcome
16:09:01 dansmith I wrote the replace_location test, so I can look into that one.. it's a glance test though. I'm sure it's poking some bug in glance, because until I wrote that we didn't really have any tests for that stuff
16:09:08 bauzas because don't expect your patches to be reviewed if most of the cores are having their days spent on fixing CI problems
16:09:08 dansmith but maybe it is resolvable
16:10:06 gibi dansmith: there is https://bugs.launchpad.net/glance/+bug/1999800 and https://bugs.launchpad.net/glance/+bug/2006473 both location tests
16:10:17 bauzas and yeah, I know, debugging a CI failure isn't exactly the best experience you may have of working on an opensource project, but let's be honest and say that's necessary to have an healthy gate
16:10:26 gibi bauzas: +1
16:10:35 dansmith okay the former is the same as bauzas' one
16:10:41 gibi yeah probably
16:11:09 dansmith yeah from the logs, the test is clearly doing something legit and glance is rejecting it but shouldn't
16:11:16 bauzas gibi: I created https://bugs.launchpad.net/nova/+bug/2004641 but it seems duplicate of https://bugs.launchpad.net/glance/+bug/1999800
16:11:27 dansmith might be because it fails to talk to the cirros site occasionally, so maybe we can use an openstack infra url instead
16:11:43 dansmith bauzas: indeed
16:11:52 sean-k-mooney i tought we tried to pull those form provider proxies in ci
16:11:54 gibi bauzas: https://bugs.launchpad.net/tempest/+bug/2004641 and https://bugs.launchpad.net/glance/+bug/2006473 are duplicates but https://bugs.launchpad.net/glance/+bug/1999800 is a separate tc
16:11:55 bauzas I can close my one as duplicate
16:12:06 dansmith bauzas: ++
16:12:20 sean-k-mooney github is more repliable for downlaoding cirrors images by the way then the cirros site
16:12:22 bauzas I just ideally would like to track that bug in our project
16:13:01 dansmith sean-k-mooney: the cirros page just redirects to the github one
16:13:16 sean-k-mooney oh they finally implmetned that
16:13:29 dansmith sean-k-mooney: and we're just using CONF.image.http_image in that test
16:14:03 sean-k-mooney oh this is not the image pulled by https://github.com/openstack/devstack/blob/master/stackrc#L670-L708
16:14:05 dansmith the github URL is crazy long with tons of tokens and other values after the redirects it does
16:14:11 bauzas gibi: ack will mark your https://bugs.launchpad.net/nova/+bug/2004641 as duplicate of mine, then
16:14:12 dansmith sean-k-mooney: this is a tempest test
16:14:33 sean-k-mooney right the one with the larger image
16:14:41 dansmith no
16:14:58 dansmith gibi: it's the same test case, different behavior, but I'm guessing its the sameish problem
16:15:37 bauzas ok, you know what, I'll add mine in the tracking etherpad, and we'll figure out
16:15:49 bauzas the three of them are set against Glance either way
16:15:57 sean-k-mooney im surpised that the tempest test is not using the one we prestage in the vm but ok
16:16:48 sean-k-mooney i was expecting CONF.image.http_image to be file:///opt/devstack/data/cirros...
16:17:05 gibi dansmith: yeah probably similar root cause
16:17:06 dansmith sean-k-mooney: it can't be because that is specifically for testing fetching an image server-side from http
16:17:22 sean-k-mooney ah thanks i was missing that context
16:17:52 sean-k-mooney oh that that in https://bugs.launchpad.net/glance/+bug/2006473 i was only familar with https://bugs.launchpad.net/glance/+bug/1999800
16:18:08 dansmith they're the same test
16:18:31 dansmith sorry, the same test helper
16:18:48 bauzas and probably the rootcause
16:18:52 sean-k-mooney ya so likely the same cause
16:18:53 bauzas same rootcause
16:19:03 bauzas which is a flakey httpservice
16:19:48 bauzas either way, seems we have a path forward with the github image repo then ?
16:21:09 bauzas looks like people are gone
16:21:33 dansmith bauzas: no, it's already using that via redirect
16:21:36 bauzas there is another CI failure I'd like to talk about
16:21:40 sean-k-mooney from the name i would not expect either to depned on downloading an image over http but i have not looked at the detail of the test. i was expecting tempest to upload the image form disk.
16:21:41 dansmith bauzas: I'll take it and work something out
16:21:54 bauzas dansmith: very much appreciated, trust me.
16:22:10 bauzas dansmith: fwiw, the hits number seems low compared to other bits
16:22:29 bauzas bites*
16:22:41 bauzas so, about https://bugs.launchpad.net/nova/+bug/1946339
16:22:42 dansmith yeah, but if we have no other obvious ones to work on, at least I can make some progress on this :)
16:22:50 bauzas dansmith: heh
16:23:15 bauzas so, after a day of co-investigation with my CSI partner gibi on https://bugs.launchpad.net/nova/+bug/1946339
16:23:30 bauzas we identified this may come from a non-poisoned libvirt
16:23:56 bauzas the funny part is that we hit this in a thread, not in the main test
16:24:04 bauzas hence why we missed it before
16:24:21 bauzas I have a question
16:24:40 bauzas do people agree with merging https://review.opendev.org/c/openstack/nova/+/872975 even if it says it's a dnm ?
16:24:54 bauzas (tbc, I can make an update and remove the dnm title)
16:25:04 dansmith we should remove the dnm for sure
16:25:18 sean-k-mooney bauzas: melwitt had a patch to poison importing libvrt that should catuch this by the way
16:25:29 opendevreview Sylvain Bauza proposed openstack/nova master: Add logging for leaking out the non-poisoned libvirt testcase https://review.opendev.org/c/openstack/nova/+/872975
16:25:42 dansmith bauzas: do you know about the thing you can do to add additional test payload report sections?
16:25:46 bauzas dansmith: acked ^
16:25:57 dansmith depending on what you're trying to do, that can be more useful than logging sometimes
16:26:09 bauzas dansmith: nope, hence my sending the bottle to the sea, asking for advices
16:26:15 sean-k-mooney bauzas: can you put a sleep in that busy loop too
16:26:29 dansmith it's not a busy loop is it?
16:26:33 bauzas nope
16:26:44 gibi it is walking a tree up
16:26:55 bauzas we're trying to find an attribute from an eventlet object and if we can't find it, we walk the ascendance
16:26:57 sean-k-mooney it will loop until the test_case_id is not None
16:27:10 gibi it walks along the eventlet.parent link
16:27:12 sean-k-mooney i guess its proably fine
16:27:31 gibi so while it busy it is bounded
16:27:38 sean-k-mooney oh sorry your right it is doing that
16:27:41 sean-k-mooney ok
16:27:42 bauzas dansmith: so, about the payload reporting, you gained my interest
16:27:52 dansmith bauzas: https://github.com/openstack/glance/blob/master/glance/tests/functional/__init__.py#L1129-L1130
16:28:13 dansmith bauzas: that adds another section of the test failure reporting, like "here's the stdout I captured" and "here are the log lines I captured"
16:28:23 bauzas ffff
16:28:37 bauzas dansmith: ++
16:28:43 dansmith helps to separate nova-logging from something specifically to be reported by the test case
16:28:52 dansmith especially if debug logging isn't captured, or is being mocked out, etc
16:29:18 dansmith in glance I found it useful because their functional workers run outside the main process, but also in some cases where I needed to debug failures
16:29:28 dansmith (failures that happen infrequently)
16:29:39 dansmith anyway, just FYI, might be helpful
16:29:48 bauzas it could be
16:29:48 gibi dansmith: ohh that is good to know :)
16:30:22 sean-k-mooney oh addDetail
16:30:31 bauzas dansmith: the problem is that we get an exception from a test which is actually not due by this test but rather by a leaked eventlet thread that blows up at that point in time
16:30:40 sean-k-mooney i have seen that before but never looked into it ya look useful
16:31:18 dansmith gibi: yeah, it's kinda nice :)
16:31:39 bauzas ideally I would like to trace the whole parenting stack that triggered the leaky thread

Earlier   Later