Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2023-02-07
16:05:26 bauzas ++
16:05:31 bauzas artom: no worries
16:05:37 bauzas and thanks
16:05:41 gibi o/
16:06:20 dansmith o/
16:06:28 bauzas ok moving on
16:06:34 bauzas #topic Gate status
16:06:39 bauzas #link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs
16:06:42 bauzas new item
16:06:52 bauzas #link https://etherpad.opendev.org/p/nova-ci-failures Etherpad for tracking CI failures
16:07:14 gibi there is a fairly long list but please add to it if you see failures
16:07:18 gibi that are not on the list
16:07:31 bauzas we have some of them are hit us in some place between the chair and the man
16:08:05 dansmith oof yeah
16:08:05 bauzas I think the hardest one is at the bottom of the document
16:08:23 bauzas I had a lovely morning and a half-afternoon spent on that one
16:08:42 bauzas in the context of a soonish feature freeze, more hands are more than welcome
16:09:01 dansmith I wrote the replace_location test, so I can look into that one.. it's a glance test though. I'm sure it's poking some bug in glance, because until I wrote that we didn't really have any tests for that stuff
16:09:08 dansmith but maybe it is resolvable
16:09:08 bauzas because don't expect your patches to be reviewed if most of the cores are having their days spent on fixing CI problems
16:10:06 gibi dansmith: there is https://bugs.launchpad.net/glance/+bug/1999800 and https://bugs.launchpad.net/glance/+bug/2006473 both location tests
16:10:17 bauzas and yeah, I know, debugging a CI failure isn't exactly the best experience you may have of working on an opensource project, but let's be honest and say that's necessary to have an healthy gate
16:10:26 gibi bauzas: +1
16:10:35 dansmith okay the former is the same as bauzas' one
16:10:41 gibi yeah probably
16:11:09 dansmith yeah from the logs, the test is clearly doing something legit and glance is rejecting it but shouldn't
16:11:16 bauzas gibi: I created https://bugs.launchpad.net/nova/+bug/2004641 but it seems duplicate of https://bugs.launchpad.net/glance/+bug/1999800
16:11:27 dansmith might be because it fails to talk to the cirros site occasionally, so maybe we can use an openstack infra url instead
16:11:43 dansmith bauzas: indeed
16:11:52 sean-k-mooney i tought we tried to pull those form provider proxies in ci
16:11:54 gibi bauzas: https://bugs.launchpad.net/tempest/+bug/2004641 and https://bugs.launchpad.net/glance/+bug/2006473 are duplicates but https://bugs.launchpad.net/glance/+bug/1999800 is a separate tc
16:11:55 bauzas I can close my one as duplicate
16:12:06 dansmith bauzas: ++
16:12:20 sean-k-mooney github is more repliable for downlaoding cirrors images by the way then the cirros site
16:12:22 bauzas I just ideally would like to track that bug in our project
16:13:01 dansmith sean-k-mooney: the cirros page just redirects to the github one
16:13:16 sean-k-mooney oh they finally implmetned that
16:13:29 dansmith sean-k-mooney: and we're just using CONF.image.http_image in that test
16:14:03 sean-k-mooney oh this is not the image pulled by https://github.com/openstack/devstack/blob/master/stackrc#L670-L708
16:14:05 dansmith the github URL is crazy long with tons of tokens and other values after the redirects it does
16:14:11 bauzas gibi: ack will mark your https://bugs.launchpad.net/nova/+bug/2004641 as duplicate of mine, then
16:14:12 dansmith sean-k-mooney: this is a tempest test
16:14:33 sean-k-mooney right the one with the larger image
16:14:41 dansmith no
16:14:58 dansmith gibi: it's the same test case, different behavior, but I'm guessing its the sameish problem
16:15:37 bauzas ok, you know what, I'll add mine in the tracking etherpad, and we'll figure out
16:15:49 bauzas the three of them are set against Glance either way
16:15:57 sean-k-mooney im surpised that the tempest test is not using the one we prestage in the vm but ok
16:16:48 sean-k-mooney i was expecting CONF.image.http_image to be file:///opt/devstack/data/cirros...
16:17:05 gibi dansmith: yeah probably similar root cause
16:17:06 dansmith sean-k-mooney: it can't be because that is specifically for testing fetching an image server-side from http
16:17:22 sean-k-mooney ah thanks i was missing that context
16:17:52 sean-k-mooney oh that that in https://bugs.launchpad.net/glance/+bug/2006473 i was only familar with https://bugs.launchpad.net/glance/+bug/1999800
16:18:08 dansmith they're the same test
16:18:31 dansmith sorry, the same test helper
16:18:48 bauzas and probably the rootcause
16:18:52 sean-k-mooney ya so likely the same cause
16:18:53 bauzas same rootcause
16:19:03 bauzas which is a flakey httpservice
16:19:48 bauzas either way, seems we have a path forward with the github image repo then ?
16:21:09 bauzas looks like people are gone
16:21:33 dansmith bauzas: no, it's already using that via redirect
16:21:36 bauzas there is another CI failure I'd like to talk about
16:21:40 sean-k-mooney from the name i would not expect either to depned on downloading an image over http but i have not looked at the detail of the test. i was expecting tempest to upload the image form disk.
16:21:41 dansmith bauzas: I'll take it and work something out
16:21:54 bauzas dansmith: very much appreciated, trust me.
16:22:10 bauzas dansmith: fwiw, the hits number seems low compared to other bits
16:22:29 bauzas bites*
16:22:41 bauzas so, about https://bugs.launchpad.net/nova/+bug/1946339
16:22:42 dansmith yeah, but if we have no other obvious ones to work on, at least I can make some progress on this :)
16:22:50 bauzas dansmith: heh
16:23:15 bauzas so, after a day of co-investigation with my CSI partner gibi on https://bugs.launchpad.net/nova/+bug/1946339
16:23:30 bauzas we identified this may come from a non-poisoned libvirt
16:23:56 bauzas the funny part is that we hit this in a thread, not in the main test
16:24:04 bauzas hence why we missed it before
16:24:21 bauzas I have a question
16:24:40 bauzas do people agree with merging https://review.opendev.org/c/openstack/nova/+/872975 even if it says it's a dnm ?
16:24:54 bauzas (tbc, I can make an update and remove the dnm title)
16:25:04 dansmith we should remove the dnm for sure
16:25:18 sean-k-mooney bauzas: melwitt had a patch to poison importing libvrt that should catuch this by the way
16:25:29 opendevreview Sylvain Bauza proposed openstack/nova master: Add logging for leaking out the non-poisoned libvirt testcase https://review.opendev.org/c/openstack/nova/+/872975
16:25:42 dansmith bauzas: do you know about the thing you can do to add additional test payload report sections?
16:25:46 bauzas dansmith: acked ^
16:25:57 dansmith depending on what you're trying to do, that can be more useful than logging sometimes
16:26:09 bauzas dansmith: nope, hence my sending the bottle to the sea, asking for advices
16:26:15 sean-k-mooney bauzas: can you put a sleep in that busy loop too
16:26:29 dansmith it's not a busy loop is it?
16:26:33 bauzas nope
16:26:44 gibi it is walking a tree up
16:26:55 bauzas we're trying to find an attribute from an eventlet object and if we can't find it, we walk the ascendance
16:26:57 sean-k-mooney it will loop until the test_case_id is not None
16:27:10 gibi it walks along the eventlet.parent link
16:27:12 sean-k-mooney i guess its proably fine
16:27:31 gibi so while it busy it is bounded
16:27:38 sean-k-mooney oh sorry your right it is doing that
16:27:41 sean-k-mooney ok
16:27:42 bauzas dansmith: so, about the payload reporting, you gained my interest
16:27:52 dansmith bauzas: https://github.com/openstack/glance/blob/master/glance/tests/functional/__init__.py#L1129-L1130
16:28:13 dansmith bauzas: that adds another section of the test failure reporting, like "here's the stdout I captured" and "here are the log lines I captured"
16:28:23 bauzas ffff

Earlier   Later