Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2024-07-03
15:02:26 fungi you're welcome
16:19:53 frickler another gate failure, timeout on tempest-integrated-compute-enforce-scope-new-defaults, luckily only on the 3rd patch and #4 was out already anyway with sdk failure. will re-enqueue once zuul finishes on them https://zuul.opendev.org/t/openstack/build/4c80df617e06413e9635e0968947809f
16:23:20 frickler hmm, that did reset the whole bunch of 15 other bunches behind it, though. so CI will be fully loaded for quite some more time
16:23:25 fungi tempest-integrated-compute-enforce-scope-new-defaults is what i was seeing repeated timeouts on yesterday as well
16:23:36 opendevreview Merged openstack/nova master: Reject qcow files with data-file attributes https://review.opendev.org/c/openstack/nova/+/923255
16:23:59 frickler \o/ 1 done, 20 or so to go? ;)
16:24:50 fungi 20 or so if you're just talking about nova. across the three affected projects and four maintained branches for the full ossa it's about 50 changes
16:27:20 fungi maybe more now
16:27:44 fungi i think we linked 48 in the advisory, but that's not including the additional testing fixes that ended up going in yesterday
16:37:35 frickler yes, we should seriously reconsider whether doing more than a single patch per branch is really a good idea before next time
16:38:02 frickler unless of course we fix all gate stability and capacity issues until then :-D
16:38:46 clarkb its probably also worth considering reducing the bar set by the gate (both prior to major time sensitive changes showing up and generally)
16:39:56 fungi well, usually we try very, very, VERY hard not to do feature development for emergency security fixes and make them as concise and minimal as possible. this issue was an exception because of discovering that we basically needed to rip out any reliance on qemu-img being safe to rely on for security-sensitive codepaths
16:40:04 clarkb once upon a time removing jobs that were unreliable was a relatively common expectation (this is part of the reason we have the silent and experimental pipelines (though maybe silent is gone now))
16:40:11 frickler well that's more questionable IMO. the high bar did at least notice a regression in cinder that went unnoticed before
16:41:08 clarkb yes it is a tradeoff. The ideal is that if you've got a test job because you believe it is valuable to run that it will also be maintained to keep it reliable
16:41:44 clarkb but unfortunately doing so is often difficult and openstack has basically never managed to do so consistently over time and constantly flip flops between sad and more happy states
16:42:03 fungi the original fix to just address the problem initially described in the bug report, and for which the ossa text focuses on, was going to be a single concise patch for each service
16:42:42 fungi it was complicated by the discovery that the method we'd been relying on already to do those things isn't considered safe by the maintainers of the tool we were using for it
16:43:36 fungi and when we started down that road, the qemu maintainers had made it clear to us that they had no intention of fixing it to be safe in any immediate timeframe
16:44:18 fungi (turns out they did later decide to release a security fix for qemu anyway, but the approach we ended up with should be a lot more future-proof)
17:18:41 opendevreview Merged openstack/nova master: Check images with format_inspector for safety https://review.opendev.org/c/openstack/nova/+/923256
18:02:11 dansmith sean-k-mooney: melwitt: gibi: bauzas: Tempest tests for glance import are going to fail in our ceph job when glance patches merge and until a tempest fix is merged, which is currently stuck waiting on tempest-core
18:02:48 dansmith we can either mark that job n-v (which is easy) or try to strategically skip those tests, or we can just wait for tempest when gmann shows up and sees
18:02:54 dansmith what's your preference?
18:04:20 dansmith okay gmann to the rescue
18:04:32 dansmith fungi: abhishek_ I think gmann is +Wing now
18:04:40 gmann just logged in, checking
18:05:07 gmann dansmith: abhishek_: this one also right ? https://review.opendev.org/c/openstack/tempest/+/923357
18:05:07 dansmith fungi: so I would prioritize *those* and drop abhi's patch I think
18:05:15 fungi okay...
18:05:16 dansmith yes
18:05:18 fungi just a sec
18:05:57 abhishek_ and this one as well, https://review.opendev.org/c/openstack/tempest/+/923352
18:07:36 fungi 923357,3 and 923352,1 both for tempest
18:07:46 abhishek_ yes
18:08:08 dansmith I hope fungi is doing blowing smoke from his finger guns after each of these like I imagine in my head
18:08:22 fungi if only i were that cool
18:08:57 abhishek_ you are
18:09:04 dansmith +2
18:09:14 gmann abhishek_: +w another one also, one comment for releasenotes but can be done in follow up https://review.opendev.org/c/openstack/tempest/+/923357/3/tempest/config.py#648
18:09:15 fungi and both have approvals now, so doing
18:09:36 gmann ++
18:09:41 dansmith thank you gmann
18:10:28 gmann np! sorry for login late. my toddler took a lot of time for his breakfast today :)
18:10:50 fungi okay all done and dequeued glance's 923433,2 at the same time
18:11:08 abhishek_ gmann: ack, thank you!
18:11:20 dansmith fungi: do it do it!
18:11:21 abhishek_ fungi ++
18:11:48 dansmith YASS
18:12:05 abhishek_ :D
18:12:07 fungi count on me for all your fanservice needs
18:12:14 dansmith heh
18:19:11 sean-k-mooney dansmith: looks like gmann +w'd it
18:19:30 sean-k-mooney so i guess we want to wait for https://review.opendev.org/c/openstack/tempest/+/923352 to merge
18:19:36 dansmith sean-k-mooney: yes, it just unfolded as soon as I brought it up :D
18:19:59 dansmith sorry for the noise
20:36:04 fungi and both merged about 35 minutes ago now
20:51:13 dansmith ++
21:23:12 fungi looking into options to less-disruptively speed merging for the remaining security patches... are there any other changes not explicitly mentioned in the ossa which i should avoid sending to the back of the line?
21:25:02 fungi going to start fiddling with some surgical queue reordering in about an hour once i get my dinner down
21:27:24 melwitt fungi: https://review.opendev.org/c/openstack/nova/+/923344 is a backport that some glance CVE patches depend on
21:27:54 melwitt (that I don't see mentioned in the ossa)
21:56:30 sean-k-mooney melwitt: that not part of the cve fix really
21:56:38 sean-k-mooney melwitt: thats just a ci change
21:57:29 melwitt sean-k-mooney: I know, but it's my understanding that it's required for some of the glance CVE backports. so it probably should be avoided to send to the back of the line
22:02:13 clarkb the entire gate just reset I think
22:02:20 clarkb so now is a good time to reorder cc fungi
22:02:46 sean-k-mooney melwitt: ah good point
22:11:54 fungi just got back
22:12:52 fungi assembling the list now
22:23:09 fungi mmm, this is going to be tricky because i have to pass in both the project name and the change,revision for each change i evict and re-add, so a simple loop won't suffice
22:23:12 dansmith yeah, that one is required for the glance on the same branch
22:25:21 dansmith poor glance hasn't merged anything yet, but the head change is in gate right now
22:26:19 fungi should https://review.opendev.org/c/openstack/glance/+/923433 be preserved too? it switches a job to non-voting
22:26:39 dansmith fungi: this got kicked out of gate but should be able to right back in: https://review.opendev.org/c/openstack/nova/+/923258/3
22:26:43 dansmith and the parent of it is in gate now I think
22:27:02 fungi i can insert it between the dequeue and reenqueue steps, sure
22:27:30 dansmith fungi: I think abhi still wanted that mostly to insulate them from job timeouts, but it's not strictly required and I think it'd be better if we don't drop all that testing now that tempest is merged
22:27:43 dansmith so you'd have my blessing to kick that out and make space, or I can if you want
22:27:43 fungi wfm
22:28:27 fungi adding it to the list of stuff i'll move to the back of the line
22:28:32 dansmith ++
22:36:34 fungi okay, surgery complete
22:37:10 fungi though in retrospect, as none of the changes at the top were being kept, it ended up being the same as if i'd done a promote of the ossa changes i guess
22:45:50 fungi i guess i expected there to be more changes from the ossa already approved
22:46:11 dansmith meaning the stable ones?
22:46:18 fungi turned out there were only 5 in the gate (plus the one that was kicked out and re-added)
22:46:20 dansmith all the master ones should be approved I think
22:46:46 dansmith given how things have been going with the insane cross-dependencies I think we've been kinda trying to make sure they are merging in release order
22:47:01 dansmith but if you really want them all +Wd we can probable just do that
22:47:15 fungi nah, it's okay
22:47:15 dansmith I've heard great things about melwitt's +W finger
22:47:27 melwitt lol
22:47:43 melwitt yup, I'm really great at clicking buttons
#openstack-nova - 2024-07-04
03:06:23 opendevreview Merged openstack/nova master: Additional qemu safety checking on base images https://review.opendev.org/c/openstack/nova/+/923257
06:13:40 opendevreview Merged openstack/nova stable/2023.1: Fix disk_formats in ceph job tempest config https://review.opendev.org/c/openstack/nova/+/923344
07:19:25 opendevreview Maxime Lubin proposed openstack/nova-specs master: USB over IP https://review.opendev.org/c/openstack/nova-specs/+/920687
07:25:43 frickler more gate resets happening all the time. I'll regate https://review.opendev.org/c/openstack/nova/+/923258 now, but not promote in order to give the glance stack a chance first
07:28:37 abhishekk thank you!

Earlier   Later