Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2024-07-05
13:11:51 sean-k-mooney fungi: i reappoved https://review.opendev.org/c/openstack/nova/+/923291 so you can try an promotet it if you want
13:11:57 fungi supporting dependencies across branches was a major design requirement because of grenade
13:12:11 fungi it'll get enqueued automatically in the current state anyway
13:12:40 fungi which it just did
13:12:49 sean-k-mooney ok im pretty sure if hit edgecase with that but it may have been related to somethign other then the merger
13:13:04 fungi now to see if nova-tox-validate-backport passes
13:13:58 sean-k-mooney if it does that will be good to know
13:15:38 fungi yeah, if it does pass then that means in theory it should also be able to pass in check with a depends-on to the version on the newer branch
13:16:21 fungi if it fails then i should be able to trace through the steps in the job to see where assumptions about the constructed future state on the other branch are getting invalidated
13:20:03 fungi multi-branch speculative states are something we've had for ages because it's necessary for making sure that grenade jobs run in the gate pipeline reflect the end state when the commits on different branches later merge, so we don't land changes on two different branches that then break upgrade tests when used together
13:23:13 fungi validate-backport: OK
13:23:19 fungi looks like it's going to succeed
13:23:31 fungi yep, just went green
13:24:00 fungi so the job is actually correctly designed to support this, there's not an actual need to wait for the other change to merge before you approve
13:25:44 fungi and using depends-on in cases like this would have two benefits: 1. you'll get an actual passing result (you don't even necessarily have to keep the job non-voting in check if you don't want to), and 2. zuul will ensure that the changes can't get enqueued into the gate pipeline in the wrong order so you can just approve them in any order you like
13:27:05 fungi the only caveat i'm aware of is that you need to be careful you don't inadvertently create dependency loops when you do that, but the same can be said for any use of depends-on
13:40:06 frickler ok, looks like I could reenqueue the deferred patches, then
13:44:22 dansmith frickler: sean-k-mooney: FWIW, I'd much rather have broken iso support and angry users than have this exposure.. I expect images to start showing up on download sites that look to contain something awesome, but are just the exploit
13:44:26 fungi in cases like this where there are series of changes for each branch, i'd probably conservatively just set depends-on in the commit message of the first change in the series pointing to the change url for the last change in the series on the newer branch
13:44:40 dansmith something a user of a private cloud might download and then upload into the internal glance and then ..boom
13:44:50 frickler humm, third nova-multi-cell failure on https://review.opendev.org/c/openstack/nova/+/923287 in a row, looks like something more serious
13:46:25 dansmith frickler: why serious? one fail..
13:46:48 dansmith looks like failure to talk to metadata server,
13:47:01 dansmith which could be the same sort of thing as prevents us from SSHing to the guest
13:47:10 dansmith oh actuall,
13:47:24 dansmith we're doing it from inside the guest, so "ssh guest curl metadata"
13:48:09 fungi so could be exactly the same problem in that case
13:48:17 opendevreview Merged openstack/nova stable/2023.1: Reject qcow files with data-file attributes https://review.opendev.org/c/openstack/nova/+/923288
13:48:21 frickler ok, I miscounted, twice in a row still https://zuul.opendev.org/t/openstack/build/51577af322ee4809889698bde352efac
13:49:47 frickler actually https://review.opendev.org/923289 is going to fail, too, stuck in devstack not finishing, waiting for the timeout to strike
13:50:56 frickler going to do some gate reshuffling to override
13:52:50 fungi seems like a good opportunity
13:55:55 opendevreview sean mooney proposed openstack/nova master: [WIP] add iso file format inspector https://review.opendev.org/c/openstack/nova/+/923573
13:57:00 sean-k-mooney oh i messed up that commit message
13:57:20 sean-k-mooney let me abandon the new review
13:57:53 opendevreview sean mooney proposed openstack/nova master: [WIP] add iso file format inspector https://review.opendev.org/c/openstack/nova/+/923533
14:54:31 frickler and another gate failure in the cve stack https://zuul.opendev.org/t/openstack/build/90c78eec9270430a8fcaad8652348d0c
14:57:31 dansmith nova.exception.InternalError: Unexpected vif_type=binding_failed
14:57:36 dansmith I assume that means some neutron fail
14:57:38 dansmith sean-k-mooney: ^
14:58:33 dansmith (frickler that was from the gate fail you quoted, in case it's not clear)
15:01:19 frickler yes, it is well possible that it is all unrelated and just unstable tests, I just wanted someone to check it
15:03:46 dansmith yep, definitely doesn't look related
15:19:52 fungi did 923288 get reenqueued into the gate accidentally? it's already merged
15:20:21 fungi if we're reshuffling for the current failures anyway, should probably eject that one at the same time
15:21:14 dansmith fungi: maybe someone wanted to be really really sure it landed?
15:21:23 fungi double-merged!
15:21:42 fungi a change so nice we'll merge it twice
15:21:43 dansmith there were definitely things in check and forced into gate at the same time yesterday, perhaps a zuul bug or race?
15:24:09 frickler hmm, according to my bash history, I didn't reenqueue 923288, but zuul has been doing some additional requeues of patches on top of others
15:25:00 frickler maybe a race condition when I did 923289 and zuul wasn't aware yet of 923288 having been merged
15:25:20 fungi normally only if they're not yet merged and meet criteria to be enqueued and depend on a change you've enqueued (either explicitly via depends-on or implicitly through git commit parentage)
15:26:08 fungi but yeah, can't rule out bugs either
15:26:19 frickler but likely if that buildset passes, it will result in a merge failure and cause a gate reset anyway, so we can as well abandon it now IMO
15:27:12 fungi i agree we should kick it out, i'd give 50/50 odds on whether it would actually cause a gate reset or just no-op (not sure what gerrit's behavior is if you call the submit api on an already-merged change)
15:27:22 frickler and I'm reaching EOD quickly (time to watch some soccer), so if you could take over, fungi, that'd be nice
15:27:32 fungi glad to!
15:28:06 fungi enjoy the fußball!
15:28:34 frickler there's also a new set of deferred patches in https://etherpad.opendev.org/p/cve-gate-reshuffling in case the gate gets lets crowded, else I'll handle these tomorrow
15:29:11 fungi thanks, i saw that too when i was looking back at the pad to see if i had maybe been the one to accidentally reenqueue the already-merged change
15:29:14 opendevreview sean mooney proposed openstack/nova master: add iso file format inspector https://review.opendev.org/c/openstack/nova/+/923533
15:30:39 sean-k-mooney frickler: is that the saem one i looked at eariler
15:31:08 sean-k-mooney i commented on the binding failed failure and recheck one fo the jobs for it when i was reviewing 2023.1
15:31:58 sean-k-mooney https://review.opendev.org/c/openstack/nova/+/923289/2#message-348bc1fded02d0ec6cd19807bcc2aec0245f606f
15:32:43 fungi okay, so going to reshuffle the gate putting these first: 923287,3 923289,2 923290,2 923291,3 (and drop 923288,2 since it's already merged)
15:33:05 fungi any other requests while i'm causing a full gate reset anyway?
15:35:12 fungi taking your silence as a resounding "no"
16:51:44 mnaser has anyone pushed/proposed the idea of rack-level failure domains in openstack?
16:51:55 mnaser I know a cloud is a cloud, but VMware people really feel like they see the need to get there
16:52:44 sean-k-mooney failure domaisn are really not a thing in openstack
16:52:51 JayF FYI; I know it's pretty low on you all's list right now, but the fix for the un-upgraded rlocks in eventlet for py3.12 is in the next release
16:53:07 sean-k-mooney the only way to really do that is using seperate keystone regions and isntalls of all the service per rack
16:53:55 sean-k-mooney mnaser: looking only at nova netiern cells or AZ map to failure domains
16:54:15 sean-k-mooney i.e. if the host running your schduer or api dies that affacts all hosts in that region
16:54:37 sean-k-mooney neutron also spanc all cells/AZs
16:54:49 fungi pretty sure the same can be said for the host running your vcenter api in a vmware environment
16:55:16 sean-k-mooney yep so it depens on your defintion of failure domains and if your inlcuding the contol plane in that
16:55:38 sean-k-mooney i.e. workload failrue domains vs netwrok vs storage vs contolplane
16:55:51 sean-k-mooney cinder backend are stroage level failure domains
16:56:39 sean-k-mooney of couse that is only true if you access them over seperate stoage networks
16:57:07 fungi or have some sort of high availability for the storage network
16:57:23 sean-k-mooney yep
16:57:45 mnaser sean-k-mooney: workload failure domains
16:57:46 sean-k-mooney mlag or similar with two diffent top of rack switch ectra
16:57:53 mnaser im thinking a scheduler hint maybe of some sorts + a filter?
16:57:57 fungi like cross-chassis lag
16:58:30 sean-k-mooney mnaser: we brifly discussed having somethign like cephs cluster map
16:58:37 mnaser yeah
16:59:11 sean-k-mooney basically building a tree of host aggrates and grouping things in a heriachy then allowing affintiy/anti affinity ectra
16:59:17 mnaser exactly
16:59:27 mnaser im thinking server group + an extra scheduler hint other than `group` to get that
16:59:39 mnaser but im wondering if im walking a path someone tried before and I can pick up work from or
16:59:56 sean-k-mooney we said it was proably too complex and out of scope of nova but that was also like 6 years ago so thngs have changes somewhat
17:00:18 sean-k-mooney the last time we really talks about this properly was pre/start of the pandmeic
17:00:51 sean-k-mooney mnaser: there is a request ot do az anti affinty this/last cycle
17:01:28 mnaser working in this case, it feels odd to have 1 az per rack
17:01:28 sean-k-mooney in generally i woul dnot be agsint this in principal. it just would not be high on my list of thing to do next
17:01:50 mnaser like that feels like its brute forcing the az feature if you're on rack level
17:01:52 sean-k-mooney az's are just named aggreates
17:02:37 mnaser yeah but I feel like in the perspective of a consumer of the cloud they might be like "oh isn't this like a azure/aws/etc az?"

Earlier   Later