Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2024-07-05
10:24:59 sean-k-mooney ok downstream email and pings sorted
10:25:19 sean-k-mooney frickler: anywhere i should start with the backports
10:25:50 sean-k-mooney or will i just work my way form top to bottom
10:27:06 frickler sean-k-mooney: well bottom to top I'd say in how I look at the stack (and gerrit UI seems to agree)
10:27:27 sean-k-mooney i ment form newest to oldes branch
10:27:36 sean-k-mooney but yes bottom to top within the chain
10:27:58 sean-k-mooney i was just checkign where we are in teh diffentb branches https://review.opendev.org/q/topic:%22bug/2059809%22+project:openstack/nova
10:28:14 sean-k-mooney looks like 2023.2 is almost done
10:28:30 sean-k-mooney so 2023.1 is next
10:28:37 frickler yes, only 2023.1 open, so starting at https://review.opendev.org/c/openstack/nova/+/923288/2 should be fine
10:29:18 frickler I'll ignore unmaintained branches fwiw
10:30:51 sean-k-mooney mnaser: may have na invested intreset :) they have the zed backports up
10:31:03 sean-k-mooney but i cant appoved them there anyway so ill stick to stabel
10:37:19 sean-k-mooney on a lighter note this is the before https://fileshare.seanmooney.info/average_tempest_runtime_for_tempest-integrated-compute_by_provider.png and after https://fileshare.seanmooney.info/average_tempest_runtime_for_tempest-integrated-compute_by_provider_2024-07-03.png for truning on the the devstack optimisations. both graphs are build form 600-700 ish succesful job runs orver
10:37:21 sean-k-mooney the proceeding 2 weeks
10:38:26 sean-k-mooney new runs are faster but still not quite enough to account for the job to job varince sometimes causing timeouts on slow nodews but we went form about a 10% timout rate to aroudn 5% and hopefully that is still droping
10:40:22 frickler sean-k-mooney: do you also want to approve the stack or do you think you need to wait for another review? just for comparison melwitt single-approved the 2023.2 ones
10:40:51 sean-k-mooney frickler: ill approve it once i get to the top im goign to +w all of the rest and then ill +w the bottom patch
10:41:35 sean-k-mooney im hoping the 2 that are pending in 2023.2 will merge before then so that the backport check does not kick the final patches out of gate
10:42:49 frickler does that check work on the speculative state in zuul or on the actual repos? the former should be fine as long as the other patches are ahead in the queue
10:42:59 sean-k-mooney actual
10:43:14 sean-k-mooney although if you have stable-only in the commit i think it skip it
10:43:24 sean-k-mooney that how we gerenally workaround it when we need to fast merge
10:43:35 sean-k-mooney we mark it stable only even when its not
10:44:46 frickler hmm, ok, then let's wait for the 2023.2 patches, then I can simply promote the whole 2023.1 stack once it is approved and don't need to do more reshuffling
10:45:22 sean-k-mooney ok in that case ill leave +w off the base until those merge then we can send the full set
10:45:42 frickler ack
10:59:10 sean-k-mooney looks like there is about 30 left on thje 2023.2 gate jobs so ill keep an eye on them
11:00:17 sean-k-mooney ok coffee then back to the iso regression
11:00:51 sean-k-mooney part of me does not like release the cve fixes without that as its a pretty common usecase but im hoping we can have that fixed properly today or very early next week
11:15:20 frickler I wouldn't worry about a release too much at this stage, as most people are running with the provisional patches currently anyway and might switch to head of master after today, but getting the iso issue fixed soon certainly will be a good idea
11:16:18 frickler humm, nova-multi-cell failing on https://review.opendev.org/c/openstack/nova/+/923287
11:16:21 sean-k-mooney personally im not sure i would deploy this in production without the iso fixes but ya
11:17:15 sean-k-mooney annorying but letss see why
11:17:59 sean-k-mooney its one of the volume test sbut not one of the normal kernel panics or similar
11:18:15 frickler you would rather run your cloud vulnerable?
11:18:38 frickler security group cleanup failure https://zuul.opendev.org/t/openstack/build/1f6d9308d0bb46999682ee43e856ee19
11:19:11 sean-k-mooney then break all usage of iso in nova (and ami iamges if you dont have the stable version of the patches)
11:19:21 sean-k-mooney it would depend on what my custoemrs are using
11:19:38 sean-k-mooney and if its a public or private cloud
11:20:17 sean-k-mooney for a private cloud with semi trusted workloads or iamges like our internal ci cloud i would wait
11:20:36 sean-k-mooney if i was vexhost or ovh i proably woudl not
11:21:14 sean-k-mooney frickler: anyway lookign at the failure i dont think its related so we can recheck when the job reports back
11:23:41 frickler I wouldn't even recheck but promote into gate again immediately, just behind the 2023.1 stack
11:24:25 sean-k-mooney fair, that works for me too
11:35:51 opendevreview Merged openstack/nova stable/2023.2: Additional qemu safety checking on base images https://review.opendev.org/c/openstack/nova/+/923286
11:36:11 frickler sean-k-mooney: ready to approve 2023.1 ^^
11:36:44 sean-k-mooney dont we still need to wait for https://review.opendev.org/c/openstack/nova/+/923287/3
11:37:33 frickler sean-k-mooney: no, that is the one that failed, I'm re-enqueueing it now
11:37:34 sean-k-mooney i can appove and most of them will merge but i think its imporant to not cover the vmdk part too
11:37:56 sean-k-mooney oh ok
11:39:07 sean-k-mooney ill send them but we might need to handel that on 2023.1 seperatly
11:39:13 sean-k-mooney we will see what the timeing looks like
11:40:26 frickler sean-k-mooney: ah, right, maybe drop the W+1 from 923291 for now, then
11:41:49 sean-k-mooney sure done
11:44:45 frickler thx, promoted all the other changes now, we'll see more in 2h hopefully
11:45:38 sean-k-mooney let me know if i can help with anything else, ill check on them perodicly
11:50:40 frickler ack
12:45:30 fungi looks like https://review.opendev.org/c/openstack/nova/+/923291 got unapproved at some point, are we ready to reapprove it?
12:47:15 fungi oh, though it's not explicitly listed in the ossa so i guess it's not as urgent
12:47:59 sean-k-mooney i unapproved it because the 2023.2 version failed and it would fail if it went to the gate on the backport check
12:48:25 sean-k-mooney fungi: so ill reappoave it when https://review.opendev.org/c/openstack/nova/+/923287 is merged
12:48:58 fungi hopefully the backport check wouldn't fail when the 2023.2 was ahead of it in the gate unless that test isn't properly speculative, but it seems there's no harm in waiting on that one either
12:49:06 sean-k-mooney it will
12:49:16 sean-k-mooney its intentially not sepculiive
12:49:35 fungi what's the reasoning for that?
12:49:58 sean-k-mooney the reason it exits is to make sure we never merge soemthign in an older branch if tis not merged in the newer branch
12:50:12 sean-k-mooney we can overried it by marking the change stable-only
12:50:28 sean-k-mooney which is what we normally do for CVEs but didnt this time
12:50:41 fungi yeah, but making it check zuul's speculative state would still accomplish that, because it would get retested if the newer branch's version gets kicked out for any reason and then would fail
12:51:15 sean-k-mooney fungi: if you know how to update the script to do that feel free to propse it :)
12:51:39 fungi i don't even know what job you're talking about, but if you have a pointer to the script i can take a look
12:51:47 sean-k-mooney https://github.com/openstack/nova/blob/master/tools/check-cherry-picks.sh
12:51:52 fungi it's likely just a matter of making the script do less than it does now
12:52:05 sean-k-mooney this also need to work locally
12:54:32 sean-k-mooney the end goal of the script is to make sure we dont regress going form 2023.1 to 2023.2 ectra
12:54:43 fungi i guess i'll need to look at the context where the script is run, because i don't see it doing anything that should inherently break speculation, but if some earlier step is checking out from the origin remote instead of the current branch state that would break it
12:55:45 sean-k-mooney it runs in check as non voteing and gate as voting
12:56:13 fungi i mean the job that runs that script
12:56:15 sean-k-mooney fungi: from my perspective not doing speclicitive execution is safer and more correct
12:56:20 sean-k-mooney so i dont consider this to be a bug
12:56:37 fungi i don't see it, but not my circus, not my monkeys ;)
12:57:06 sean-k-mooney its called nova-tox-validate-backport
12:57:17 sean-k-mooney https://zuul.opendev.org/t/openstack/build/a6be643734b44f139bca7b4eb8963814
12:57:37 sean-k-mooney Cherry pick hash 11301e7e3f0d81a3368632f90608e30d9c647111 not on any master or stable branches
12:57:42 fungi if you want backports to be delayed from being enqueued until other changes merge, and slow down throughput as a result even though zuul can ensure the correctness of the thing you actually want without that additional inefficiency, i'm not going to argue
12:58:34 sean-k-mooney fungi: can you expalin how zuul can ensure that a patch is merged on all newer stable branches before it will allow it to merge on the older ones
12:58:45 fungi yes, speculative testing
12:58:54 sean-k-mooney fungi: because as far as im aware it cant. it would be incorrect to use depend on
12:59:07 sean-k-mooney and it does not have awareness of the depeency in any other way
12:59:20 fungi zuul assumes the newer branch backport ahead of it will merge, but if it doesn't then testing gets rerun without that change present in the queue and the job will fail
12:59:40 sean-k-mooney that only true if they are both in the queu at the same time
12:59:45 sean-k-mooney that is not normlaly the case
13:00:07 fungi yes, if they're not in the queue at the same time then the job will just fail when that change isn't already merged, or succeed if it is
13:00:38 sean-k-mooney why woudl it fail
13:01:00 fungi zuul is presenting all the branches to the job with a speculative future state, but if that speculative state changes then the presumed future is invalidated and a new possible future is constructed and tests are rerun
13:01:03 sean-k-mooney fungi: what your stating does not fit with my mental model of how zuul works and im pretty sure it not corect
13:01:35 sean-k-mooney fungi: how would zuul know about the precondition
13:02:00 sean-k-mooney its not express in the normal jobs or via depned on
13:02:12 fungi zuul assumes that changes in the same queue share a possible future

Earlier   Later