Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2024-07-03
00:20:08 fungi once it (hopefully) merges, i'll reenqueue and promote the nova change again
00:25:11 fungi okay, glance change landed, nova change is back to the top of the gate for a third try
00:48:36 dansmith fungi++
00:54:54 dansmith fungi: that new defaults job has been timeout-heavy lately for sure, I think sean had a thing to bump the timeout, which I suppose we might have to do..
01:55:19 opendevreview Dan Smith proposed openstack/nova stable/2024.1: Fix disk_formats in ceph job tempest config https://review.opendev.org/c/openstack/nova/+/923342
01:55:31 opendevreview Dan Smith proposed openstack/nova stable/2023.2: Fix disk_formats in ceph job tempest config https://review.opendev.org/c/openstack/nova/+/923343
01:55:43 opendevreview Dan Smith proposed openstack/nova stable/2023.1: Fix disk_formats in ceph job tempest config https://review.opendev.org/c/openstack/nova/+/923344
01:55:44 dansmith abhishek_: ^
01:56:24 dansmith melwitt: if you're around by chance and could monitor those ^ we need them in ASAP (master still pending) so we can get glance CVE patches merged that depend on them
01:56:41 abhishek_ dansmith: ++
02:12:11 melwitt dansmith: ok, I can do it
02:14:14 dansmith melwitt: thanks, master has been rechecked several times already, so who knows
02:16:51 melwitt yeah. we can at least get some more dice rolls in before EU timezone
02:21:16 abhishek_ is there someone in this timezone who can prioritise the patch in gate (just in case if required)
02:21:29 dansmith melwitt: ++
02:21:46 dansmith abhishek_: I'm not sure, frickler will be coming online, might be the earliest
02:21:58 abhishek_ ack
02:22:00 dansmith I think he can do it, but not sure
02:22:15 abhishek_ will ask him if required, thank you!
02:37:38 opendevreview Merged openstack/nova master: Fix disk_formats in ceph job tempest config https://review.opendev.org/c/openstack/nova/+/923322
02:37:46 fungi yay!
02:37:51 dansmith amazing
02:38:18 dansmith fungi: do we prioritize the stable ones too or no?
02:39:17 fungi might not be necessary this time of day if they already have passing check results and there's not much else in the gate, i haven't looked
02:41:53 dansmith ack
02:43:25 fungi but if it will help in getting them merged faster we still can
02:43:54 fungi just depends on whether there's much of a queue or jobs are continuing to be problematic
02:44:00 dansmith abhishek_: ^ dunno how important it is
02:44:52 abhishek_ dansmith: I think we can wait, since it will going to take a day long to merge master patches
02:44:59 dansmith ack
02:46:17 abhishek_ nova merged
05:13:29 frickler well the gate is currently looking ok-ish after https://review.opendev.org/923344 caused a gate reset. let's hope the current stack merges without further failures. I'll try to keep an eye on it and look into reshuffling if further issues should happen
06:00:12 frickler and there goes hoping ... weird early failure in grenade-multinode that I haven't seen before, but looks 99% unrelated https://zuul.opendev.org/t/openstack/build/322833ee6e7a43b0aee2ca9803fc4f57
07:27:51 bauzas good morning folks
07:28:05 bauzas frickler: I was on PTO yesterday, so I just got the emails
07:28:16 bauzas but AFAICS, we needed to update Tempest rightN?
07:28:34 bauzas https://review.opendev.org/c/openstack/nova/+/923322
07:28:51 bauzas I'm now chasing https://review.opendev.org/c/openstack/nova/+/923255/2
07:32:28 frickler bauzas: the latter is failing in gate with the error I posted before, IMO it could be re-enqueued into gate immediately once zuul reports the failures, together with the stack on top of it
07:34:05 frickler I found keystone (once again) to be essentially undebuggable with the amount of errors generated during normal operations
07:34:58 bauzas yeah https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_322/923255/2/gate/nova-grenade-multinode/322833e/controller/logs/grenade.sh_log.txt
07:35:11 bauzas my only concern is that I'm not sure this is a transient issue
07:35:21 bauzas looks more a keystone regression to me
07:35:29 bauzas 2024-07-03 05:20:04.386 | testtools.matchers._impl.MismatchError: '56fc53e2e0724c0284ab80060ec55421' != '39aa3070a75e446fb6aa6d48bab74c58'
07:36:42 bauzas unless someone keystone-ed able to tell me this is something like a race condition in Keystone
08:00:43 opendevreview Merged openstack/nova stable/2024.1: Fix disk_formats in ceph job tempest config https://review.opendev.org/c/openstack/nova/+/923342
08:14:53 frickler bauzas: if it is a regression it should fail in gate next time, too, but I don't see any recent changes in keystone. so I still suggest to reenqueue the stack into gate, unless your prefer to only recheck it
08:15:11 bauzas I'll recheck it
08:15:18 bauzas and we'll see
08:17:25 bauzas done
08:29:47 opendevreview Maxime Lubin proposed openstack/nova-specs master: USB over IP https://review.opendev.org/c/openstack/nova-specs/+/923362
08:43:38 opendevreview Maxime Lubin proposed openstack/nova-specs master: USB over IP https://review.opendev.org/c/openstack/nova-specs/+/920687
08:56:31 opendevreview Maxime Lubin proposed openstack/nova-specs master: USB over IP https://review.opendev.org/c/openstack/nova-specs/+/920687
09:16:29 opendevreview Maxime Lubin proposed openstack/nova-specs master: USB over IP https://review.opendev.org/c/openstack/nova-specs/+/920687
09:20:23 Uggla @gibi, can you have a look at this patch : https://review.opendev.org/c/openstack/nova/+/868089/8 can it be merged ?
10:07:23 frickler next failure, this time one of the "regulars" tempest.api.compute.servers.test_server_actions.ServerActionsV293TestJSON.test_rebuild_volume_backed_server
10:08:49 bauzas yeepeekay
10:43:57 sean-k-mooney frickler: bauzas i requeued the first patch but i kind fo feel like waiting for that to merge before queueign the rest
10:44:05 sean-k-mooney what do ye think
10:44:22 sean-k-mooney shoudl we hold all other nova patches until this lands and ask peopel not to recheck on the mailing list
10:45:03 sean-k-mooney the random failure are not nessisarly load related so im not really sure if it will help
10:45:48 sean-k-mooney but we are consumign a lot of ci resouces enquing all 4 patches just to have them kicked out because of a failure in the first one
11:15:44 frickler sean-k-mooney: on one hand I understand that concern, on the other hand these patches are being used in production in a lot of systems already, so IMO it should have highest priority to at least get them into master, even if other projects have to suffer a bit from that
11:16:54 sean-k-mooney ok so your in faovr of quein gthe rest. i was not sure if it would increase the likely hood fo a failure but ok ill recheck the remaining 3 and get them moving
11:17:40 sean-k-mooney actully im not sure the second one need a recheck it still has +1
11:18:33 sean-k-mooney i have rechecked the third pathch
11:18:44 frickler as I said earlier I would also be in favor of moving these to gate directly
11:18:59 frickler as the failures pretty certainly look unrelated
11:19:18 sean-k-mooney im not agaisnt taht i just dont want to keep pinging fungi, you or clark ot do that
11:19:38 zhhuabj hi team, does anyone know why ci can't run in this patch - https://review.opendev.org/c/openstack/nova/+/909611
11:19:49 sean-k-mooney oh ya they are not related any more. we had a fully green run last night on the last patch so the set as a whole i think is good
11:19:51 frickler well currently it is me who keeps pinging nova people with the offer ;)
11:20:32 sean-k-mooney i think we have got enouch results form check in agggreate to be ok with mergeing them as is at this point
11:20:37 frickler zhhuabj: that patch needs a rebase, see the "not current" on the relation chain
11:24:24 zhhuabj thanks frickler, I will ping patch owner to do rebase to have a try, thanks
11:30:18 sean-k-mooney zhhuabj: i left some feedback on the parent patch https://review.opendev.org/c/openstack/nova/+/920374
11:31:02 sean-k-mooney the unit tests you added are not inline with our code style, specifcaly you are addming mulitle diffent test cases in a single test method and you are not using asserts correctly.
11:31:18 frickler ok, so now I need to refresh my memory on how that "enqueue into gate" actually works ;)
11:44:42 frickler hmm, got myself a keycloak account, but that only seems to allow dequeue + promote, need to read more docs
12:07:20 zhhuabj hi sean-k-mooney , I saw your comments , thanks for your review. you mentioned using assert_called_with instead of 'assert str(expected_call) == str(actual_call)', deepcopy may change object's memory address as well, so assert_called_with will not work
12:07:37 fungi also, i'm done with my morning errands and can help (re)enqueue changes into the gate, reorder the queue, delete verified -2 results, et cetera
12:08:18 fungi frickler: i keep forgetting about the webui, just been using the zuul-client cli
12:10:02 zhhuabj sean-k-mooney: this is error log when using assert_called_with, we can see actually these two strings are equal, but assert-called-with will throw the exception, so I think that's the reason deepcopy has changed memory address of object - https://paste.openstack.org/show/boR6m5V8Zegg5p9iRXlG/
12:11:57 sean-k-mooney that is proably because instance jobjstefc has changed field tahre are not being printed
12:13:05 sean-k-mooney you could compare the object_to_primitive dump but just comaprign the srting wont compare them properly
12:13:45 frickler fungi: yes, I just needed to figure out how to drop the V-2 first, but found that in the force-merge guide
12:14:17 frickler now I was just a little bit confused because I only touched one change, but the whole stack landed in gate at the same time
12:14:43 frickler but it seems all the others had already finished their rechecks
12:15:03 fungi zuul will auto-enqueue dependent changes if they meet its requirements
12:25:59 zhhuabj sean-k-moonkey: do you mean this one - https://paste.openstack.org/show/bd510w2dMMZdwOOjEvS7/
12:35:02 sean-k-mooney kindof depending on which parmater had the untracked changes
12:36:23 opendevreview Takashi Kajinami proposed openstack/os-vif master: Remove old excludes https://review.opendev.org/c/openstack/os-vif/+/917611
12:52:28 opendevreview Zhang Hua proposed openstack/nova master: BlockDeviceMapping object has no attribute copy https://review.opendev.org/c/openstack/nova/+/920374
12:54:44 zhhuabj hi sean-k-mooney , I posted a new change to address your comment - https://review.opendev.org/c/openstack/nova/+/920374/3/nova/tests/unit/virt/libvirt/test_blockinfo.py
12:56:24 opendevreview Merged openstack/nova stable/2023.2: Fix disk_formats in ceph job tempest config https://review.opendev.org/c/openstack/nova/+/923343
12:58:05 opendevreview Sahid Orentino Ferdjaoui proposed openstack/nova master: scheduler: fix _get_sharing_providers to support unlimited aggr https://review.opendev.org/c/openstack/nova/+/921665
13:04:18 sean-k-mooney zhhuabj: thansk we are currently focusing on upstream and downstream tasks related to the cve discustion yesterdaty but we will loop back and review when we have time
13:10:17 opendevreview Andrei Yachmenev proposed openstack/nova master: Fix processing uniqueness of instance host names https://review.opendev.org/c/openstack/nova/+/923395
13:23:22 dansmith head nova patch is going to fail openstacksdk again
13:25:18 dansmith same autoallocate thing it appears :(

Earlier   Later