| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-07-03 | |||
| 02:22:00 | dansmith | I think he can do it, but not sure | |
| 02:22:15 | abhishek_ | will ask him if required, thank you! | |
| 02:37:38 | opendevreview | Merged openstack/nova master: Fix disk_formats in ceph job tempest config https://review.opendev.org/c/openstack/nova/+/923322 | |
| 02:37:46 | fungi | yay! | |
| 02:37:51 | dansmith | amazing | |
| 02:38:18 | dansmith | fungi: do we prioritize the stable ones too or no? | |
| 02:39:17 | fungi | might not be necessary this time of day if they already have passing check results and there's not much else in the gate, i haven't looked | |
| 02:41:53 | dansmith | ack | |
| 02:43:25 | fungi | but if it will help in getting them merged faster we still can | |
| 02:43:54 | fungi | just depends on whether there's much of a queue or jobs are continuing to be problematic | |
| 02:44:00 | dansmith | abhishek_: ^ dunno how important it is | |
| 02:44:52 | abhishek_ | dansmith: I think we can wait, since it will going to take a day long to merge master patches | |
| 02:44:59 | dansmith | ack | |
| 02:46:17 | abhishek_ | nova merged | |
| 05:13:29 | frickler | well the gate is currently looking ok-ish after https://review.opendev.org/923344 caused a gate reset. let's hope the current stack merges without further failures. I'll try to keep an eye on it and look into reshuffling if further issues should happen | |
| 06:00:12 | frickler | and there goes hoping ... weird early failure in grenade-multinode that I haven't seen before, but looks 99% unrelated https://zuul.opendev.org/t/openstack/build/322833ee6e7a43b0aee2ca9803fc4f57 | |
| 07:27:51 | bauzas | good morning folks | |
| 07:28:05 | bauzas | frickler: I was on PTO yesterday, so I just got the emails | |
| 07:28:16 | bauzas | but AFAICS, we needed to update Tempest rightN? | |
| 07:28:34 | bauzas | https://review.opendev.org/c/openstack/nova/+/923322 | |
| 07:28:51 | bauzas | I'm now chasing https://review.opendev.org/c/openstack/nova/+/923255/2 | |
| 07:32:28 | frickler | bauzas: the latter is failing in gate with the error I posted before, IMO it could be re-enqueued into gate immediately once zuul reports the failures, together with the stack on top of it | |
| 07:34:05 | frickler | I found keystone (once again) to be essentially undebuggable with the amount of errors generated during normal operations | |
| 07:34:58 | bauzas | yeah https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_322/923255/2/gate/nova-grenade-multinode/322833e/controller/logs/grenade.sh_log.txt | |
| 07:35:11 | bauzas | my only concern is that I'm not sure this is a transient issue | |
| 07:35:21 | bauzas | looks more a keystone regression to me | |
| 07:35:29 | bauzas | 2024-07-03 05:20:04.386 | testtools.matchers._impl.MismatchError: '56fc53e2e0724c0284ab80060ec55421' != '39aa3070a75e446fb6aa6d48bab74c58' | |
| 07:36:42 | bauzas | unless someone keystone-ed able to tell me this is something like a race condition in Keystone | |
| 08:00:43 | opendevreview | Merged openstack/nova stable/2024.1: Fix disk_formats in ceph job tempest config https://review.opendev.org/c/openstack/nova/+/923342 | |
| 08:14:53 | frickler | bauzas: if it is a regression it should fail in gate next time, too, but I don't see any recent changes in keystone. so I still suggest to reenqueue the stack into gate, unless your prefer to only recheck it | |
| 08:15:11 | bauzas | I'll recheck it | |
| 08:15:18 | bauzas | and we'll see | |
| 08:17:25 | bauzas | done | |
| 08:29:47 | opendevreview | Maxime Lubin proposed openstack/nova-specs master: USB over IP https://review.opendev.org/c/openstack/nova-specs/+/923362 | |
| 08:43:38 | opendevreview | Maxime Lubin proposed openstack/nova-specs master: USB over IP https://review.opendev.org/c/openstack/nova-specs/+/920687 | |
| 08:56:31 | opendevreview | Maxime Lubin proposed openstack/nova-specs master: USB over IP https://review.opendev.org/c/openstack/nova-specs/+/920687 | |
| 09:16:29 | opendevreview | Maxime Lubin proposed openstack/nova-specs master: USB over IP https://review.opendev.org/c/openstack/nova-specs/+/920687 | |
| 09:20:23 | Uggla | @gibi, can you have a look at this patch : https://review.opendev.org/c/openstack/nova/+/868089/8 can it be merged ? | |
| 10:07:23 | frickler | next failure, this time one of the "regulars" tempest.api.compute.servers.test_server_actions.ServerActionsV293TestJSON.test_rebuild_volume_backed_server | |
| 10:08:49 | bauzas | yeepeekay | |
| 10:43:57 | sean-k-mooney | frickler: bauzas i requeued the first patch but i kind fo feel like waiting for that to merge before queueign the rest | |
| 10:44:05 | sean-k-mooney | what do ye think | |
| 10:44:22 | sean-k-mooney | shoudl we hold all other nova patches until this lands and ask peopel not to recheck on the mailing list | |
| 10:45:03 | sean-k-mooney | the random failure are not nessisarly load related so im not really sure if it will help | |
| 10:45:48 | sean-k-mooney | but we are consumign a lot of ci resouces enquing all 4 patches just to have them kicked out because of a failure in the first one | |
| 11:15:44 | frickler | sean-k-mooney: on one hand I understand that concern, on the other hand these patches are being used in production in a lot of systems already, so IMO it should have highest priority to at least get them into master, even if other projects have to suffer a bit from that | |
| 11:16:54 | sean-k-mooney | ok so your in faovr of quein gthe rest. i was not sure if it would increase the likely hood fo a failure but ok ill recheck the remaining 3 and get them moving | |
| 11:17:40 | sean-k-mooney | actully im not sure the second one need a recheck it still has +1 | |
| 11:18:33 | sean-k-mooney | i have rechecked the third pathch | |
| 11:18:44 | frickler | as I said earlier I would also be in favor of moving these to gate directly | |
| 11:18:59 | frickler | as the failures pretty certainly look unrelated | |
| 11:19:18 | sean-k-mooney | im not agaisnt taht i just dont want to keep pinging fungi, you or clark ot do that | |
| 11:19:38 | zhhuabj | hi team, does anyone know why ci can't run in this patch - https://review.opendev.org/c/openstack/nova/+/909611 | |
| 11:19:49 | sean-k-mooney | oh ya they are not related any more. we had a fully green run last night on the last patch so the set as a whole i think is good | |
| 11:19:51 | frickler | well currently it is me who keeps pinging nova people with the offer ;) | |
| 11:20:32 | sean-k-mooney | i think we have got enouch results form check in agggreate to be ok with mergeing them as is at this point | |
| 11:20:37 | frickler | zhhuabj: that patch needs a rebase, see the "not current" on the relation chain | |
| 11:24:24 | zhhuabj | thanks frickler, I will ping patch owner to do rebase to have a try, thanks | |
| 11:30:18 | sean-k-mooney | zhhuabj: i left some feedback on the parent patch https://review.opendev.org/c/openstack/nova/+/920374 | |
| 11:31:02 | sean-k-mooney | the unit tests you added are not inline with our code style, specifcaly you are addming mulitle diffent test cases in a single test method and you are not using asserts correctly. | |
| 11:31:18 | frickler | ok, so now I need to refresh my memory on how that "enqueue into gate" actually works ;) | |
| 11:44:42 | frickler | hmm, got myself a keycloak account, but that only seems to allow dequeue + promote, need to read more docs | |
| 12:07:20 | zhhuabj | hi sean-k-mooney , I saw your comments , thanks for your review. you mentioned using assert_called_with instead of 'assert str(expected_call) == str(actual_call)', deepcopy may change object's memory address as well, so assert_called_with will not work | |
| 12:07:37 | fungi | also, i'm done with my morning errands and can help (re)enqueue changes into the gate, reorder the queue, delete verified -2 results, et cetera | |
| 12:08:18 | fungi | frickler: i keep forgetting about the webui, just been using the zuul-client cli | |
| 12:10:02 | zhhuabj | sean-k-mooney: this is error log when using assert_called_with, we can see actually these two strings are equal, but assert-called-with will throw the exception, so I think that's the reason deepcopy has changed memory address of object - https://paste.openstack.org/show/boR6m5V8Zegg5p9iRXlG/ | |
| 12:11:57 | sean-k-mooney | that is proably because instance jobjstefc has changed field tahre are not being printed | |
| 12:13:05 | sean-k-mooney | you could compare the object_to_primitive dump but just comaprign the srting wont compare them properly | |
| 12:13:45 | frickler | fungi: yes, I just needed to figure out how to drop the V-2 first, but found that in the force-merge guide | |
| 12:14:17 | frickler | now I was just a little bit confused because I only touched one change, but the whole stack landed in gate at the same time | |
| 12:14:43 | frickler | but it seems all the others had already finished their rechecks | |
| 12:15:03 | fungi | zuul will auto-enqueue dependent changes if they meet its requirements | |
| 12:25:59 | zhhuabj | sean-k-moonkey: do you mean this one - https://paste.openstack.org/show/bd510w2dMMZdwOOjEvS7/ | |
| 12:35:02 | sean-k-mooney | kindof depending on which parmater had the untracked changes | |
| 12:36:23 | opendevreview | Takashi Kajinami proposed openstack/os-vif master: Remove old excludes https://review.opendev.org/c/openstack/os-vif/+/917611 | |
| 12:52:28 | opendevreview | Zhang Hua proposed openstack/nova master: BlockDeviceMapping object has no attribute copy https://review.opendev.org/c/openstack/nova/+/920374 | |
| 12:54:44 | zhhuabj | hi sean-k-mooney , I posted a new change to address your comment - https://review.opendev.org/c/openstack/nova/+/920374/3/nova/tests/unit/virt/libvirt/test_blockinfo.py | |
| 12:56:24 | opendevreview | Merged openstack/nova stable/2023.2: Fix disk_formats in ceph job tempest config https://review.opendev.org/c/openstack/nova/+/923343 | |
| 12:58:05 | opendevreview | Sahid Orentino Ferdjaoui proposed openstack/nova master: scheduler: fix _get_sharing_providers to support unlimited aggr https://review.opendev.org/c/openstack/nova/+/921665 | |
| 13:04:18 | sean-k-mooney | zhhuabj: thansk we are currently focusing on upstream and downstream tasks related to the cve discustion yesterdaty but we will loop back and review when we have time | |
| 13:10:17 | opendevreview | Andrei Yachmenev proposed openstack/nova master: Fix processing uniqueness of instance host names https://review.opendev.org/c/openstack/nova/+/923395 | |
| 13:23:22 | dansmith | head nova patch is going to fail openstacksdk again | |
| 13:25:18 | dansmith | same autoallocate thing it appears :( | |
| 13:28:07 | sean-k-mooney | that has been flaky for a while it seams do we want to make it non voting until the cve patches are laneded or just recheck | |
| 13:28:22 | dansmith | idk | |
| 13:28:43 | sean-k-mooney | we could quickly update it with the flaky decorator | |
| 13:28:52 | sean-k-mooney | that will skip if it fails and run it otherwise | |
| 13:29:10 | sean-k-mooney | but the sdk team would need to be aware they need to fix it | |
| 13:33:24 | frickler | there's a fix already proposed for the sdk job https://review.opendev.org/c/openstack/openstacksdk/+/923379 | |
| 13:36:44 | frickler | I guess we should fast-approve that and fix stephenfin's comments in a followup? | |
| 13:37:19 | sean-k-mooney | stephenfin: ^ | |
| 13:37:23 | sean-k-mooney | yes i think so | |
| 13:41:53 | frickler | done and promoted to top of gate. I think we should also dequeue and re-enqueue the nova stack? | |
| 13:42:30 | frickler | oh, the promotion took care of that already | |
| 13:46:55 | bauzas | sorry my IRC bouncer has some problems | |
| 13:47:02 | bauzas | frickler: where are we now ? | |
| 13:48:45 | frickler | bauzas: the nova stack was failing again due to an issue in the sdk job. this is to be fixed by https://review.opendev.org/c/openstack/openstacksdk/+/923379 which I've now promoted to the head of the gate pipeline | |
| 13:49:23 | bauzas | ack | |
| 13:49:55 | frickler | also https://meetings.opendev.org/irclogs/%23openstack-nova/latest.log.html is always there to help you, but updated only every 15 minutes | |
| 13:59:31 | fungi | if you're in a hurry and don't want to wait for the update, the less fancy txt version should be more continuously updated too | |