Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2024-07-03
07:27:51 bauzas good morning folks
07:28:05 bauzas frickler: I was on PTO yesterday, so I just got the emails
07:28:16 bauzas but AFAICS, we needed to update Tempest rightN?
07:28:34 bauzas https://review.opendev.org/c/openstack/nova/+/923322
07:28:51 bauzas I'm now chasing https://review.opendev.org/c/openstack/nova/+/923255/2
07:32:28 frickler bauzas: the latter is failing in gate with the error I posted before, IMO it could be re-enqueued into gate immediately once zuul reports the failures, together with the stack on top of it
07:34:05 frickler I found keystone (once again) to be essentially undebuggable with the amount of errors generated during normal operations
07:34:58 bauzas yeah https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_322/923255/2/gate/nova-grenade-multinode/322833e/controller/logs/grenade.sh_log.txt
07:35:11 bauzas my only concern is that I'm not sure this is a transient issue
07:35:21 bauzas looks more a keystone regression to me
07:35:29 bauzas 2024-07-03 05:20:04.386 | testtools.matchers._impl.MismatchError: '56fc53e2e0724c0284ab80060ec55421' != '39aa3070a75e446fb6aa6d48bab74c58'
07:36:42 bauzas unless someone keystone-ed able to tell me this is something like a race condition in Keystone
08:00:43 opendevreview Merged openstack/nova stable/2024.1: Fix disk_formats in ceph job tempest config https://review.opendev.org/c/openstack/nova/+/923342
08:14:53 frickler bauzas: if it is a regression it should fail in gate next time, too, but I don't see any recent changes in keystone. so I still suggest to reenqueue the stack into gate, unless your prefer to only recheck it
08:15:11 bauzas I'll recheck it
08:15:18 bauzas and we'll see
08:17:25 bauzas done
08:29:47 opendevreview Maxime Lubin proposed openstack/nova-specs master: USB over IP https://review.opendev.org/c/openstack/nova-specs/+/923362
08:43:38 opendevreview Maxime Lubin proposed openstack/nova-specs master: USB over IP https://review.opendev.org/c/openstack/nova-specs/+/920687
08:56:31 opendevreview Maxime Lubin proposed openstack/nova-specs master: USB over IP https://review.opendev.org/c/openstack/nova-specs/+/920687
09:16:29 opendevreview Maxime Lubin proposed openstack/nova-specs master: USB over IP https://review.opendev.org/c/openstack/nova-specs/+/920687
09:20:23 Uggla @gibi, can you have a look at this patch : https://review.opendev.org/c/openstack/nova/+/868089/8 can it be merged ?
10:07:23 frickler next failure, this time one of the "regulars" tempest.api.compute.servers.test_server_actions.ServerActionsV293TestJSON.test_rebuild_volume_backed_server
10:08:49 bauzas yeepeekay
10:43:57 sean-k-mooney frickler: bauzas i requeued the first patch but i kind fo feel like waiting for that to merge before queueign the rest
10:44:05 sean-k-mooney what do ye think
10:44:22 sean-k-mooney shoudl we hold all other nova patches until this lands and ask peopel not to recheck on the mailing list
10:45:03 sean-k-mooney the random failure are not nessisarly load related so im not really sure if it will help
10:45:48 sean-k-mooney but we are consumign a lot of ci resouces enquing all 4 patches just to have them kicked out because of a failure in the first one
11:15:44 frickler sean-k-mooney: on one hand I understand that concern, on the other hand these patches are being used in production in a lot of systems already, so IMO it should have highest priority to at least get them into master, even if other projects have to suffer a bit from that
11:16:54 sean-k-mooney ok so your in faovr of quein gthe rest. i was not sure if it would increase the likely hood fo a failure but ok ill recheck the remaining 3 and get them moving
11:17:40 sean-k-mooney actully im not sure the second one need a recheck it still has +1
11:18:33 sean-k-mooney i have rechecked the third pathch
11:18:44 frickler as I said earlier I would also be in favor of moving these to gate directly
11:18:59 frickler as the failures pretty certainly look unrelated
11:19:18 sean-k-mooney im not agaisnt taht i just dont want to keep pinging fungi, you or clark ot do that
11:19:38 zhhuabj hi team, does anyone know why ci can't run in this patch - https://review.opendev.org/c/openstack/nova/+/909611
11:19:49 sean-k-mooney oh ya they are not related any more. we had a fully green run last night on the last patch so the set as a whole i think is good
11:19:51 frickler well currently it is me who keeps pinging nova people with the offer ;)
11:20:32 sean-k-mooney i think we have got enouch results form check in agggreate to be ok with mergeing them as is at this point
11:20:37 frickler zhhuabj: that patch needs a rebase, see the "not current" on the relation chain
11:24:24 zhhuabj thanks frickler, I will ping patch owner to do rebase to have a try, thanks
11:30:18 sean-k-mooney zhhuabj: i left some feedback on the parent patch https://review.opendev.org/c/openstack/nova/+/920374
11:31:02 sean-k-mooney the unit tests you added are not inline with our code style, specifcaly you are addming mulitle diffent test cases in a single test method and you are not using asserts correctly.
11:31:18 frickler ok, so now I need to refresh my memory on how that "enqueue into gate" actually works ;)
11:44:42 frickler hmm, got myself a keycloak account, but that only seems to allow dequeue + promote, need to read more docs
12:07:20 zhhuabj hi sean-k-mooney , I saw your comments , thanks for your review. you mentioned using assert_called_with instead of 'assert str(expected_call) == str(actual_call)', deepcopy may change object's memory address as well, so assert_called_with will not work
12:07:37 fungi also, i'm done with my morning errands and can help (re)enqueue changes into the gate, reorder the queue, delete verified -2 results, et cetera
12:08:18 fungi frickler: i keep forgetting about the webui, just been using the zuul-client cli
12:10:02 zhhuabj sean-k-mooney: this is error log when using assert_called_with, we can see actually these two strings are equal, but assert-called-with will throw the exception, so I think that's the reason deepcopy has changed memory address of object - https://paste.openstack.org/show/boR6m5V8Zegg5p9iRXlG/
12:11:57 sean-k-mooney that is proably because instance jobjstefc has changed field tahre are not being printed
12:13:05 sean-k-mooney you could compare the object_to_primitive dump but just comaprign the srting wont compare them properly
12:13:45 frickler fungi: yes, I just needed to figure out how to drop the V-2 first, but found that in the force-merge guide
12:14:17 frickler now I was just a little bit confused because I only touched one change, but the whole stack landed in gate at the same time
12:14:43 frickler but it seems all the others had already finished their rechecks
12:15:03 fungi zuul will auto-enqueue dependent changes if they meet its requirements
12:25:59 zhhuabj sean-k-moonkey: do you mean this one - https://paste.openstack.org/show/bd510w2dMMZdwOOjEvS7/
12:35:02 sean-k-mooney kindof depending on which parmater had the untracked changes
12:36:23 opendevreview Takashi Kajinami proposed openstack/os-vif master: Remove old excludes https://review.opendev.org/c/openstack/os-vif/+/917611
12:52:28 opendevreview Zhang Hua proposed openstack/nova master: BlockDeviceMapping object has no attribute copy https://review.opendev.org/c/openstack/nova/+/920374
12:54:44 zhhuabj hi sean-k-mooney , I posted a new change to address your comment - https://review.opendev.org/c/openstack/nova/+/920374/3/nova/tests/unit/virt/libvirt/test_blockinfo.py
12:56:24 opendevreview Merged openstack/nova stable/2023.2: Fix disk_formats in ceph job tempest config https://review.opendev.org/c/openstack/nova/+/923343
12:58:05 opendevreview Sahid Orentino Ferdjaoui proposed openstack/nova master: scheduler: fix _get_sharing_providers to support unlimited aggr https://review.opendev.org/c/openstack/nova/+/921665
13:04:18 sean-k-mooney zhhuabj: thansk we are currently focusing on upstream and downstream tasks related to the cve discustion yesterdaty but we will loop back and review when we have time
13:10:17 opendevreview Andrei Yachmenev proposed openstack/nova master: Fix processing uniqueness of instance host names https://review.opendev.org/c/openstack/nova/+/923395
13:23:22 dansmith head nova patch is going to fail openstacksdk again
13:25:18 dansmith same autoallocate thing it appears :(
13:28:07 sean-k-mooney that has been flaky for a while it seams do we want to make it non voting until the cve patches are laneded or just recheck
13:28:22 dansmith idk
13:28:43 sean-k-mooney we could quickly update it with the flaky decorator
13:28:52 sean-k-mooney that will skip if it fails and run it otherwise
13:29:10 sean-k-mooney but the sdk team would need to be aware they need to fix it
13:33:24 frickler there's a fix already proposed for the sdk job https://review.opendev.org/c/openstack/openstacksdk/+/923379
13:36:44 frickler I guess we should fast-approve that and fix stephenfin's comments in a followup?
13:37:19 sean-k-mooney stephenfin: ^
13:37:23 sean-k-mooney yes i think so
13:41:53 frickler done and promoted to top of gate. I think we should also dequeue and re-enqueue the nova stack?
13:42:30 frickler oh, the promotion took care of that already
13:46:55 bauzas sorry my IRC bouncer has some problems
13:47:02 bauzas frickler: where are we now ?
13:48:45 frickler bauzas: the nova stack was failing again due to an issue in the sdk job. this is to be fixed by https://review.opendev.org/c/openstack/openstacksdk/+/923379 which I've now promoted to the head of the gate pipeline
13:49:23 bauzas ack
13:49:55 frickler also https://meetings.opendev.org/irclogs/%23openstack-nova/latest.log.html is always there to help you, but updated only every 15 minutes
13:59:31 fungi if you're in a hurry and don't want to wait for the update, the less fancy txt version should be more continuously updated too
14:00:33 fungi the opendevmeet logbot buffers messages somewhat but streams them more quickly to the txt file, then a cronjob periodically generates the html version from that
14:03:24 fungi which is the reason for the apparent delay if you're only looking at the html version of the file
14:05:42 fungi for example https://meetings.opendev.org/irclogs/%23openstack-nova/%23openstack-nova.2024-07-03.log (we don't have a "latest" redirector or any direct links for those, but could probably add them without too much work)
14:18:24 frickler sdk patch got updated and gate restarted. I'll leave it like that unless the sdk job fails again in the top nova patch (doesn't seem to be happening 100%)
14:19:50 ygk_12345 hi folks
14:20:31 ygk_12345 can anyone tell me why we need to check resource limits when executing a qemu command in nova-compute using oslo prlimit utility ?
14:24:27 fungi ygk_12345: because qemu is not designed for running untrusted images, and broken or malicious images can cause it to consume unlimited resources (among other things)
14:25:24 ygk_12345 fungi: u mean something like ddos attacks
14:31:24 fungi ygk_12345: well, not ddos (the first d in ddos means "distributed")
14:31:45 fungi this would just be a plain old denial of service
14:34:26 fungi ygk_12345: though qemu tools can be coerced into doing worse things than just that, which is why we try to deeply inspect image files. see https://security.openstack.org/ossa/OSSA-2024-001.html and https://bugzilla.redhat.com/show_bug.cgi?id=2278875 for examples
14:38:32 ygk_12345 fungi: is this a new addition in bobcat release or is it in earlier versions as well ?
14:39:56 fungi ygk_12345: the openstack security advisory links to patches for releases as far back as 2023.1 (antelope), but there are also patches in the linked bug that make it possible to backport as far back as victoria (with some additional effort)
14:40:19 fungi or possibly even train
14:42:44 ygk_12345 fungi: one question though. the prlimit code, is it included in oslo_concurrency tool long back before this vulnerability was reported ? Also, do our antelope setups also have to be patched ?
14:45:03 fungi the prlimit work was done previously, yes, in order to mitigate other known resource consumption risks in qemu. i was mentioning yesterday's openstack security advisory and that qemu bug as examples of other risks

Earlier   Later