| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-07-02 | |||
| 20:23:51 | sean-k-mooney | that job has a 2 hour timeout so it got very close | |
| 20:26:01 | sean-k-mooney | the last job on the 3rd patch is almost done too | |
| 20:28:45 | sean-k-mooney | ok thats done too | |
| 20:29:21 | sean-k-mooney | ill wait aorund for another 5 minutes otherwise ill check back in an hour or 2 before i go to sleep | |
| 20:29:32 | opendevreview | Dan Smith proposed openstack/nova master: Reject qcow files with data-file attributes https://review.opendev.org/c/openstack/nova/+/923255 | |
| 20:29:32 | opendevreview | Dan Smith proposed openstack/nova master: Check images with format_inspector for safety https://review.opendev.org/c/openstack/nova/+/923256 | |
| 20:29:33 | opendevreview | Dan Smith proposed openstack/nova master: Additional qemu safety checking on base images https://review.opendev.org/c/openstack/nova/+/923257 | |
| 20:29:33 | opendevreview | Dan Smith proposed openstack/nova master: Fix vmdk_allowed_types checking https://review.opendev.org/c/openstack/nova/+/923258 | |
| 20:30:08 | sean-k-mooney | im not sure you ment to update the first patch.. | |
| 20:30:33 | dansmith | oh crap, yeah I rebased my origin master chasing the other fire | |
| 20:30:37 | dansmith | oh well, that sucks | |
| 20:30:58 | dansmith | it was rechecking anyway | |
| 20:31:50 | sean-k-mooney | i +2w'd the set | |
| 20:31:56 | dansmith | thanks much | |
| 20:32:12 | dansmith | I'm just about exhausted and fading pretty fast here | |
| 20:32:21 | sean-k-mooney | ok im going to go eat. ill check back in an hour or two before i go to sleep and ill check them in the morning if they have any issues | |
| 20:32:43 | sean-k-mooney | dansmith: go rest | |
| 20:32:43 | dansmith | I have one other thing to do and then I will probably need to take a break for a bit | |
| 20:33:04 | sean-k-mooney | well theere in the ci so not much more for you do do on the nova side | |
| 20:33:27 | sean-k-mooney | unless your plannig to respin the backports but we can do that once they start to merge | |
| 20:34:11 | dansmith | I'm going to hold off on those until we make progress yeah | |
| 22:08:37 | fungi | argh, one job timed out on the glance change | |
| 22:09:00 | fungi | during volume tests, though that may have been coincidence | |
| 22:10:11 | fungi | though the one remaining job on the nova change may be about to do the same | |
| 22:14:15 | fungi | they're both starting over at the top of the gate pipeline again | |
| #openstack-nova - 2024-07-03 | |||
| 00:19:15 | fungi | yeah, seems like tempest-integrated-compute-enforce-scope-new-defaults is repeatedly timing out (during volume tests but those aren't necessarily what's taking so long) for both the glance and nova changes previously discussed | |
| 00:19:42 | fungi | though the glance one might barely finish just under the wire this time around | |
| 00:20:08 | fungi | once it (hopefully) merges, i'll reenqueue and promote the nova change again | |
| 00:25:11 | fungi | okay, glance change landed, nova change is back to the top of the gate for a third try | |
| 00:48:36 | dansmith | fungi++ | |
| 00:54:54 | dansmith | fungi: that new defaults job has been timeout-heavy lately for sure, I think sean had a thing to bump the timeout, which I suppose we might have to do.. | |
| 01:55:19 | opendevreview | Dan Smith proposed openstack/nova stable/2024.1: Fix disk_formats in ceph job tempest config https://review.opendev.org/c/openstack/nova/+/923342 | |
| 01:55:31 | opendevreview | Dan Smith proposed openstack/nova stable/2023.2: Fix disk_formats in ceph job tempest config https://review.opendev.org/c/openstack/nova/+/923343 | |
| 01:55:43 | opendevreview | Dan Smith proposed openstack/nova stable/2023.1: Fix disk_formats in ceph job tempest config https://review.opendev.org/c/openstack/nova/+/923344 | |
| 01:55:44 | dansmith | abhishek_: ^ | |
| 01:56:24 | dansmith | melwitt: if you're around by chance and could monitor those ^ we need them in ASAP (master still pending) so we can get glance CVE patches merged that depend on them | |
| 01:56:41 | abhishek_ | dansmith: ++ | |
| 02:12:11 | melwitt | dansmith: ok, I can do it | |
| 02:14:14 | dansmith | melwitt: thanks, master has been rechecked several times already, so who knows | |
| 02:16:51 | melwitt | yeah. we can at least get some more dice rolls in before EU timezone | |
| 02:21:16 | abhishek_ | is there someone in this timezone who can prioritise the patch in gate (just in case if required) | |
| 02:21:29 | dansmith | melwitt: ++ | |
| 02:21:46 | dansmith | abhishek_: I'm not sure, frickler will be coming online, might be the earliest | |
| 02:21:58 | abhishek_ | ack | |
| 02:22:00 | dansmith | I think he can do it, but not sure | |
| 02:22:15 | abhishek_ | will ask him if required, thank you! | |
| 02:37:38 | opendevreview | Merged openstack/nova master: Fix disk_formats in ceph job tempest config https://review.opendev.org/c/openstack/nova/+/923322 | |
| 02:37:46 | fungi | yay! | |
| 02:37:51 | dansmith | amazing | |
| 02:38:18 | dansmith | fungi: do we prioritize the stable ones too or no? | |
| 02:39:17 | fungi | might not be necessary this time of day if they already have passing check results and there's not much else in the gate, i haven't looked | |
| 02:41:53 | dansmith | ack | |
| 02:43:25 | fungi | but if it will help in getting them merged faster we still can | |
| 02:43:54 | fungi | just depends on whether there's much of a queue or jobs are continuing to be problematic | |
| 02:44:00 | dansmith | abhishek_: ^ dunno how important it is | |
| 02:44:52 | abhishek_ | dansmith: I think we can wait, since it will going to take a day long to merge master patches | |
| 02:44:59 | dansmith | ack | |
| 02:46:17 | abhishek_ | nova merged | |
| 05:13:29 | frickler | well the gate is currently looking ok-ish after https://review.opendev.org/923344 caused a gate reset. let's hope the current stack merges without further failures. I'll try to keep an eye on it and look into reshuffling if further issues should happen | |
| 06:00:12 | frickler | and there goes hoping ... weird early failure in grenade-multinode that I haven't seen before, but looks 99% unrelated https://zuul.opendev.org/t/openstack/build/322833ee6e7a43b0aee2ca9803fc4f57 | |
| 07:27:51 | bauzas | good morning folks | |
| 07:28:05 | bauzas | frickler: I was on PTO yesterday, so I just got the emails | |
| 07:28:16 | bauzas | but AFAICS, we needed to update Tempest rightN? | |
| 07:28:34 | bauzas | https://review.opendev.org/c/openstack/nova/+/923322 | |
| 07:28:51 | bauzas | I'm now chasing https://review.opendev.org/c/openstack/nova/+/923255/2 | |
| 07:32:28 | frickler | bauzas: the latter is failing in gate with the error I posted before, IMO it could be re-enqueued into gate immediately once zuul reports the failures, together with the stack on top of it | |
| 07:34:05 | frickler | I found keystone (once again) to be essentially undebuggable with the amount of errors generated during normal operations | |
| 07:34:58 | bauzas | yeah https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_322/923255/2/gate/nova-grenade-multinode/322833e/controller/logs/grenade.sh_log.txt | |
| 07:35:11 | bauzas | my only concern is that I'm not sure this is a transient issue | |
| 07:35:21 | bauzas | looks more a keystone regression to me | |
| 07:35:29 | bauzas | 2024-07-03 05:20:04.386 | testtools.matchers._impl.MismatchError: '56fc53e2e0724c0284ab80060ec55421' != '39aa3070a75e446fb6aa6d48bab74c58' | |
| 07:36:42 | bauzas | unless someone keystone-ed able to tell me this is something like a race condition in Keystone | |
| 08:00:43 | opendevreview | Merged openstack/nova stable/2024.1: Fix disk_formats in ceph job tempest config https://review.opendev.org/c/openstack/nova/+/923342 | |
| 08:14:53 | frickler | bauzas: if it is a regression it should fail in gate next time, too, but I don't see any recent changes in keystone. so I still suggest to reenqueue the stack into gate, unless your prefer to only recheck it | |
| 08:15:11 | bauzas | I'll recheck it | |
| 08:15:18 | bauzas | and we'll see | |
| 08:17:25 | bauzas | done | |
| 08:29:47 | opendevreview | Maxime Lubin proposed openstack/nova-specs master: USB over IP https://review.opendev.org/c/openstack/nova-specs/+/923362 | |
| 08:43:38 | opendevreview | Maxime Lubin proposed openstack/nova-specs master: USB over IP https://review.opendev.org/c/openstack/nova-specs/+/920687 | |
| 08:56:31 | opendevreview | Maxime Lubin proposed openstack/nova-specs master: USB over IP https://review.opendev.org/c/openstack/nova-specs/+/920687 | |
| 09:16:29 | opendevreview | Maxime Lubin proposed openstack/nova-specs master: USB over IP https://review.opendev.org/c/openstack/nova-specs/+/920687 | |
| 09:20:23 | Uggla | @gibi, can you have a look at this patch : https://review.opendev.org/c/openstack/nova/+/868089/8 can it be merged ? | |
| 10:07:23 | frickler | next failure, this time one of the "regulars" tempest.api.compute.servers.test_server_actions.ServerActionsV293TestJSON.test_rebuild_volume_backed_server | |
| 10:08:49 | bauzas | yeepeekay | |
| 10:43:57 | sean-k-mooney | frickler: bauzas i requeued the first patch but i kind fo feel like waiting for that to merge before queueign the rest | |
| 10:44:05 | sean-k-mooney | what do ye think | |
| 10:44:22 | sean-k-mooney | shoudl we hold all other nova patches until this lands and ask peopel not to recheck on the mailing list | |
| 10:45:03 | sean-k-mooney | the random failure are not nessisarly load related so im not really sure if it will help | |
| 10:45:48 | sean-k-mooney | but we are consumign a lot of ci resouces enquing all 4 patches just to have them kicked out because of a failure in the first one | |
| 11:15:44 | frickler | sean-k-mooney: on one hand I understand that concern, on the other hand these patches are being used in production in a lot of systems already, so IMO it should have highest priority to at least get them into master, even if other projects have to suffer a bit from that | |
| 11:16:54 | sean-k-mooney | ok so your in faovr of quein gthe rest. i was not sure if it would increase the likely hood fo a failure but ok ill recheck the remaining 3 and get them moving | |
| 11:17:40 | sean-k-mooney | actully im not sure the second one need a recheck it still has +1 | |
| 11:18:33 | sean-k-mooney | i have rechecked the third pathch | |
| 11:18:44 | frickler | as I said earlier I would also be in favor of moving these to gate directly | |
| 11:18:59 | frickler | as the failures pretty certainly look unrelated | |
| 11:19:18 | sean-k-mooney | im not agaisnt taht i just dont want to keep pinging fungi, you or clark ot do that | |
| 11:19:38 | zhhuabj | hi team, does anyone know why ci can't run in this patch - https://review.opendev.org/c/openstack/nova/+/909611 | |
| 11:19:49 | sean-k-mooney | oh ya they are not related any more. we had a fully green run last night on the last patch so the set as a whole i think is good | |
| 11:19:51 | frickler | well currently it is me who keeps pinging nova people with the offer ;) | |
| 11:20:32 | sean-k-mooney | i think we have got enouch results form check in agggreate to be ok with mergeing them as is at this point | |