| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-07-02 | |||
| 16:46:26 | sean-k-mooney | i have seen that fail on othe rpatches and it seams to just be flaky | |
| 16:46:47 | sean-k-mooney | so for now ill leave the curernt set complete | |
| 16:47:25 | sean-k-mooney | it will get rerun once i rebase the patches for the ami/aki/ari fix anyway | |
| 16:47:46 | dansmith | ack | |
| 16:48:31 | sean-k-mooney | im assumign that merged reletivly recently because i have only started to see it fail like that in the last 2-3 weeks | |
| 16:48:41 | sean-k-mooney | i dont recall that job failing previously to that | |
| 16:50:43 | sean-k-mooney | hum no apprently that was added 8 years ago | |
| 16:51:05 | sean-k-mooney | that makes me feel old given the autoallocate stuff was added to enable people to move form nova-networks... | |
| 16:51:53 | sean-k-mooney | ofcouse everythign pre covide feels ancient at this point | |
| 17:40:53 | dansmith | sean-k-mooney: going to need a very quick approval on a job change in nova's ceph job to unblock glance in a sec | |
| 17:45:50 | opendevreview | Dan Smith proposed openstack/nova master: Fix disk_formats in ceph job tempest config https://review.opendev.org/c/openstack/nova/+/923322 | |
| 17:46:03 | dansmith | sean-k-mooney: ^ | |
| 17:48:38 | dansmith | or any other nova cores | |
| 17:49:16 | dansmith | frickler: that one will take a long time in the current queue to merge and glance can't merge their stuff without it, so if you have thoughts... :) | |
| 17:52:15 | gmann | dansmith: one question, this is tempest default, that is what you mean in commit msg https://github.com/openstack/tempest/blob/6b8ff89c8fffcc231d9e5b639d57d6e4a8bd47d3/tempest/config.py#L657 | |
| 17:52:30 | dansmith | gmann: yes | |
| 17:52:32 | dansmith | abhishek_: ^ | |
| 17:52:50 | dansmith | gmann: this is basically the problem: https://github.com/openstack/tempest/blob/master/tempest/api/image/base.py#L103 | |
| 17:52:50 | abhishek_ | yes | |
| 17:56:06 | gmann | dansmith: ohk, +2 but it will be good if we can also fix test to fetch the required image instead of replying on list order in config option | |
| 17:56:34 | gmann | *relying | |
| 17:56:39 | dansmith | replied agree of course | |
| 17:57:03 | gmann | dansmith: thanks | |
| 17:57:38 | abhishek_ | gmann: agree to fix the test as well | |
| 17:57:54 | gmann | +1 | |
| 18:17:38 | frickler | dansmith: the only chance to speed up the CI in the current situation likely would be to temporarily abandon all the stable branch changes that eat up resources but fill anyway, and restore them once all fixes have merged in master | |
| 18:17:53 | dansmith | seems like a bad plan | |
| 18:18:00 | frickler | s/fill/will fail/ | |
| 18:18:32 | frickler | other then that, just wait until zuul has worked through the queue | |
| 18:18:41 | dansmith | roger that | |
| 18:20:16 | JayF | sean-k-mooney: https://github.com/eventlet/eventlet/pull/970/files is the fix for that eventlet "couldn't green locks" issue we saw popup in python 3.12. I'd expect it to merge+be in next eventlet release. TL;DR: eventlet monkey-patching was happening after we greened locks + as a side effect of running the lock greening, we'd create more locks to be greened AIUI | |
| 18:38:43 | sean-k-mooney | JayF: ack ill try and take a look | |
| 18:39:02 | JayF | sean-k-mooney: consider it just an FYI and notice that I made sure it was taken care of | |
| 18:39:06 | JayF | not sure you have an AI | |
| 18:40:11 | sean-k-mooney | JayF: thanks it more i was on calls for the last hour or so and i want to check back in on the status fo the cve | |
| 18:40:26 | sean-k-mooney | so ill read over that tomorrow | |
| 18:40:33 | JayF | yeah that is more important, only reason I'm letting you know is to get it off your list if it was ever on it | |
| 18:40:47 | sean-k-mooney | :) | |
| 19:10:49 | fungi | with regard to speeding up fixes, we *can* enqueue the more urgent changes directly into the gate pipeline without waiting for them to clear check. the gate pipeline automatically gets resource priority over the check pipeline | |
| 19:11:20 | dansmith | fungi: up to you really... we have the nova and glance patches that both need to go | |
| 19:11:24 | fungi | once they're in the gate, we can also reorder the queue to put them ahead of the rest | |
| 19:11:46 | dansmith | they're just tempest/zuul changes and gmann is +2, so... might be good | |
| 19:12:25 | fungi | they'll need workflow +1 votes before i can enqueue them, but they'll still have to pass the same tests in the gate before they can merge anyway so that should be safe | |
| 19:12:48 | dansmith | the nova one should be good | |
| 19:13:08 | fungi | i'm juggling a lot of things, so if someone can explicitly list the more urgent change numbers i need to move to the gate pipeline i'm happy to do that as long as they've gotten approval votes from someone | |
| 19:13:10 | dansmith | https://review.opendev.org/c/openstack/glance/+/923321 | |
| 19:13:12 | dansmith | and yeah the glance one too | |
| 19:13:13 | fungi | thanks | |
| 19:13:43 | dansmith | nova one is https://review.opendev.org/c/openstack/nova/+/923322 in case you didn't et it | |
| 19:14:46 | fungi | yep, i'll enqueue those momentarily | |
| 19:19:40 | fungi | they're both at the top of the gate pipeline now | |
| 19:20:21 | dansmith | thanks fungi | |
| 19:22:05 | fungi | and already getting nodes assigned | |
| 19:40:51 | dansmith | fungi: we may have spoken too soon | |
| 19:40:53 | dansmith | abhishek_: ^ | |
| 19:41:08 | dansmith | looks like that small change had some knock-on effects in tempest I don't fully understand | |
| 19:41:46 | dansmith | gmann: might could use some help | |
| 19:41:55 | fungi | i'm around to re-enqueue a revision if it's worked out | |
| 19:42:30 | dansmith | gmann: does image.disk_formats look right here? https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_991/923321/2/check/tempest-integrated-storage-import/991d0c2/controller/logs/tempest_conf.txt | |
| 19:43:58 | dansmith | looks like the list syntax is not parsed and we grab a chunk of it | |
| 19:43:58 | dansmith | https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_991/923321/2/check/tempest-integrated-storage-import/991d0c2/testr_results.html | |
| 19:44:04 | dansmith | maybe the inner quotes are not needed? | |
| 19:44:07 | dansmith | abhishek_: ^ | |
| 19:44:41 | dansmith | I bet that's the case and oslo.config is splitting on comma only | |
| 19:44:46 | dansmith | so it gets everything up to the first comma | |
| 19:44:57 | abhishek_ | ok | |
| 19:45:12 | abhishek_ | so without inner '' quotes? | |
| 19:45:48 | dansmith | so maybe should be: | |
| 19:45:49 | dansmith | disk_formats: qcow2,ari,aki,vhd,vmdk,raw,ami,vdi,iso,vhdx | |
| 19:45:50 | dansmith | ? | |
| 19:47:04 | dansmith | tbh I dunno because the example above that one has inner and outer quotes | |
| 19:47:13 | abhishek_ | not sure how it is working for image_import_plugins: "['inject_image_metadata', 'image_conversion']" | |
| 19:47:17 | dansmith | but it sure looks to me like it needs no quotes at all | |
| 19:47:18 | dansmith | right | |
| 19:48:06 | abhishek_ | enabled_import_methods: "[\"copy-image\", \"glance-direct\"]" | |
| 19:48:12 | abhishek_ | or should do something like this? | |
| 19:48:48 | dansmith | idk | |
| 19:50:44 | abhishek_ | available_type_drivers: flat,geneve,vlan,gre,local,vxlan | |
| 19:50:56 | abhishek_ | I see somewhwere this | |
| 19:51:13 | abhishek_ | in neutron jobs | |
| 19:51:37 | dansmith | ack, I think what matters is how the tempest.conf is read, but I dunno | |
| 19:51:49 | dansmith | might as well change it I guess, it's not going to work as-is | |
| 19:52:20 | dansmith | oh yeah, | |
| 19:52:30 | dansmith | I see in tempest config for neutron | |
| 19:52:36 | dansmith | I think that's right | |
| 19:53:08 | opendevreview | Dan Smith proposed openstack/nova master: Fix disk_formats in ceph job tempest config https://review.opendev.org/c/openstack/nova/+/923322 | |
| 19:53:14 | abhishek_ | hmm, ket me do that | |
| 19:53:40 | fungi | lmk as soon as it has an approval vote and i can reenqueue it into the gate | |
| 19:54:53 | dansmith | I guess this is why we don't (normally) skip check | |
| 19:55:37 | abhishek_ | ++ | |
| 19:56:16 | fungi | heh | |
| 19:56:34 | fungi | we're not normally in quite so much of a hurry ;) | |
| 19:58:04 | dansmith | I'm want to get off the merry-go-round | |
| 19:59:16 | fungi | if we get it spinning fast enough, you can take advantage of centripetal force to make a speedy exit | |
| 19:59:51 | fungi | you'll probably land somewhere over by the cotton candy booth | |
| 20:01:35 | sean-k-mooney | given the week in it i expect a lot of people to dissapre after tomorrow one way or another | |
| 20:02:53 | dansmith | fungi: both of those are fixed (we think) and re-approved.. up to you if you want to take another gamble on the roulette wheel | |
| 20:03:49 | dansmith | sean-k-mooney: looks like unless ceph fails, everything passed except for that live migration job, which is fairly effing amazing | |
| 20:04:00 | dansmith | actually ceph already passed on the second one | |
| 20:04:00 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/923317/1 | |