| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-07-15 | |||
| 12:16:19 | opendevreview | sean mooney proposed openstack/nova master: add detailed logging to get_updated_guest_xml https://review.opendev.org/c/openstack/nova/+/955056 | |
| 13:07:03 | opendevreview | Balazs Gibizer proposed openstack/nova master: DNM: Force librbd1 19.2.0 to fix ceph job https://review.opendev.org/c/openstack/nova/+/954956 | |
| 15:29:37 | bauzas | nova meeting in 30 mins now | |
| 16:00:25 | opendevmeet | The meeting name has been set to 'nova' | |
| 16:00:25 | opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | |
| 16:00:25 | opendevmeet | Meeting started Tue Jul 15 16:00:25 2025 UTC and is due to finish in 60 minutes. The chair is bauzas. Information about MeetBot at http://wiki.debian.org/MeetBot. | |
| 16:00:25 | bauzas | #startmeeting nova | |
| 16:00:46 | bauzas | #link https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting | |
| 16:01:05 | bauzas | howdy folks, I'll be your humble chair for today's meeting | |
| 16:01:08 | bauzas | who's around ? | |
| 16:01:18 | elodilles | o/ | |
| 16:01:31 | fwiesel | o/ | |
| 16:01:36 | ratailor_ | o/ | |
| 16:01:56 | gmaan | o/ | |
| 16:02:30 | gibi | o/ | |
| 16:03:00 | bauzas | okay, let's start | |
| 16:03:08 | bauzas | #topic Bugs (stuck/critical) | |
| 16:03:10 | bauzas | doh | |
| 16:03:15 | bauzas | #topic Bugs (stuck/critical) | |
| 16:03:40 | bauzas | #info One Critical bug | |
| 16:03:42 | gibi | https://bugs.launchpad.net/nova/+bug/2116852 | |
| 16:03:46 | gibi | nova-ceph-multistore job fails constantly with memory allocation error Edit | |
| 16:03:47 | bauzas | #link https://bugs.launchpad.net/nova/+bug/2116852 | |
| 16:03:54 | bauzas | yeah | |
| 16:03:58 | bauzas | just saw that | |
| 16:04:06 | bauzas | any proposed patch ? | |
| 16:04:09 | gibi | patch to sidable the single tempest test to unblock the gate: https://review.opendev.org/c/openstack/tempest/+/954949 | |
| 16:04:21 | bauzas | ack | |
| 16:04:29 | gibi | patch that proves that it is caused by ceph package update https://review.opendev.org/c/openstack/nova/+/954956 | |
| 16:04:40 | gmaan | did recheck again this morning. I think gerrit downtime did not work last night | |
| 16:04:40 | gibi | with downgraded packaged the test is green | |
| 16:04:49 | bauzas | thanks gmaan | |
| 16:05:16 | gibi | I'm trying to report a bug to ceph upstream but I does not seem to be able to register in their tracker successfully | |
| 16:05:58 | gmaan | gibi: nice, you want to go with your nova change with downgraded package or tempest change to skip test? | |
| 16:06:15 | gibi | I suggest to skip the test until we get some feedback from the ceph folks | |
| 16:06:24 | gibi | I don't like forcing packages manually | |
| 16:06:28 | gmaan | ok, make sense | |
| 16:06:31 | bauzas | cool | |
| 16:06:42 | gibi | does anybody have active acc to the ceph bug tracker? | |
| 16:06:48 | gibi | https://tracker.ceph.com/login | |
| 16:07:00 | gibi | mine is blocked on "Your account was created and is now pending administrator approval. | |
| 16:07:20 | gibi | (and yes they disabled the github issues in the repo) | |
| 16:07:47 | gmaan | not me, I have not created yet | |
| 16:08:16 | gibi | anyhow lets see what happens. If I cannot report a bug upstream I will use my RH powers downstream | |
| 16:08:27 | gibi | we can move on :) | |
| 16:08:33 | bauzas | ack | |
| 16:09:00 | bauzas | #topic Gate status | |
| 16:09:07 | bauzas | #link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs | |
| 16:09:11 | gibi | we rae blocked by the above bug | |
| 16:09:17 | gibi | as nova-ceph-multistore is red | |
| 16:09:25 | bauzas | #link https://zuul.openstack.org/builds?project=openstack%2Fnova&project=openstack%2Fplacement&branch=stable%2F*&branch=master&pipeline=periodic-weekly&skip=0 Nova&Placement periodic jobs status | |
| 16:09:31 | bauzas | yeah | |
| 16:09:38 | bauzas | and the periodics are in a bad shape too | |
| 16:10:26 | gibi | centos jobs failing with | |
| 16:10:27 | gibi | ./stack.sh: line 504: /opt/stack/devstack/tools/outfilter.py: Permission denied | |
| 16:10:51 | gibi | but I did not dig deeper | |
| 16:10:51 | gmaan | latest run seems passing | |
| 16:10:53 | gmaan | #link https://zuul.opendev.org/t/openstack/builds?job_name=tempest-centos9-stream-fips&skip=0 | |
| 16:11:27 | gmaan | tempest run them periodic nightly and at least last two run pass | |
| 16:11:44 | gibi | the fip one seems to be different | |
| 16:11:47 | gmaan | though did not dig why outfilter.py: Permission denied error | |
| 16:12:03 | gibi | https://zuul.openstack.org/builds?job_name=tempest-integrated-compute-centos-9-stream&skip=0 | |
| 16:12:09 | gibi | this fails with the outfilter | |
| 16:12:40 | gmaan | ah another c9s job | |
| 16:12:50 | bauzas | all c9s jobs seem impacted | |
| 16:13:13 | gmaan | its same error in fips job also | |
| 16:13:15 | gmaan | #link https://zuul.opendev.org/t/openstack/build/e8f309cd649b4f0ca38ab7777e99de9f | |
| 16:13:23 | gmaan | which passed in recent run | |
| 16:14:27 | elodilles | it seems to be passing on master but not on stable branches | |
| 16:15:35 | gmaan | last run on stable was 2025-07-12 and we are seeing passing on master after that | |
| 16:15:59 | gmaan | maybe we can run it manually if its there in experimental pipeline or wait for next run | |
| 16:17:26 | gmaan | it seems they are not in experimental queue https://github.com/openstack/nova/blob/stable/2025.1/.zuul.yaml#L874 | |
| 16:17:57 | gmaan | sorry I found it | |
| 16:17:59 | gmaan | #link https://github.com/openstack/nova/blob/stable/2025.1/.zuul.yaml#L899 | |
| 16:18:38 | gmaan | checked experimental in this #link https://review.opendev.org/c/openstack/nova/+/952960 | |
| 16:19:20 | bauzas | cool | |
| 16:19:22 | bauzas | moving on ? | |
| 16:19:25 | gibi | one more | |
| 16:19:28 | bauzas | k | |
| 16:19:59 | gibi | https://bugs.launchpad.net/grenade/+bug/2116336 this happens semi frequently but it is annoying as it makes the zuul feedback very slow | |
| 16:20:12 | gibi | grenade times out on dumping DBs | |
| 16:20:24 | gibi | so I proposed https://review.opendev.org/c/openstack/grenade/+/954629 to prevent looong timeout | |
| 16:20:35 | gibi | but I don't have any idea why it hangs | |
| 16:20:41 | gibi | I found nothing in the logs | |
| 16:21:33 | gibi | if somebody has ideas then please take a look at the bug | |
| 16:21:40 | bauzas | what a lovely week | |
| 16:22:23 | gibi | (I have no more) | |
| 16:22:45 | bauzas | this is fine (c) dog | |
| 16:22:55 | bauzas | okay, moving on | |
| 16:23:59 | bauzas | #topic tempest-with-latest-microversion job status | |
| 16:24:06 | bauzas | #link https://zuul.opendev.org/t/openstack/builds?job_name=tempest-with-latest-microversion&skip=0 | |
| 16:24:18 | bauzas | red red red red | |
| 16:24:22 | gmaan | no updates on this. did not get chance to work last week | |
| 16:24:30 | bauzas | no worries | |
| 16:24:33 | bauzas | moving on | |
| 16:24:45 | bauzas | #topic Release Planning | |
| 16:24:51 | bauzas | #link https://releases.openstack.org/flamingo/schedule.html | |
| 16:24:53 | gmaan | still 21 more tetts to fix | |
| 16:24:59 | bauzas | #info Nova deadlines are set in the above schedule | |
| 16:25:29 | bauzas | we have around 7 weeks before ff, if I recount correctly | |