| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-07-25 | |||
| 16:00:52 | gmann | o/ | |
| 16:01:01 | auniyal | o/ | |
| 16:01:22 | elodilles | o/ | |
| 16:01:24 | bauzas | ok, starting | |
| 16:01:34 | bauzas | if someone wants to continue discussing, I can pass the chair | |
| 16:01:40 | bauzas | #topic Bugs (stuck/critical) | |
| 16:01:44 | bauzas | #info No Critical bug | |
| 16:01:48 | bauzas | #link https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New 40 new untriaged bugs (+0 since the last meeting) | |
| 16:01:58 | bauzas | Uggla made a good effort on triaging this week, thanks | |
| 16:02:11 | bauzas | he shared to me https://etherpad.opendev.org/p/nova-bug-triage-20230725 | |
| 16:02:27 | bauzas | #info Add yourself in the team bug roster if you want to help https://etherpad.opendev.org/p/nova-bug-triage-roster | |
| 16:02:39 | bauzas | elodilles: can you be the baton owner for this week ? | |
| 16:02:44 | elodilles | yepp | |
| 16:02:49 | bauzas | cool thanks ! | |
| 16:02:53 | bauzas | #info bug baton is being passed to elodilles | |
| 16:02:57 | bauzas | any bug to discuss ? | |
| 16:02:58 | elodilles | i'll be off on Friday, but otherwise fine! | |
| 16:03:09 | bauzas | elodilles: I'll be off next week so :) | |
| 16:03:18 | elodilles | touche ;) | |
| 16:03:18 | bauzas | #topic Gate status | |
| 16:03:22 | bauzas | #link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs | |
| 16:03:27 | bauzas | #link https://zuul.openstack.org/builds?project=openstack%2Fnova&project=openstack%2Fplacement&pipeline=periodic-weekly Nova&Placement periodic jobs status | |
| 16:03:31 | bauzas | #info nova-emulation back to work | |
| 16:03:35 | bauzas | #link https://zuul.openstack.org/build/2eb47b5dcb944b7eb03224e6ee599a57 | |
| 16:03:41 | dansmith | gmann and I have been working on a bunch of patches to address job timeout issues | |
| 16:03:46 | bauzas | now we have a failure on tempest-integrated-placement | |
| 16:03:54 | dansmith | still lots in flight, but we're hopeful that it will improve those issues at least | |
| 16:03:56 | bauzas | but we'll see next week | |
| 16:04:01 | dansmith | but there are still lots of other spurious fails | |
| 16:04:10 | bauzas | dansmith: I was about to mention the other pipelines but the periodic-weekly | |
| 16:04:21 | bauzas | #info Please look at the gate failures and file a bug report with the gate-failure tag. | |
| 16:04:26 | dansmith | really need all hands on deck to improve this situation | |
| 16:04:32 | gmann | yeah, timeout now a days is due to multiple reasons and we are trying to improve a few of them | |
| 16:04:33 | bauzas | dansmith: so, yeah, there are a bunch of patches in flight | |
| 16:05:03 | bauzas | dansmith: gmann: I don't know if you have seen my pings today but yeah, there is another suspect | |
| 16:05:22 | bauzas | #link https://bugs.launchpad.net/tempest/+bug/2024859 | |
| 16:05:26 | dansmith | bauzas: the ping was about the mkfs issue? | |
| 16:05:33 | dansmith | that's known and unclear how to resolve, AFAICT | |
| 16:05:34 | bauzas | yup | |
| 16:05:35 | gmann | bauzas: I saw that, I did not look into that expect melwitt trying with some change | |
| 16:05:42 | gmann | yeah mkfs | |
| 16:05:50 | dansmith | but I'm highly concerned that we've got a group of workers that is suddenly very very IO constrained | |
| 16:06:03 | bauzas | sean-k-mooney has a WIP proposal | |
| 16:06:04 | dansmith | because we're seeing 100% slowdown on some nodes that seems IO-related, like that mkfs issue | |
| 16:06:18 | dansmith | melwitt had a proposal but it turned out not to help, last I checked | |
| 16:06:24 | gmann | yeah | |
| 16:06:27 | sean-k-mooney | the nova-lvm job passed with it btu i dont know if it was failing consitently before | |
| 16:06:28 | bauzas | about using disk cachinh | |
| 16:06:39 | bauzas | but I'm torn enabling it | |
| 16:06:49 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/889383 | |
| 16:07:00 | bauzas | anyway, I don't disagree, we need to continue digging into it | |
| 16:07:01 | sean-k-mooney | well its partly enabled already | |
| 16:07:04 | gmann | sean-k-mooney: might not be so consistent but I see this 2-3 times in a week | |
| 16:07:30 | bauzas | dansmith: gmann: from the most remaining issues, is nova just a coal canary or is responsible for some of them ? | |
| 16:07:31 | sean-k-mooney | i was suggeting setting disk_cachemode: "file=none,block=unsafe,network=writeback" | |
| 16:07:43 | sean-k-mooney | block could also be writeback | |
| 16:08:22 | dansmith | bauzas: need more triage to know really | |
| 16:08:30 | dansmith | bauzas: I've been focusing on the timeout stuff for a week | |
| 16:08:47 | gmann | bauzas: timeout is more of test runner unbalancing on test worker and slow test etc etc | |
| 16:08:51 | bauzas | ok, I'll try to get more hands on it before I leave | |
| 16:09:08 | gmann | because timeout hold many of the gate fixes so that should be fixed first :) | |
| 16:09:18 | bauzas | ack | |
| 16:09:31 | bauzas | moving on | |
| 16:09:33 | bauzas | #topic Release Planning | |
| 16:09:37 | bauzas | #link https://releases.openstack.org/bobcat/schedule.html | |
| 16:09:41 | bauzas | #link https://etherpad.opendev.org/p/nova-bobcat-blueprint-status Etherpad for tracking blueprints status | |
| 16:09:45 | bauzas | #info 5 weeks before FeatureFreeze | |
| 16:10:00 | bauzas | as I said before, my attention will go to features sets this week | |
| 16:10:29 | bauzas | in theory, today was a Feature Review Day | |
| 16:10:40 | bauzas | but I feel many people having different priorities | |
| 16:10:50 | bauzas | so I haven't really called it | |
| 16:11:14 | bauzas | #topic Review priorities | |
| 16:11:18 | bauzas | #link https://review.opendev.org/q/status:open+(project:openstack/nova+OR+project:openstack/placement+OR+project:openstack/os-traits+OR+project:openstack/os-resource-classes+OR+project:openstack/os-vif+OR+project:openstack/python-novaclient+OR+project:openstack/osc-placement)+(label:Review-Priority%252B1+OR+label:Review-Priority%252B2) | |
| 16:11:22 | bauzas | #info As a reminder, people eager to review changes can +1 to indicate their interest, +2 for asking cores to also review | |
| 16:11:26 | bauzas | #topic Stable Branches | |
| 16:11:31 | bauzas | elodilles: your time | |
| 16:11:35 | elodilles | #info nova-lvm / nova-ceph-multistore jobs are fixed on all branches (2023.1, zed, yoga) \o/ | |
| 16:11:38 | elodilles | #info stable/victoria gate fix has also landed | |
| 16:11:40 | elodilles | #info gates from 2023.1 back till train should be OK | |
| 16:11:47 | elodilles | #info stable branch status / gate failures tracking etherpad: https://etherpad.opendev.org/p/nova-stable-branch-ci | |
| 16:11:52 | elodilles | EOM | |
| 16:11:55 | bauzas | thanks | |
| 16:12:02 | bauzas | #info train-eol patch proposed https://review.opendev.org/c/openstack/releases/+/885365 | |
| 16:12:09 | bauzas | we're missing a second release core vote | |
| 16:12:14 | bauzas | but should happen eventually | |
| 16:12:20 | elodilles | yepp | |
| 16:12:32 | bauzas | auniyal had a point, I'll rephrase it | |
| 16:12:44 | bauzas | #info Please review these backport patches of stable 2023.1, zed and yoga for next minor release | |
| 16:12:49 | bauzas | #info most of these already have one +2. | |
| 16:12:52 | bauzas | #link https://etherpad.opendev.org/p/release-liaison-PatchesToReview | |
| 16:13:09 | bauzas | and the stable branches are now back in healthy state | |
| 16:13:18 | bauzas | #topic Open discussion | |
| 16:13:26 | bauzas | I have one item | |
| 16:14:17 | bauzas | as I said previously, I'll be on PTO starting next Tues | |
| 16:14:31 | bauzas | gibi volunteered for running the Aug 15 meeting | |
| 16:14:38 | bauzas | (and Aug 22) | |
| 16:14:49 | bauzas | but we may need someone to chair the Aug 1 and Aug 8 meetings | |
| 16:14:57 | bauzas | or we skip them | |
| 16:15:40 | gibi | jepp I still OK with the 15 and 22 | |
| 16:15:51 | bauzas | so, anyone fancy volunteering ? if not, nevermind, I'll send an email to cancel the two weeks | |