| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-07-25 | |||
| 16:03:46 | bauzas | now we have a failure on tempest-integrated-placement | |
| 16:03:54 | dansmith | still lots in flight, but we're hopeful that it will improve those issues at least | |
| 16:03:56 | bauzas | but we'll see next week | |
| 16:04:01 | dansmith | but there are still lots of other spurious fails | |
| 16:04:10 | bauzas | dansmith: I was about to mention the other pipelines but the periodic-weekly | |
| 16:04:21 | bauzas | #info Please look at the gate failures and file a bug report with the gate-failure tag. | |
| 16:04:26 | dansmith | really need all hands on deck to improve this situation | |
| 16:04:32 | gmann | yeah, timeout now a days is due to multiple reasons and we are trying to improve a few of them | |
| 16:04:33 | bauzas | dansmith: so, yeah, there are a bunch of patches in flight | |
| 16:05:03 | bauzas | dansmith: gmann: I don't know if you have seen my pings today but yeah, there is another suspect | |
| 16:05:22 | bauzas | #link https://bugs.launchpad.net/tempest/+bug/2024859 | |
| 16:05:26 | dansmith | bauzas: the ping was about the mkfs issue? | |
| 16:05:33 | dansmith | that's known and unclear how to resolve, AFAICT | |
| 16:05:34 | bauzas | yup | |
| 16:05:35 | gmann | bauzas: I saw that, I did not look into that expect melwitt trying with some change | |
| 16:05:42 | gmann | yeah mkfs | |
| 16:05:50 | dansmith | but I'm highly concerned that we've got a group of workers that is suddenly very very IO constrained | |
| 16:06:03 | bauzas | sean-k-mooney has a WIP proposal | |
| 16:06:04 | dansmith | because we're seeing 100% slowdown on some nodes that seems IO-related, like that mkfs issue | |
| 16:06:18 | dansmith | melwitt had a proposal but it turned out not to help, last I checked | |
| 16:06:24 | gmann | yeah | |
| 16:06:27 | sean-k-mooney | the nova-lvm job passed with it btu i dont know if it was failing consitently before | |
| 16:06:28 | bauzas | about using disk cachinh | |
| 16:06:39 | bauzas | but I'm torn enabling it | |
| 16:06:49 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/889383 | |
| 16:07:00 | bauzas | anyway, I don't disagree, we need to continue digging into it | |
| 16:07:01 | sean-k-mooney | well its partly enabled already | |
| 16:07:04 | gmann | sean-k-mooney: might not be so consistent but I see this 2-3 times in a week | |
| 16:07:30 | bauzas | dansmith: gmann: from the most remaining issues, is nova just a coal canary or is responsible for some of them ? | |
| 16:07:31 | sean-k-mooney | i was suggeting setting disk_cachemode: "file=none,block=unsafe,network=writeback" | |
| 16:07:43 | sean-k-mooney | block could also be writeback | |
| 16:08:22 | dansmith | bauzas: need more triage to know really | |
| 16:08:30 | dansmith | bauzas: I've been focusing on the timeout stuff for a week | |
| 16:08:47 | gmann | bauzas: timeout is more of test runner unbalancing on test worker and slow test etc etc | |
| 16:08:51 | bauzas | ok, I'll try to get more hands on it before I leave | |
| 16:09:08 | gmann | because timeout hold many of the gate fixes so that should be fixed first :) | |
| 16:09:18 | bauzas | ack | |
| 16:09:31 | bauzas | moving on | |
| 16:09:33 | bauzas | #topic Release Planning | |
| 16:09:37 | bauzas | #link https://releases.openstack.org/bobcat/schedule.html | |
| 16:09:41 | bauzas | #link https://etherpad.opendev.org/p/nova-bobcat-blueprint-status Etherpad for tracking blueprints status | |
| 16:09:45 | bauzas | #info 5 weeks before FeatureFreeze | |
| 16:10:00 | bauzas | as I said before, my attention will go to features sets this week | |
| 16:10:29 | bauzas | in theory, today was a Feature Review Day | |
| 16:10:40 | bauzas | but I feel many people having different priorities | |
| 16:10:50 | bauzas | so I haven't really called it | |
| 16:11:14 | bauzas | #topic Review priorities | |
| 16:11:18 | bauzas | #link https://review.opendev.org/q/status:open+(project:openstack/nova+OR+project:openstack/placement+OR+project:openstack/os-traits+OR+project:openstack/os-resource-classes+OR+project:openstack/os-vif+OR+project:openstack/python-novaclient+OR+project:openstack/osc-placement)+(label:Review-Priority%252B1+OR+label:Review-Priority%252B2) | |
| 16:11:22 | bauzas | #info As a reminder, people eager to review changes can +1 to indicate their interest, +2 for asking cores to also review | |
| 16:11:26 | bauzas | #topic Stable Branches | |
| 16:11:31 | bauzas | elodilles: your time | |
| 16:11:35 | elodilles | #info nova-lvm / nova-ceph-multistore jobs are fixed on all branches (2023.1, zed, yoga) \o/ | |
| 16:11:38 | elodilles | #info stable/victoria gate fix has also landed | |
| 16:11:40 | elodilles | #info gates from 2023.1 back till train should be OK | |
| 16:11:47 | elodilles | #info stable branch status / gate failures tracking etherpad: https://etherpad.opendev.org/p/nova-stable-branch-ci | |
| 16:11:52 | elodilles | EOM | |
| 16:11:55 | bauzas | thanks | |
| 16:12:02 | bauzas | #info train-eol patch proposed https://review.opendev.org/c/openstack/releases/+/885365 | |
| 16:12:09 | bauzas | we're missing a second release core vote | |
| 16:12:14 | bauzas | but should happen eventually | |
| 16:12:20 | elodilles | yepp | |
| 16:12:32 | bauzas | auniyal had a point, I'll rephrase it | |
| 16:12:44 | bauzas | #info Please review these backport patches of stable 2023.1, zed and yoga for next minor release | |
| 16:12:49 | bauzas | #info most of these already have one +2. | |
| 16:12:52 | bauzas | #link https://etherpad.opendev.org/p/release-liaison-PatchesToReview | |
| 16:13:09 | bauzas | and the stable branches are now back in healthy state | |
| 16:13:18 | bauzas | #topic Open discussion | |
| 16:13:26 | bauzas | I have one item | |
| 16:14:17 | bauzas | as I said previously, I'll be on PTO starting next Tues | |
| 16:14:31 | bauzas | gibi volunteered for running the Aug 15 meeting | |
| 16:14:38 | bauzas | (and Aug 22) | |
| 16:14:49 | bauzas | but we may need someone to chair the Aug 1 and Aug 8 meetings | |
| 16:14:57 | bauzas | or we skip them | |
| 16:15:40 | gibi | jepp I still OK with the 15 and 22 | |
| 16:15:51 | bauzas | so, anyone fancy volunteering ? if not, nevermind, I'll send an email to cancel the two weeks | |
| 16:16:31 | bauzas | I can try to handle Aug 1 meeting as I won't (yet) be on a plane, but I can't promise | |
| 16:16:48 | bauzas | ok, let's consider the meetings cancelled | |
| 16:17:04 | bauzas | #info Aug 1 and Aug 8 Nova meetings are CANCELLED | |
| 16:17:15 | bauzas | #action bauzas to notify the ML accordingly | |
| 16:17:17 | bauzas | that's it | |
| 16:17:19 | bauzas | for me | |
| 16:17:24 | bauzas | anything anyone ? | |
| 16:18:24 | bauzas | ok, thanks then | |
| 16:18:41 | opendevmeet | Log: https://meetings.opendev.org/meetings/nova/2023/nova.2023-07-25-16.00.log.html | |
| 16:18:41 | opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/nova/2023/nova.2023-07-25-16.00.txt | |
| 16:18:41 | opendevmeet | Minutes: https://meetings.opendev.org/meetings/nova/2023/nova.2023-07-25-16.00.html | |
| 16:18:41 | opendevmeet | Meeting ended Tue Jul 25 16:18:41 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | |
| 16:18:41 | bauzas | #endmeeting | |
| 16:19:00 | gibi | o/ | |
| 16:19:09 | elodilles | thanks, too o/ | |
| 16:21:14 | gmann | thanks o/ | |
| 16:58:39 | opendevreview | sean mooney proposed openstack/nova master: [WIP] use disk caching to hide slow cinder performance https://review.opendev.org/c/openstack/nova/+/889383 | |
| 16:59:11 | sean-k-mooney | if we decied this is the correct approch in general i can do ^ in devstack instead | |
| 16:59:45 | sean-k-mooney | or in the base job but im adding it to all the in tree job to get more input on if it affect the ci jobs positivly or negitivly | |
| 17:02:11 | dansmith | sean-k-mooney: I think doing it in devstack itself is the wrong move, but maybe in some of the devstack job defs.. this is really an optimization for our own CI and not something we'd want anyone running without knowing it, even in devstack, IMHO | |
| 17:02:41 | dansmith | I'll be interested to see if that helps significantly.. I've seen that bug manifest only occasionally in my surveying recently | |
| 17:02:42 | sean-k-mooney | i was thinking of doing it like the mysql memory thing | |
| 17:03:08 | sean-k-mooney | e.g. off by default with a macro/var to turn it on in ci | |
| 17:03:11 | dansmith | yeah, it could be a flag, sure | |
| 17:03:41 | sean-k-mooney | basically i just dont want to have to copy paste that to even more jobs over time :) | |