| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-02-28 | |||
| 15:13:27 | stephenfin | lowercase: This is for the next environment, of course | |
| 15:13:54 | bauzas | dansmith: https://review.opendev.org/c/openstack/nova/+/874932 | |
| 15:13:58 | lowercase | stephenfin: this is my qa cluster that is on xena, and is not being upgraded to yoga yet. d67eeaabee36 | |
| 15:15:06 | dansmith | bauzas: ah, yeah I had but I didn't connect the dots | |
| 15:15:44 | bauzas | dansmith: so are you agreeing on the change ? (like, just continuing to support Yoga) | |
| 15:15:54 | stephenfin | lowercase: Cool, so alembic_version looks correct. Now to ensure that the columns that migration b30f573d3377 removes are present there, as expected (the 'describe' command) | |
| 15:16:27 | bauzas | dansmith: and just after Antelope RC1, bumping the service version to Antelope | |
| 15:16:40 | dansmith | bauzas: yeah I mean, I think I'm okay with that.. the change doesn't really change anything (other than get ready for the future with the missing service versions) right? | |
| 15:17:10 | lowercase | stephenfin: https://paste.centos.org/view/529a12ed | |
| 15:17:16 | dansmith | bauzas: meaning I'm happy to loosen our requirements a cycle before we officially do | |
| 15:17:41 | bauzas | dansmith: yeah the problem that I have is that I need to understand again every cycle how we can support SLURP | |
| 15:17:59 | bauzas | so I want to make sure we do this correctlyu | |
| 15:19:34 | stephenfin | lowercase: Okay, they're all there as expected. I would expect the migrations to apply cleanly in that environment. So the question is how is that test environment created? | |
| 15:20:41 | lowercase | stephenfin: the test and qa environments are created using openstack-ansible and I do a really good job ensuring it stays very close to our remaining environments | |
| 15:20:59 | stephenfin | lowercase: In that case, I suspect either (a) someone has changed something behind alembic's back or (b) alembic ran but failed to record the updated version in the alembic_version table. I think you already ruled out (a)? | |
| 15:21:09 | lowercase | uhh.. so i just dropped that table in test and it now errors with 1146, \"Table 'nova_api.build_requests' doesn't exist\")", "[SQL: ALTER TABLE build_requests DROP COLUMN vm_state]", "(Background on this error at: https://sqlalche.me/e/14/f405)"]} bahaha | |
| 15:21:24 | lowercase | in test, this is the one that is going to yoga | |
| 15:22:22 | stephenfin | lowercase: that's expected. You've modified the schema behind alembic's back. It (intentionally) doesn't do introspection. It expects the schema to be in a given state and makes changes based on that | |
| 15:23:08 | stephenfin | The only thing that should be issuing CREATE, DROP or ALTER operations is alembic. Not the operator. | |
| 15:23:21 | lowercase | stephenfin: that's a fair statement. its just funny that im going to need to reverse engineer the creation of a table just so the script can remove it | |
| 15:23:39 | stephenfin | lowercase: It's not removing the table - it's modifying it | |
| 15:24:23 | lowercase | oh yep. okay i see that now. just dropping everything inside of it and emptying the schema | |
| 15:26:46 | stephenfin | lowercase: Nah, it doesn't drop everything inside of it either. As the name would suggest, the build requests table contains build requests. A record is created in there when a build is requested. It's deleted once that build is completed (either success or failure) | |
| 15:27:23 | stephenfin | That's why it looks empty. If you watched the table while creating instances though, you'd see stuff appearing and disappearing. | |
| 15:27:47 | stephenfin | https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L554-L563 | |
| 15:29:33 | stephenfin | lowercase: Also, this is the schema from a Zed-based DevStack deployment https://paste.opendev.org/show/bm3psmcu51ltEvP1sT9D/ | |
| 15:32:14 | lowercase | i just dumped the table out of qa cluster, and im using that to build the dev cluster to get it to pass | |
| 15:33:29 | lowercase | okay, that looks pretty good.. rerunning the code! | |
| 15:40:55 | lowercase | stephenfin: "Error: (pymysql.err.OperationalError) (1091, \"Can't DROP INDEX `shadow_instance_extra_idx`; check that it exists\")", "[SQL: ", "DROP INDEX shadow_instance_extra_idx ON shadow_instance_extra]", | |
| 15:41:40 | lowercase | regredably, im getting swamped over here. I don't have time to look into this further. I'll take some time to look at this and get back to you. | |
| 15:51:20 | bauzas | reminder : nova meeting in 9 mins here | |
| 15:51:49 | bauzas | I'll also need to bail out quickly after the meeting | |
| 16:02:00 | bauzas | #startmeeting nova | |
| 16:02:00 | opendevmeet | Meeting started Tue Feb 28 16:02:00 2023 UTC and is due to finish in 60 minutes. The chair is bauzas. Information about MeetBot at http://wiki.debian.org/MeetBot. | |
| 16:02:00 | opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | |
| 16:02:00 | opendevmeet | The meeting name has been set to 'nova' | |
| 16:02:06 | bauzas | sorry, bit late here | |
| 16:02:22 | Uggla | o/ | |
| 16:02:30 | elodilles | o/ | |
| 16:02:31 | bauzas | #link https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting | |
| 16:02:40 | bauzas | let's start | |
| 16:02:44 | bauzas | #topic Bugs (stuck/critical) | |
| 16:02:44 | dansmith | o/ | |
| 16:02:50 | bauzas | #info No Critical bug | |
| 16:02:54 | bauzas | #link https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New 17 new untriaged bugs (+1 since the last meeting) | |
| 16:03:05 | bauzas | Uggla: any bug you wanna raise ? | |
| 16:03:11 | gibi | o/ | |
| 16:03:23 | bauzas | if not, let's skip | |
| 16:03:28 | bauzas | #info Add yourself in the team bug roster if you want to help https://etherpad.opendev.org/p/nova-bug-triage-roster | |
| 16:03:41 | bauzas | sean-k-mooney: can you be the next bug baton owner ? | |
| 16:03:46 | Uggla | bauzas, no nothing special | |
| 16:03:50 | bauzas | ack | |
| 16:04:04 | sean-k-mooney | i guess so | |
| 16:04:23 | bauzas | cool ta | |
| 16:04:27 | bauzas | #info bug baton is being passed to sean-k-mooney | |
| 16:04:31 | bauzas | moving on so | |
| 16:04:36 | bauzas | #topic Gate status | |
| 16:04:41 | bauzas | #link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs | |
| 16:04:47 | bauzas | #link https://etherpad.opendev.org/p/nova-ci-failures | |
| 16:05:01 | bauzas | I'll be honest, I didn't had a lot of time to look at the failures | |
| 16:05:11 | bauzas | have you seen some CI failure ? | |
| 16:05:24 | dansmith | definitely improving | |
| 16:05:39 | bauzas | indeed | |
| 16:05:42 | dansmith | the volume detach on cleanup failures have all but gone away except for the live mgiration tests, | |
| 16:05:48 | dansmith | and I have a patch up for that right now | |
| 16:06:15 | dansmith | both of those are failures in cinder tests that have gone on for years without getting proper attention | |
| 16:06:29 | dansmith | but they've been spiking a lot lately, so hopefully this will really improve things | |
| 16:06:36 | bauzas | ok | |
| 16:06:42 | gibi | glad to hear that | |
| 16:07:01 | elodilles | btw, can we expect the same for stable branches with this fix as well? (less volume detach and cleanup failures) | |
| 16:07:14 | dansmith | elodilles: the fixes were in tempest | |
| 16:07:24 | dansmith | elodilles: so that's branchless and should apply to stable right? | |
| 16:07:28 | bauzas | I don't see a lot of failures for the same job | |
| 16:07:29 | elodilles | so it means that where the tempest is not pinned | |
| 16:07:37 | dansmith | ah, right | |
| 16:07:37 | elodilles | (non-EM branches) | |
| 16:07:50 | elodilles | cool, thanks! | |
| 16:08:26 | bauzas | ok, let's then move on | |
| 16:08:53 | bauzas | #link https://zuul.openstack.org/builds?project=openstack%2Fnova&project=openstack%2Fplacement&pipeline=periodic-weekly Nova&Placement periodic jobs status | |
| 16:08:56 | bauzas | all greens | |
| 16:09:01 | bauzas | #info Please look at the gate failures and file a bug report with the gate-failure tag. | |
| 16:09:05 | bauzas | #info STOP DOING BLIND RECHECKS aka. 'recheck' https://docs.openstack.org/project-team-guide/testing.html#how-to-handle-test-failures | |
| 16:09:25 | bauzas | #topic Release Planning | |
| 16:09:30 | bauzas | #link https://releases.openstack.org/antelope/schedule.html | |
| 16:09:34 | bauzas | #info Antelope-rc1 is in 0.2 weeks | |
| 16:09:40 | bauzas | (which is on Thursday) | |
| 16:09:46 | bauzas | #link https://etherpad.opendev.org/p/nova-antelope-rc-potential | |
| 16:09:49 | elodilles | even the rc1 patches are generated! ;) | |
| 16:10:00 | elodilles | * release patches | |
| 16:10:31 | bauzas | elodilles cool but we need to update at least the nova one | |
| 16:10:53 | elodilles | ack, that is expected | |
| 16:11:03 | elodilles | don't forget to signal this (if not yet done) | |
| 16:11:10 | elodilles | with a -1 on it | |
| 16:11:33 | bauzas | at least we need to merge first https://review.opendev.org/c/openstack/nova/+/874932 (min servion version), https://review.opendev.org/c/openstack/nova/+/873584 (logging revert) and https://review.opendev.org/c/openstack/nova/+/875380 (reno prelude) | |
| 16:11:41 | elodilles | (and thanks in advance o:)) | |
| 16:11:58 | bauzas | elodilles: indeed, doing it now | |
| 16:12:39 | bauzas | so | |
| 16:13:20 | bauzas | about https://review.opendev.org/c/openstack/nova/+/874932 I think we are OK with how to use rolling upgrades with SLURP and non-SLURP releases | |
| 16:14:12 | bauzas | sean-k-mooney had a concern with https://review.opendev.org/c/openstack/nova/+/875380 but we'll discuss it tomorrow | |
| 16:14:45 | bauzas | anything people want to discuss with RC1 ? | |