| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-02-28 | |||
| 15:08:16 | bauzas | bobcat will open in 2 days :) | |
| 15:09:01 | lowercase | show tables; -> build_requests is present. select * from build_requests ; -> Empty set (0.001 sec) show full columns from build_requests ; -> there are about 23 columns there from a quick count. | |
| 15:09:25 | dansmith | bauzas: I was going to say it actually matters when the relevant projects have branches, but that's the grenade holdup.. since this is two back, we should be able to do it immediately, yeah | |
| 15:10:10 | dansmith | bauzas: the only concern would be people using the job to merge things on their rc branches that need coverage.. but I think we're going to need a grenade-skip-level-previous (or something) job anyway to keep stable happy, so maybe this is the time to practice that | |
| 15:10:36 | bauzas | hmmm | |
| 15:11:06 | lowercase | stephenfin: see previous response | |
| 15:12:27 | stephenfin | lowercase: can you dump the output of 'describe build_requests;' to a pastebin? | |
| 15:12:33 | bauzas | dansmith: have you seen my service version change ? We shouldn't longer need to use this https://github.com/openstack/grenade/blob/master/.zuul.yaml#L401 | |
| 15:13:04 | stephenfin | lowercase: and 'select * from alembic_version;' | |
| 15:13:19 | dansmith | bauzas: something other than 875621? | |
| 15:13:27 | stephenfin | lowercase: This is for the next environment, of course | |
| 15:13:54 | bauzas | dansmith: https://review.opendev.org/c/openstack/nova/+/874932 | |
| 15:13:58 | lowercase | stephenfin: this is my qa cluster that is on xena, and is not being upgraded to yoga yet. d67eeaabee36 | |
| 15:15:06 | dansmith | bauzas: ah, yeah I had but I didn't connect the dots | |
| 15:15:44 | bauzas | dansmith: so are you agreeing on the change ? (like, just continuing to support Yoga) | |
| 15:15:54 | stephenfin | lowercase: Cool, so alembic_version looks correct. Now to ensure that the columns that migration b30f573d3377 removes are present there, as expected (the 'describe' command) | |
| 15:16:27 | bauzas | dansmith: and just after Antelope RC1, bumping the service version to Antelope | |
| 15:16:40 | dansmith | bauzas: yeah I mean, I think I'm okay with that.. the change doesn't really change anything (other than get ready for the future with the missing service versions) right? | |
| 15:17:10 | lowercase | stephenfin: https://paste.centos.org/view/529a12ed | |
| 15:17:16 | dansmith | bauzas: meaning I'm happy to loosen our requirements a cycle before we officially do | |
| 15:17:41 | bauzas | dansmith: yeah the problem that I have is that I need to understand again every cycle how we can support SLURP | |
| 15:17:59 | bauzas | so I want to make sure we do this correctlyu | |
| 15:19:34 | stephenfin | lowercase: Okay, they're all there as expected. I would expect the migrations to apply cleanly in that environment. So the question is how is that test environment created? | |
| 15:20:41 | lowercase | stephenfin: the test and qa environments are created using openstack-ansible and I do a really good job ensuring it stays very close to our remaining environments | |
| 15:20:59 | stephenfin | lowercase: In that case, I suspect either (a) someone has changed something behind alembic's back or (b) alembic ran but failed to record the updated version in the alembic_version table. I think you already ruled out (a)? | |
| 15:21:09 | lowercase | uhh.. so i just dropped that table in test and it now errors with 1146, \"Table 'nova_api.build_requests' doesn't exist\")", "[SQL: ALTER TABLE build_requests DROP COLUMN vm_state]", "(Background on this error at: https://sqlalche.me/e/14/f405)"]} bahaha | |
| 15:21:24 | lowercase | in test, this is the one that is going to yoga | |
| 15:22:22 | stephenfin | lowercase: that's expected. You've modified the schema behind alembic's back. It (intentionally) doesn't do introspection. It expects the schema to be in a given state and makes changes based on that | |
| 15:23:08 | stephenfin | The only thing that should be issuing CREATE, DROP or ALTER operations is alembic. Not the operator. | |
| 15:23:21 | lowercase | stephenfin: that's a fair statement. its just funny that im going to need to reverse engineer the creation of a table just so the script can remove it | |
| 15:23:39 | stephenfin | lowercase: It's not removing the table - it's modifying it | |
| 15:24:23 | lowercase | oh yep. okay i see that now. just dropping everything inside of it and emptying the schema | |
| 15:26:46 | stephenfin | lowercase: Nah, it doesn't drop everything inside of it either. As the name would suggest, the build requests table contains build requests. A record is created in there when a build is requested. It's deleted once that build is completed (either success or failure) | |
| 15:27:23 | stephenfin | That's why it looks empty. If you watched the table while creating instances though, you'd see stuff appearing and disappearing. | |
| 15:27:47 | stephenfin | https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L554-L563 | |
| 15:29:33 | stephenfin | lowercase: Also, this is the schema from a Zed-based DevStack deployment https://paste.opendev.org/show/bm3psmcu51ltEvP1sT9D/ | |
| 15:32:14 | lowercase | i just dumped the table out of qa cluster, and im using that to build the dev cluster to get it to pass | |
| 15:33:29 | lowercase | okay, that looks pretty good.. rerunning the code! | |
| 15:40:55 | lowercase | stephenfin: "Error: (pymysql.err.OperationalError) (1091, \"Can't DROP INDEX `shadow_instance_extra_idx`; check that it exists\")", "[SQL: ", "DROP INDEX shadow_instance_extra_idx ON shadow_instance_extra]", | |
| 15:41:40 | lowercase | regredably, im getting swamped over here. I don't have time to look into this further. I'll take some time to look at this and get back to you. | |
| 15:51:20 | bauzas | reminder : nova meeting in 9 mins here | |
| 15:51:49 | bauzas | I'll also need to bail out quickly after the meeting | |
| 16:02:00 | opendevmeet | The meeting name has been set to 'nova' | |
| 16:02:00 | opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | |
| 16:02:00 | opendevmeet | Meeting started Tue Feb 28 16:02:00 2023 UTC and is due to finish in 60 minutes. The chair is bauzas. Information about MeetBot at http://wiki.debian.org/MeetBot. | |
| 16:02:00 | bauzas | #startmeeting nova | |
| 16:02:06 | bauzas | sorry, bit late here | |
| 16:02:22 | Uggla | o/ | |
| 16:02:30 | elodilles | o/ | |
| 16:02:31 | bauzas | #link https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting | |
| 16:02:40 | bauzas | let's start | |
| 16:02:44 | dansmith | o/ | |
| 16:02:44 | bauzas | #topic Bugs (stuck/critical) | |
| 16:02:50 | bauzas | #info No Critical bug | |
| 16:02:54 | bauzas | #link https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New 17 new untriaged bugs (+1 since the last meeting) | |
| 16:03:05 | bauzas | Uggla: any bug you wanna raise ? | |
| 16:03:11 | gibi | o/ | |
| 16:03:23 | bauzas | if not, let's skip | |
| 16:03:28 | bauzas | #info Add yourself in the team bug roster if you want to help https://etherpad.opendev.org/p/nova-bug-triage-roster | |
| 16:03:41 | bauzas | sean-k-mooney: can you be the next bug baton owner ? | |
| 16:03:46 | Uggla | bauzas, no nothing special | |
| 16:03:50 | bauzas | ack | |
| 16:04:04 | sean-k-mooney | i guess so | |
| 16:04:23 | bauzas | cool ta | |
| 16:04:27 | bauzas | #info bug baton is being passed to sean-k-mooney | |
| 16:04:31 | bauzas | moving on so | |
| 16:04:36 | bauzas | #topic Gate status | |
| 16:04:41 | bauzas | #link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs | |
| 16:04:47 | bauzas | #link https://etherpad.opendev.org/p/nova-ci-failures | |
| 16:05:01 | bauzas | I'll be honest, I didn't had a lot of time to look at the failures | |
| 16:05:11 | bauzas | have you seen some CI failure ? | |
| 16:05:24 | dansmith | definitely improving | |
| 16:05:39 | bauzas | indeed | |
| 16:05:42 | dansmith | the volume detach on cleanup failures have all but gone away except for the live mgiration tests, | |
| 16:05:48 | dansmith | and I have a patch up for that right now | |
| 16:06:15 | dansmith | both of those are failures in cinder tests that have gone on for years without getting proper attention | |
| 16:06:29 | dansmith | but they've been spiking a lot lately, so hopefully this will really improve things | |
| 16:06:36 | bauzas | ok | |
| 16:06:42 | gibi | glad to hear that | |
| 16:07:01 | elodilles | btw, can we expect the same for stable branches with this fix as well? (less volume detach and cleanup failures) | |
| 16:07:14 | dansmith | elodilles: the fixes were in tempest | |
| 16:07:24 | dansmith | elodilles: so that's branchless and should apply to stable right? | |
| 16:07:28 | bauzas | I don't see a lot of failures for the same job | |
| 16:07:29 | elodilles | so it means that where the tempest is not pinned | |
| 16:07:37 | elodilles | (non-EM branches) | |
| 16:07:37 | dansmith | ah, right | |
| 16:07:50 | elodilles | cool, thanks! | |
| 16:08:26 | bauzas | ok, let's then move on | |
| 16:08:53 | bauzas | #link https://zuul.openstack.org/builds?project=openstack%2Fnova&project=openstack%2Fplacement&pipeline=periodic-weekly Nova&Placement periodic jobs status | |
| 16:08:56 | bauzas | all greens | |
| 16:09:01 | bauzas | #info Please look at the gate failures and file a bug report with the gate-failure tag. | |
| 16:09:05 | bauzas | #info STOP DOING BLIND RECHECKS aka. 'recheck' https://docs.openstack.org/project-team-guide/testing.html#how-to-handle-test-failures | |
| 16:09:25 | bauzas | #topic Release Planning | |
| 16:09:30 | bauzas | #link https://releases.openstack.org/antelope/schedule.html | |
| 16:09:34 | bauzas | #info Antelope-rc1 is in 0.2 weeks | |
| 16:09:40 | bauzas | (which is on Thursday) | |
| 16:09:46 | bauzas | #link https://etherpad.opendev.org/p/nova-antelope-rc-potential | |
| 16:09:49 | elodilles | even the rc1 patches are generated! ;) | |
| 16:10:00 | elodilles | * release patches | |
| 16:10:31 | bauzas | elodilles cool but we need to update at least the nova one | |