| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-02-27 | |||
| 17:49:49 | sean-k-mooney | oh ok | |
| 17:50:11 | sean-k-mooney | sound like an issue tha twould only happne if you created the db after it was removed form the schema | |
| 17:50:19 | sean-k-mooney | which happned in https://github.com/openstack/nova/commit/98a05bc637e4c2e485d5aa5b62945102ea71d08b | |
| 17:50:36 | sean-k-mooney | so was your cloud/db created in newton or later | |
| 17:51:02 | lowercase | mmm.. this cloud could be traced to around train. | |
| 17:51:42 | lowercase | regardless. I dropped the entire nova database, and created a fresh one. No tables present, and the error persisted. | |
| 17:52:13 | sean-k-mooney | stephenfin: this sounds like it might be relate to swaping to alembic | |
| 17:52:46 | lowercase | I have an env that is about 30 servers or so that is exclusive for dev testing releases before upgrading our other environments. | |
| 17:54:41 | sean-k-mooney | lowercase: so in yoga we have two ways to upgrade the db | |
| 17:54:54 | sean-k-mooney | we have the legacy migration based on sqlalchemy migrate | |
| 17:55:04 | sean-k-mooney | and we have alembic which is what we use now | |
| 17:55:44 | sean-k-mooney | the base schema we use for train + has the build_requests table https://github.com/openstack/nova/blob/stable/yoga/nova/db/api/legacy_migrations/versions/067_train.py#L159-L209 | |
| 17:56:35 | sean-k-mooney | and then in yoga we removed it | |
| 17:56:56 | sean-k-mooney | so somehow you seam to have droped it already | |
| 17:57:44 | stephenfin | lowercase: there should be no reason to check if it's present because we would have created it in the initial migration | |
| 17:58:04 | stephenfin | and if someone dropped it outside of the context of nova then all bets are off | |
| 17:58:55 | stephenfin | lowercase: you said you dropped the table and ran api_db sync again to see what happened. Did | |
| 17:59:02 | stephenfin | *dropped the database | |
| 17:59:03 | lowercase | What is happening right now, is openstack-ansible includes an api sync during its installation/upgrade path. | |
| 17:59:24 | stephenfin | Did you drop the database or just the tables? | |
| 17:59:44 | lowercase | so if i was to run the playbook multiple times.. could result in a situation where the api_sync has already removed the build_requests table being removed. | |
| 18:00:08 | stephenfin | No, alembic records the migrations it has already applied. It won't reapply them | |
| 18:00:19 | lowercase | i dropped the database as a test, and i confirmed the table was already dropped. I then created the table and continued my testing. | |
| 18:01:04 | stephenfin | if you look, you should see an 'alembic_version' table with this information in it | |
| 18:01:18 | lowercase | Running upgrade d67eeaabee36 -> b30f573d3377, Remove unused build_requests columns | |
| 18:01:49 | stephenfin | I then created the table and continued my testing. <-- how? | |
| 18:02:25 | stephenfin | The thing I'm trying to figure out is how much stuff, if anything, you are doing outside of nova-manage/alembic | |
| 18:04:00 | lowercase | Let me explain the order of events. I ran the nova playbook. resulted in an error. I logged intot he nova databases and confirmed the build_requests column was indeed absent. To isolate the issue, i created a dummy build_requests with vm_state. Error persisted. I dropped build_requests again, as expected. The error persisted. I then, backed up, dropped and recreated the nova database entirely. Resulting in the same issue. | |
| 18:04:59 | lowercase | I've been picking at this issue for about two weeks, only today was i able to find the resulting code that resulted in my issue. | |
| 18:06:03 | sean-k-mooney | right but it sound like you are startign form an invalid db state | |
| 18:06:09 | sean-k-mooney | im jus tlooking at the osa code now | |
| 18:06:13 | stephenfin | lowercase: can you drop the database again then run the following: 'nova-manage api_db sync d67eeaabee36' | |
| 18:06:13 | sean-k-mooney | to see how its creatign the db | |
| 18:06:48 | stephenfin | ...and share both the output of the command and the schema of the table after doing so | |
| 18:07:12 | stephenfin | that will apply all migrations up to the offending one but not that migration itself | |
| 18:07:38 | lowercase | show databases; -> confirmed nova was present. drop database nova; and then show databases; confirmed nova is no longer present. | |
| 18:07:48 | sean-k-mooney | https://github.com/openstack/openstack-ansible-os_nova/blob/stable/yoga/tasks/nova_db_setup.yml and https://github.com/openstack/openstack-ansible-os_nova/blob/stable/yoga/tasks/nova_db_post_setup.yml look ok | |
| 18:08:18 | lowercase | create database nova; show databases; nova is now present | |
| 18:08:56 | sean-k-mooney | lowercase: nova is the cell db | |
| 18:09:00 | sean-k-mooney | lowercase: not the api db | |
| 18:09:10 | sean-k-mooney | https://github.com/openstack/openstack-ansible-os_nova/blob/stable/yoga/defaults/main.yml#L93 | |
| 18:09:16 | sean-k-mooney | https://github.com/openstack/openstack-ansible-os_nova/blob/stable/yoga/defaults/main.yml#L107 | |
| 18:09:25 | lowercase | 0 | |
| 18:09:25 | lowercase | echo $? | |
| 18:09:25 | lowercase | Modules with known eventlet monkey patching issues were imported prior to eventlet monkey patching: urllib3. This warning can usually be ignored if the caller is only importing and not executing nova code. | |
| 18:09:25 | lowercase | /openstack/venvs/nova-25.2.1.dev3/bin/nova-manage api_db sync d67eeaabee36 | |
| 18:09:26 | sean-k-mooney | lowercase: you shoudl be droping and recreateing nova_api | |
| 18:09:54 | stephenfin | sean-k-mooney: good spot :) | |
| 18:10:34 | lowercase | okay, since we are in a different spot. please confirm you want me to repeat the steps with nova_api databases. | |
| 18:10:59 | stephenfin | yup | |
| 18:11:07 | stephenfin | though before you do | |
| 18:11:14 | lowercase | okay, stoping. | |
| 18:11:28 | stephenfin | can you dump the contents of the 'alembic_version' table from nova_api | |
| 18:11:33 | lowercase | kk | |
| 18:11:44 | stephenfin | and perhaps the migrate_version table too, if there's one | |
| 18:11:49 | stephenfin | paste.openstack.org | |
| 18:12:44 | sean-k-mooney | nova is slightly differnt in that by default it has 3 dbs nova_api nova_cell0 and nova_cell1 and osa used ot use nova for nova_cell1 i think but changed at some point | |
| 18:14:15 | sean-k-mooney | so here nova is the cell1 db https://github.com/openstack/openstack-ansible-os_nova/blob/abac462dc218b08656b56c57e5a401aa639a0f0b/tasks/main.yml#L69-L80 | |
| 18:15:14 | lowercase | sorry, my mysql isn't very strong. show columns from alembic_version; shows me what the key and type everything is. how do i dump the data from a field? | |
| 18:15:52 | sean-k-mooney | select * from alembic_versions; | |
| 18:16:11 | sean-k-mooney | i think its actully just one row | |
| 18:16:19 | lowercase | select * from alembic_version; results in d67eeaabee36 | |
| 18:16:29 | lowercase | which is what we expect | |
| 18:16:43 | stephenfin | is there a migrate_version table? | |
| 18:17:08 | lowercase | yes | |
| 18:17:23 | lowercase | | nova_api | /openstack/venvs/nova-21.2.0/lib/python3.6/site-packages/nova/db/sqlalchemy/api_migrations/migrate_repo | 87 | | |
| 18:19:14 | stephenfin | cool, that's the most recent version of the legacy migrations, so everything has been applied | |
| 18:19:20 | lowercase | Good to proceed with the dropping of the db? | |
| 18:19:35 | stephenfin | one last thing | |
| 18:20:07 | sean-k-mooney | well i would make a backup first | |
| 18:20:11 | lowercase | actually, since i was in the wrong db. there is a build_requests here. | |
| 18:20:22 | lowercase | Empty set (0.000 sec) | |
| 18:20:22 | lowercase | select * from build_requests; | |
| 18:20:43 | lowercase | sean-k-mooney: i backed up before all changes, we are good. | |
| 18:20:50 | sean-k-mooney | you could do a "show create table build_request;" | |
| 18:20:51 | stephenfin | can you dump the schema from that table? | |
| 18:21:16 | lowercase | ERROR 1146 (42S02): Table 'nova_api.build_request' doesn't exist | |
| 18:21:16 | lowercase | show create table build_request; | |
| 18:21:18 | sean-k-mooney | that what the command above does by the way | |
| 18:21:29 | sean-k-mooney | missing an s at the end sorry | |
| 18:21:42 | sean-k-mooney | show create table build_requests; | |
| 18:21:43 | lowercase | let me paste this | |
| 18:21:48 | lowercase | my fault, i should have caught that | |
| 18:22:00 | sean-k-mooney | i typo things constantly | |
| 18:22:39 | lowercase | https://paste.centos.org/view/cf6648bc | |
| 18:24:49 | lowercase | Still holding off on blowing away the db. | |
| 18:24:56 | stephenfin | You can blow it away now | |
| 18:25:00 | lowercase | kk | |
| 18:25:08 | sean-k-mooney | so that db is missing the filed in from the train schema | |
| 18:25:13 | sean-k-mooney | so it has already been migrated | |
| 18:26:03 | sean-k-mooney | or contracted/truncated in this case | |
| 18:26:11 | lowercase | drop nova_api database; show database ; -> confirmed nova-api is no longer present. | |
| 18:27:40 | lowercase | create database nova_api ; and /openstack/venvs/nova-25.2.1.dev3/bin/nova-manage api_db sync now results in echo $? == 0 | |
| 18:27:56 | sean-k-mooney | ok so it succeeded | |
| 18:28:40 | sean-k-mooney | you shoudl be able to run "nova-manage api_db sync" again | |
| 18:28:50 | sean-k-mooney | and i basically shoudl do nothing | |
| 18:29:03 | stephenfin | Cool. So something has gone wrong the first time you applied that. My guesses would be that it was already applied but the alembic_version table wasn't updated for some reason. The other guess is that someone manually dropped these at some point in the past | |
| 18:29:04 | lowercase | Correct, i ran again and no error | |
| 18:32:01 | sean-k-mooney | its also possible that you ran this pointing at the wrong db | |
| 18:32:20 | sean-k-mooney | the api db migratoins wont work on the cell dbs | |