| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-02-27 | |||
| 10:39:34 | tobias-urdin | sean-k-mooney: ack, thanks for all your help – extremely helpful :) | |
| 10:39:34 | tobias-urdin | sean-k-mooney: ack, thanks for all your help – extremely helpful :) | |
| 10:40:07 | sean-k-mooney | the pci devices in placement spec added one of the builing block features we will need for numa in placment (the ablity for filters to filter on allcoations candiates) | |
| 10:40:38 | opendevreview | Sylvain Bauza proposed openstack/nova master: Add the 2023.1 Antelope prelude section https://review.opendev.org/c/openstack/nova/+/875380 | |
| 10:41:18 | sean-k-mooney | lol i tought you had written that but that was the release highlights | |
| 10:41:59 | sean-k-mooney | did we land the spice console compressiong feature | |
| 10:42:12 | bauzas | yes | |
| 10:42:22 | opendevreview | Jorge San Emeterio proposed openstack/nova master: Moving privsep profiles to nova/__init__.py https://review.opendev.org/c/openstack/nova/+/872010 | |
| 10:42:22 | sean-k-mooney | good good | |
| 10:44:37 | sean-k-mooney | i reviewed the spec for that and planned to review the feature i just ran out of time to do that im glad it landed | |
| 11:28:38 | opendevreview | Merged openstack/nova master: doc: mark the max microversion for 2023.1 Antelope https://review.opendev.org/c/openstack/nova/+/874103 | |
| 12:23:07 | opendevreview | Merged openstack/python-novaclient stable/2023.1: Update .gitreview for stable/2023.1 https://review.opendev.org/c/openstack/python-novaclient/+/875101 | |
| 12:26:51 | opendevreview | Rajesh Tailor proposed openstack/nova stable/xena: Fix rescue volume-based instance https://review.opendev.org/c/openstack/nova/+/875343 | |
| 12:27:56 | opendevreview | Rajesh Tailor proposed openstack/nova stable/wallaby: Fix rescue volume-based instance https://review.opendev.org/c/openstack/nova/+/875344 | |
| 12:28:38 | opendevreview | Merged openstack/os-vif stable/2023.1: Update .gitreview for stable/2023.1 https://review.opendev.org/c/openstack/os-vif/+/875095 | |
| 12:28:40 | opendevreview | Merged openstack/os-vif stable/2023.1: Update TOX_CONSTRAINTS_FILE for stable/2023.1 https://review.opendev.org/c/openstack/os-vif/+/875096 | |
| 12:29:21 | opendevreview | Merged openstack/osc-placement stable/2023.1: Update .gitreview for stable/2023.1 https://review.opendev.org/c/openstack/osc-placement/+/875098 | |
| 12:29:29 | opendevreview | Rajesh Tailor proposed openstack/nova stable/xena: Handle InstanceInvalidState exception https://review.opendev.org/c/openstack/nova/+/875345 | |
| 12:30:13 | opendevreview | Rajesh Tailor proposed openstack/nova stable/wallaby: Handle InstanceInvalidState exception https://review.opendev.org/c/openstack/nova/+/875346 | |
| 12:30:38 | opendevreview | Merged openstack/python-novaclient stable/2023.1: Update TOX_CONSTRAINTS_FILE for stable/2023.1 https://review.opendev.org/c/openstack/python-novaclient/+/875102 | |
| 12:32:30 | opendevreview | Rajesh Tailor proposed openstack/nova stable/victoria: Fix rescue volume-based instance https://review.opendev.org/c/openstack/nova/+/875347 | |
| 12:33:52 | opendevreview | Rajesh Tailor proposed openstack/nova stable/victoria: Handle InstanceInvalidState exception https://review.opendev.org/c/openstack/nova/+/875348 | |
| 12:40:11 | opendevreview | Merged openstack/nova master: Doc: update live-migration cmd https://review.opendev.org/c/openstack/nova/+/875043 | |
| 12:46:48 | opendevreview | Merged openstack/osc-placement stable/2023.1: Update TOX_CONSTRAINTS_FILE for stable/2023.1 https://review.opendev.org/c/openstack/osc-placement/+/875099 | |
| 14:23:58 | opendevreview | Jorge San Emeterio proposed openstack/nova master: WIP: Creating an example of the refactor privileged functions will go through. https://review.opendev.org/c/openstack/nova/+/875497 | |
| 14:36:05 | artom | bauzas, dansmith_, is there an etherpad or something for all the CI issues currently hitting us? | |
| 14:36:08 | artom | jsanemet ^^ | |
| 14:36:36 | dansmith_ | artom: there's this at least: https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure | |
| 14:37:34 | artom | Do we still have an ELK/opensearch instance somewhere? | |
| 14:43:10 | artom | That needs a login? | |
| 14:44:47 | dansmith_ | openstack/openstack | |
| 14:44:53 | dansmith_ | it's apparently not option-able anymore | |
| 14:48:44 | artom | https://opensearch.logs.openstack.org/_dashboards/app/discover, you were missing an 'r' :) | |
| 14:52:29 | artom | jsanemet, ^^ so yeah, that allows you to search through all the logs we keep from our Zuul jobs | |
| 14:52:35 | artom | openstack/openstack to log in | |
| 14:52:49 | artom | There's a query language that I always get wrong to search for specific things | |
| 14:53:25 | jsanemet | cool, that will be useful | |
| 14:53:32 | artom | Apparently you can use https://opensearch.org/docs/latest/opensearch/query-dsl/index/, or Lucene directly | |
| 14:56:24 | artom | I don't know a lot about Lucene, https://www.lucenetutorial.com/lucene-query-syntax.html I guess? | |
| 14:56:53 | jsanemet | ok, i will play with these a little bit, see what can be done | |
| 14:57:05 | artom | So yeah, jsanemet, if you're interested in helping fix gate bugs, ^^ is a useful tool, and I'm sure dansmith's will be more than happy to take you under his wing and show you everything he knows ;) | |
| 14:57:47 | jsanemet | i need to take a look at the reported bugs from before | |
| 14:58:01 | jsanemet | but i will see if i can pick up something | |
| 14:58:12 | jsanemet | will be great to contribute | |
| 15:11:27 | opendevreview | Merged openstack/nova master: Transport context to all threads https://review.opendev.org/c/openstack/nova/+/827467 | |
| 15:33:28 | Uggla | bauzas, I need your opinion on that bug https://bugs.launchpad.net/nova/+bug/2008461, the bug seems legit, but the method is marked deprecated and might be removed at some point. How do you treat this bug ? | |
| 15:33:59 | Uggla | bauzas, I mean should I set it to valid in this case ? | |
| 15:34:39 | Uggla | s/valid/confirmed | |
| 16:05:18 | lowercase | Hey guys, i think i found an issue with https://review.opendev.org/c/openstack/nova/+/812145/2/nova/db/api/migrations/versions/b30f573d3377_remove_unused_build_requests_columns.py#30 . I'm currently upgrading from wallaby to yoga using openstack-ansible, and during the upgrade process i get the error, sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (1091, "Can't DROP COLUMN `vm_state`; check that it exists"). [SQL: | |
| 16:05:18 | lowercase | ALTER TABLE build_requests DROP COLUMN vm_state] The entire table, build_requests does not exist. Additionally, as a troubleshooting measure i droped the nova database, created a blank one and ran api_db sync again to see the same issue. | |
| 16:06:04 | lowercase | It would seem a lacking test to confirm if the table exists before attempting to drop it. | |
| 16:32:24 | lowercase | Opened this bug report to address this issue. https://bugs.launchpad.net/nova/+bug/2008716 | |
| 17:45:30 | sean-k-mooney | lowercase: you cant directly upgrade form wallaby to yoga | |
| 17:45:49 | lowercase | i miss remembered the version names, i went from xena to yoga | |
| 17:45:59 | lowercase | I corrected the bug as well | |
| 17:46:29 | lowercase | *i corrected the bug report to accuratly reflect my upgrade path. | |
| 17:47:01 | sean-k-mooney | so based on https://github.com/openstack/nova/commit/9657297dd6c63e7a1e0c84c3e943b26f1795d388 | |
| 17:47:04 | sean-k-mooney | these were never used | |
| 17:47:49 | sean-k-mooney | so im wondering | |
| 17:48:00 | sean-k-mooney | how you got entries into that db table | |
| 17:48:56 | lowercase | I do not have entries in the db table. My issue is that I do not have build_requests table and nova-manage api_db sync attempts to remove them, resulting in a python exception. | |
| 17:49:49 | sean-k-mooney | oh ok | |
| 17:50:11 | sean-k-mooney | sound like an issue tha twould only happne if you created the db after it was removed form the schema | |
| 17:50:19 | sean-k-mooney | which happned in https://github.com/openstack/nova/commit/98a05bc637e4c2e485d5aa5b62945102ea71d08b | |
| 17:50:36 | sean-k-mooney | so was your cloud/db created in newton or later | |
| 17:51:02 | lowercase | mmm.. this cloud could be traced to around train. | |
| 17:51:42 | lowercase | regardless. I dropped the entire nova database, and created a fresh one. No tables present, and the error persisted. | |
| 17:52:13 | sean-k-mooney | stephenfin: this sounds like it might be relate to swaping to alembic | |
| 17:52:46 | lowercase | I have an env that is about 30 servers or so that is exclusive for dev testing releases before upgrading our other environments. | |
| 17:54:41 | sean-k-mooney | lowercase: so in yoga we have two ways to upgrade the db | |
| 17:54:54 | sean-k-mooney | we have the legacy migration based on sqlalchemy migrate | |
| 17:55:04 | sean-k-mooney | and we have alembic which is what we use now | |
| 17:55:44 | sean-k-mooney | the base schema we use for train + has the build_requests table https://github.com/openstack/nova/blob/stable/yoga/nova/db/api/legacy_migrations/versions/067_train.py#L159-L209 | |
| 17:56:35 | sean-k-mooney | and then in yoga we removed it | |
| 17:56:56 | sean-k-mooney | so somehow you seam to have droped it already | |
| 17:57:44 | stephenfin | lowercase: there should be no reason to check if it's present because we would have created it in the initial migration | |
| 17:58:04 | stephenfin | and if someone dropped it outside of the context of nova then all bets are off | |
| 17:58:55 | stephenfin | lowercase: you said you dropped the table and ran api_db sync again to see what happened. Did | |
| 17:59:02 | stephenfin | *dropped the database | |
| 17:59:03 | lowercase | What is happening right now, is openstack-ansible includes an api sync during its installation/upgrade path. | |
| 17:59:24 | stephenfin | Did you drop the database or just the tables? | |
| 17:59:44 | lowercase | so if i was to run the playbook multiple times.. could result in a situation where the api_sync has already removed the build_requests table being removed. | |
| 18:00:08 | stephenfin | No, alembic records the migrations it has already applied. It won't reapply them | |
| 18:00:19 | lowercase | i dropped the database as a test, and i confirmed the table was already dropped. I then created the table and continued my testing. | |
| 18:01:04 | stephenfin | if you look, you should see an 'alembic_version' table with this information in it | |
| 18:01:18 | lowercase | Running upgrade d67eeaabee36 -> b30f573d3377, Remove unused build_requests columns | |
| 18:01:49 | stephenfin | I then created the table and continued my testing. <-- how? | |
| 18:02:25 | stephenfin | The thing I'm trying to figure out is how much stuff, if anything, you are doing outside of nova-manage/alembic | |
| 18:04:00 | lowercase | Let me explain the order of events. I ran the nova playbook. resulted in an error. I logged intot he nova databases and confirmed the build_requests column was indeed absent. To isolate the issue, i created a dummy build_requests with vm_state. Error persisted. I dropped build_requests again, as expected. The error persisted. I then, backed up, dropped and recreated the nova database entirely. Resulting in the same issue. | |
| 18:04:59 | lowercase | I've been picking at this issue for about two weeks, only today was i able to find the resulting code that resulted in my issue. | |
| 18:06:03 | sean-k-mooney | right but it sound like you are startign form an invalid db state | |
| 18:06:09 | sean-k-mooney | im jus tlooking at the osa code now | |
| 18:06:13 | sean-k-mooney | to see how its creatign the db | |
| 18:06:13 | stephenfin | lowercase: can you drop the database again then run the following: 'nova-manage api_db sync d67eeaabee36' | |
| 18:06:48 | stephenfin | ...and share both the output of the command and the schema of the table after doing so | |
| 18:07:12 | stephenfin | that will apply all migrations up to the offending one but not that migration itself | |
| 18:07:38 | lowercase | show databases; -> confirmed nova was present. drop database nova; and then show databases; confirmed nova is no longer present. | |
| 18:07:48 | sean-k-mooney | https://github.com/openstack/openstack-ansible-os_nova/blob/stable/yoga/tasks/nova_db_setup.yml and https://github.com/openstack/openstack-ansible-os_nova/blob/stable/yoga/tasks/nova_db_post_setup.yml look ok | |
| 18:08:18 | lowercase | create database nova; show databases; nova is now present | |
| 18:08:56 | sean-k-mooney | lowercase: nova is the cell db | |