| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-09-13 | |||
| 09:49:04 | gibi | sean-k-mooney: thanks for the quick check, I will try it in devstack | |
| 09:49:24 | sean-k-mooney | cool i was going to hack something quickly and push it | |
| 09:50:57 | gibi | your suggestion seem to work | |
| 09:50:58 | gibi | https://paste.opendev.org/show/809277/ | |
| 09:52:09 | opendevreview | Merged openstack/os-vif stable/xena: Update .gitreview for stable/xena https://review.opendev.org/c/openstack/os-vif/+/808452 | |
| 09:52:11 | opendevreview | Merged openstack/os-vif stable/xena: Update TOX_CONSTRAINTS_FILE for stable/xena https://review.opendev.org/c/openstack/os-vif/+/808453 | |
| 09:52:58 | gibi | sean-k-mooney: is it OK to you if I file a bug and push a fix? | |
| 09:53:53 | sean-k-mooney | sure | |
| 09:54:34 | gibi | OK, working on it | |
| 09:54:59 | sean-k-mooney | this is what i had | |
| 09:55:00 | opendevreview | sean mooney proposed openstack/nova master: [WIP] db version fix https://review.opendev.org/c/openstack/nova/+/808712 | |
| 09:55:14 | sean-k-mooney | feel free to reuse or ignore | |
| 09:55:21 | gibi | sean-k-mooney: OK, thanks | |
| 10:01:27 | gibi | reported a critical bug for it as it affects the execution of db sync as well https://bugs.launchpad.net/nova/+bug/1943436 | |
| 10:03:01 | sean-k-mooney | ah presumable because db sync checks the version first | |
| 10:03:25 | sean-k-mooney | do we know why this happens now | |
| 10:03:39 | sean-k-mooney | looking at git blame this has not changed for 3 months | |
| 10:04:07 | gibi | I think this is changed when we switched to alembic. was it 3 months ago? | |
| 10:04:18 | sean-k-mooney | yep | |
| 10:04:26 | gibi | I think we don't have test coverage on this path | |
| 10:04:26 | sean-k-mooney | well that is thet commit date | |
| 10:04:44 | sean-k-mooney | we should for sync however right | |
| 10:04:55 | gibi | as it only happens if the db is migrated to alembic control and then you wan to migrate it futher | |
| 10:05:18 | sean-k-mooney | ah | |
| 10:05:23 | gibi | so grenade will only hit it after we have Xena -> Yoga run | |
| 10:05:29 | sean-k-mooney | and since we have no migrations since then we would not see it | |
| 10:06:03 | sean-k-mooney | i was wondering if it was related to the alembic version bump 12 days ago | |
| 10:06:09 | sean-k-mooney | form 1.6.5 to 1.7.1 | |
| 10:06:26 | sean-k-mooney | e.g. if the engine was previously valid and nolonger is | |
| 10:06:44 | gibi | I did not see any job failing do to this. I just noticed this problem when run the CLI manually this morning | |
| 10:10:54 | gibi | hm I was mistaken, the db sync part is not affected | |
| 10:16:26 | sean-k-mooney | its still a regerssion in either case | |
| 10:16:52 | gibi | yepp | |
| 10:17:02 | gibi | trying to add repro / test coverage | |
| 10:24:20 | opendevreview | Merged openstack/os-vif master: Update master for stable/xena https://review.opendev.org/c/openstack/os-vif/+/808454 | |
| 10:42:47 | opendevreview | Merged openstack/placement master: Reproduce 404 when allocation queried with 1.38 https://review.opendev.org/c/openstack/placement/+/807154 | |
| 10:57:31 | gibi | sean-k-mooney: alembic_api.current does not return the version alembic_script.ScriptDirectory.from_config( self.config) | |
| 10:57:34 | gibi | sorry | |
| 10:57:36 | gibi | wrong buffer | |
| 10:57:43 | gibi | so here https://github.com/openstack/nova/blob/master/nova/db/migration.py#L173 | |
| 10:57:55 | gibi | the alembic_api.current does not return anything | |
| 10:58:02 | gibi | that function actually prints to stdout | |
| 11:02:29 | sean-k-mooney | hehe lovely | |
| 11:02:31 | sean-k-mooney | so 2 bugs | |
| 11:03:18 | sean-k-mooney | https://alembic.sqlalchemy.org/en/latest/api/commands.html#alembic.command.current right Display the current revision for a database. | |
| 11:04:02 | sean-k-mooney | we likely dont want to call the function form the command module | |
| 11:04:17 | sean-k-mooney | we likely want to get the saem from the engin or similar | |
| 11:10:12 | gibi | yepp | |
| 11:10:19 | gibi | https://alembic.sqlalchemy.org/en/latest/api/runtime.html#alembic.runtime.migration.MigrationContext | |
| 11:10:23 | gibi | I think that is the right way | |
| 11:10:40 | gibi | there is get_current_revision() on the MigrationContext object | |
| 11:10:43 | sean-k-mooney | https://alembic.sqlalchemy.org/en/latest/api/runtime.html#alembic.runtime.migration.MigrationContext.get_current_revision | |
| 11:10:45 | sean-k-mooney | ya | |
| 11:11:08 | sean-k-mooney | i saw that before when i first went looking for current | |
| 11:11:23 | sean-k-mooney | Return the current revision, usually that which is present in the alembic_version table in the database. | |
| 11:11:49 | sean-k-mooney | that ussaully is presumable becasue before you run the migration it may or may not match? | |
| 11:12:41 | sean-k-mooney | its likely good enough for our use case | |
| 11:14:01 | sean-k-mooney | zzzeek: if your around if we want to get teh current db version is get_current_revision() the correct function to call | |
| 11:57:54 | opendevreview | Merged openstack/placement master: Fix adding 'unknown' to the ConsumerTypeCache https://review.opendev.org/c/openstack/placement/+/807155 | |
| 12:11:41 | opendevreview | Balazs Gibizer proposed openstack/nova master: Reproduce bug/1943436 https://review.opendev.org/c/openstack/nova/+/808753 | |
| 12:13:07 | opendevreview | Balazs Gibizer proposed openstack/nova master: Fix nova-manage db version https://review.opendev.org/c/openstack/nova/+/808712 | |
| 12:13:23 | gibi | sean-k-mooney: ^^ | |
| 12:13:49 | gibi | I dropped your refactoring as the config object is not needed any more | |
| 12:14:00 | gibi | and the engine object was already created in scope | |
| 12:14:21 | gibi | bauzas, lyarwood: it is an RC critical fix ^^ | |
| 12:14:34 | bauzas | gibi: ack, will look | |
| 12:14:46 | sean-k-mooney | cool | |
| 12:15:23 | sean-k-mooney | ill take a look although it proably does not need the co-authoured by since you drop most of what i did | |
| 12:16:02 | bauzas | gibi: wow, thanks for finding the issue | |
| 12:16:43 | gibi | sean-k-mooney: you was the one that point out where the code failed so you earned the mention :) | |
| 12:17:17 | sean-k-mooney | yeah strack traces :) | |
| 12:17:31 | gibi | bauzas: I just tried to run nova-manage db version in my devstack this morning, and found things to for today :) | |
| 12:17:58 | bauzas | gibi: I just wonder why we had no find the issue with the existing tests | |
| 12:18:09 | bauzas | hadn't* | |
| 12:18:38 | sean-k-mooney | bauzas: because we would be monkey patching the calls to the alembic lib | |
| 12:19:05 | sean-k-mooney | so it would only show up in integration style test or maybe functional | |
| 12:19:11 | sean-k-mooney | not in unit test | |
| 12:20:02 | sean-k-mooney | bauzas: also i think gibi mentioned it only happens the second time you call sync | |
| 12:20:16 | sean-k-mooney | when its using alembic? | |
| 12:20:25 | gibi | it only fails if you db already under alembic control | |
| 12:20:43 | sean-k-mooney | right so with an empty db it wont fail | |
| 12:20:45 | gibi | so grenade did not detect it as it upgrades from legacy to alembic not alembic to alembiv | |
| 12:20:47 | bauzas | sean-k-mooney: we have functional tests for nova-manage, nope ? | |
| 12:20:56 | gibi | bauzas: not for db version | |
| 12:21:02 | bauzas | ah right shit | |
| 12:21:16 | gibi | becuase that is "simple" it just calls alembic ;) | |
| 12:54:40 | bauzas | mmm, looks like we no longer generate guru meditation reports using USR2 signal ? | |
| 12:54:51 | bauzas | it's just recycling the service | |
| 12:55:00 | bauzas | gibi: can you test ? | |
| 12:55:10 | bauzas | (context, other escalation meanwhile) | |
| 12:55:32 | sean-k-mooney | bauzas: it shoudl dump it into the log | |
| 12:56:14 | sean-k-mooney | bauzas: so when you do the kill --USR2 it should output the gmr to the logs | |
| 12:56:32 | sean-k-mooney | not to standard out of the terminal you run kill in | |
| 12:57:01 | sean-k-mooney | its possibel it coudl end up in the container standar out but im 99% sure it went to the logs | |
| 12:57:17 | bauzas | sean-k-mooney: gibi: no, it's recycling | |
| 12:57:22 | bauzas | sean-k-mooney: gibi: https://paste.opendev.org/show/809280/ | |
| 12:57:40 | bauzas | it should dump the stateinto the logs | |
| 12:57:51 | bauzas | but rather, it gets the signal and then dies | |
| 12:58:02 | sean-k-mooney | that is to the wsgi server | |
| 12:58:05 | bauzas | I guess this is uswsgi related | |