| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-09-13 | |||
| 10:57:31 | gibi | sean-k-mooney: alembic_api.current does not return the version alembic_script.ScriptDirectory.from_config( self.config) | |
| 10:57:34 | gibi | sorry | |
| 10:57:36 | gibi | wrong buffer | |
| 10:57:43 | gibi | so here https://github.com/openstack/nova/blob/master/nova/db/migration.py#L173 | |
| 10:57:55 | gibi | the alembic_api.current does not return anything | |
| 10:58:02 | gibi | that function actually prints to stdout | |
| 11:02:29 | sean-k-mooney | hehe lovely | |
| 11:02:31 | sean-k-mooney | so 2 bugs | |
| 11:03:18 | sean-k-mooney | https://alembic.sqlalchemy.org/en/latest/api/commands.html#alembic.command.current right Display the current revision for a database. | |
| 11:04:02 | sean-k-mooney | we likely dont want to call the function form the command module | |
| 11:04:17 | sean-k-mooney | we likely want to get the saem from the engin or similar | |
| 11:10:12 | gibi | yepp | |
| 11:10:19 | gibi | https://alembic.sqlalchemy.org/en/latest/api/runtime.html#alembic.runtime.migration.MigrationContext | |
| 11:10:23 | gibi | I think that is the right way | |
| 11:10:40 | gibi | there is get_current_revision() on the MigrationContext object | |
| 11:10:43 | sean-k-mooney | https://alembic.sqlalchemy.org/en/latest/api/runtime.html#alembic.runtime.migration.MigrationContext.get_current_revision | |
| 11:10:45 | sean-k-mooney | ya | |
| 11:11:08 | sean-k-mooney | i saw that before when i first went looking for current | |
| 11:11:23 | sean-k-mooney | Return the current revision, usually that which is present in the alembic_version table in the database. | |
| 11:11:49 | sean-k-mooney | that ussaully is presumable becasue before you run the migration it may or may not match? | |
| 11:12:41 | sean-k-mooney | its likely good enough for our use case | |
| 11:14:01 | sean-k-mooney | zzzeek: if your around if we want to get teh current db version is get_current_revision() the correct function to call | |
| 11:57:54 | opendevreview | Merged openstack/placement master: Fix adding 'unknown' to the ConsumerTypeCache https://review.opendev.org/c/openstack/placement/+/807155 | |
| 12:11:41 | opendevreview | Balazs Gibizer proposed openstack/nova master: Reproduce bug/1943436 https://review.opendev.org/c/openstack/nova/+/808753 | |
| 12:13:07 | opendevreview | Balazs Gibizer proposed openstack/nova master: Fix nova-manage db version https://review.opendev.org/c/openstack/nova/+/808712 | |
| 12:13:23 | gibi | sean-k-mooney: ^^ | |
| 12:13:49 | gibi | I dropped your refactoring as the config object is not needed any more | |
| 12:14:00 | gibi | and the engine object was already created in scope | |
| 12:14:21 | gibi | bauzas, lyarwood: it is an RC critical fix ^^ | |
| 12:14:34 | bauzas | gibi: ack, will look | |
| 12:14:46 | sean-k-mooney | cool | |
| 12:15:23 | sean-k-mooney | ill take a look although it proably does not need the co-authoured by since you drop most of what i did | |
| 12:16:02 | bauzas | gibi: wow, thanks for finding the issue | |
| 12:16:43 | gibi | sean-k-mooney: you was the one that point out where the code failed so you earned the mention :) | |
| 12:17:17 | sean-k-mooney | yeah strack traces :) | |
| 12:17:31 | gibi | bauzas: I just tried to run nova-manage db version in my devstack this morning, and found things to for today :) | |
| 12:17:58 | bauzas | gibi: I just wonder why we had no find the issue with the existing tests | |
| 12:18:09 | bauzas | hadn't* | |
| 12:18:38 | sean-k-mooney | bauzas: because we would be monkey patching the calls to the alembic lib | |
| 12:19:05 | sean-k-mooney | so it would only show up in integration style test or maybe functional | |
| 12:19:11 | sean-k-mooney | not in unit test | |
| 12:20:02 | sean-k-mooney | bauzas: also i think gibi mentioned it only happens the second time you call sync | |
| 12:20:16 | sean-k-mooney | when its using alembic? | |
| 12:20:25 | gibi | it only fails if you db already under alembic control | |
| 12:20:43 | sean-k-mooney | right so with an empty db it wont fail | |
| 12:20:45 | gibi | so grenade did not detect it as it upgrades from legacy to alembic not alembic to alembiv | |
| 12:20:47 | bauzas | sean-k-mooney: we have functional tests for nova-manage, nope ? | |
| 12:20:56 | gibi | bauzas: not for db version | |
| 12:21:02 | bauzas | ah right shit | |
| 12:21:16 | gibi | becuase that is "simple" it just calls alembic ;) | |
| 12:54:40 | bauzas | mmm, looks like we no longer generate guru meditation reports using USR2 signal ? | |
| 12:54:51 | bauzas | it's just recycling the service | |
| 12:55:00 | bauzas | gibi: can you test ? | |
| 12:55:10 | bauzas | (context, other escalation meanwhile) | |
| 12:55:32 | sean-k-mooney | bauzas: it shoudl dump it into the log | |
| 12:56:14 | sean-k-mooney | bauzas: so when you do the kill --USR2 it should output the gmr to the logs | |
| 12:56:32 | sean-k-mooney | not to standard out of the terminal you run kill in | |
| 12:57:01 | sean-k-mooney | its possibel it coudl end up in the container standar out but im 99% sure it went to the logs | |
| 12:57:17 | bauzas | sean-k-mooney: gibi: no, it's recycling | |
| 12:57:22 | bauzas | sean-k-mooney: gibi: https://paste.opendev.org/show/809280/ | |
| 12:57:40 | bauzas | it should dump the stateinto the logs | |
| 12:57:51 | bauzas | but rather, it gets the signal and then dies | |
| 12:58:02 | sean-k-mooney | that is to the wsgi server | |
| 12:58:05 | bauzas | I guess this is uswsgi related | |
| 12:58:12 | sean-k-mooney | ya | |
| 12:58:18 | sean-k-mooney | have you tried it with nova-compute | |
| 12:58:52 | sean-k-mooney | for uwsgi i dont know if it ever worked but you might need to do it to the worker process | |
| 12:59:55 | bauzas | sean-k-mooney: I can run it against n-cpu for sure | |
| 13:00:06 | sean-k-mooney | that is what the customer need to do anyway | |
| 13:00:10 | sean-k-mooney | not nova-api | |
| 13:00:32 | bauzas | sean-k-mooney: sure but I wanted to test it against the API | |
| 13:00:38 | sean-k-mooney | bauzas: i dont know if we have ever tested GMR when the api is not runnign under python | |
| 13:00:54 | sean-k-mooney | bauzas: with nova-api directly it proably works | |
| 13:01:19 | sean-k-mooney | with nova-api under uwsgi i have never seen it documented how to make that work | |
| 13:01:30 | bauzas | sean-k-mooney: okay, this works with nova-compute service directly | |
| 13:01:41 | gibi | nova-compute gmr works for me | |
| 13:03:02 | gibi | nova-api doesnt | |
| 13:03:07 | bauzas | gibi: okay, so I guess it's PEBKAC as I issued the signal against the main PID on the wsgi server | |
| 13:03:14 | bauzas | ah | |
| 13:03:40 | bauzas | so, yeah, we should configure uwsgi to *not* handle this signal | |
| 13:03:54 | bauzas | a devstack change I guess | |
| 13:04:15 | gibi | nova-api works if I run it under python directly | |
| 13:05:35 | sean-k-mooney | SIGUSR2 -USR2 prints worker status | |
| 13:05:45 | sean-k-mooney | that is what uWSGI does by default | |
| 13:06:17 | bauzas | gibi: yeah, so this is because of the server | |
| 13:06:25 | sean-k-mooney | https://uwsgi-docs-additions.readthedocs.io/en/latest/Management.html#signals-for-controlling-uwsgi | |
| 13:06:56 | sean-k-mooney | we may or may not be able to change that | |
| 13:07:13 | bauzas | https://uwsgi-docs.readthedocs.io/en/latest/Management.html#signals-for-controlling-uwsgi | |
| 13:07:16 | bauzas | this | |
| 13:07:22 | bauzas | jinxed by sean-k-mooney | |
| 13:09:15 | opendevreview | Federico Ressi proposed openstack/nova master: Debug Nova APIs call failures https://review.opendev.org/c/openstack/nova/+/806683 | |
| 13:09:30 | bauzas | sean-k-mooney: looks like Stackalytics gives me a clue https://uwsgi-docs.readthedocs.io/en/latest/Options.html?#py-call-osafterfork | |
| 13:09:38 | opendevreview | Federico Ressi proposed openstack/nova master: Check Nova project changes with Tobiko scenario test cases https://review.opendev.org/c/openstack/nova/+/806853 | |
| 13:10:01 | sean-k-mooney | maybe | |
| 13:10:18 | bauzas | anyway, not an urgent problem but we should somehow track this | |
| 13:11:17 | sean-k-mooney | https://www.techatbloomberg.com/blog/configuring-uwsgi-production-deployment/ | |
| 13:12:11 | sean-k-mooney | This feature should be enabled by default because we expect processes to respond to signals that are sent to them. Without enabling this feature, the first developer to attempt to trap signals in a uWSGI-hosted service is going to be confused until they find this option. That could cause a delay of anywhere from 0 minutes to days, depending on how good someone is at Googling | |
| 13:12:13 | sean-k-mooney | or asking coworkers for their opinions. | |
| 13:12:17 | sean-k-mooney | ^ | |
| 13:15:13 | gibi | yupp that feels like a doc bug to suggest using py-call-osafterfork = true | |