| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-09-13 | |||
| 10:16:52 | gibi | yepp | |
| 10:17:02 | gibi | trying to add repro / test coverage | |
| 10:24:20 | opendevreview | Merged openstack/os-vif master: Update master for stable/xena https://review.opendev.org/c/openstack/os-vif/+/808454 | |
| 10:42:47 | opendevreview | Merged openstack/placement master: Reproduce 404 when allocation queried with 1.38 https://review.opendev.org/c/openstack/placement/+/807154 | |
| 10:57:31 | gibi | sean-k-mooney: alembic_api.current does not return the version alembic_script.ScriptDirectory.from_config( self.config) | |
| 10:57:34 | gibi | sorry | |
| 10:57:36 | gibi | wrong buffer | |
| 10:57:43 | gibi | so here https://github.com/openstack/nova/blob/master/nova/db/migration.py#L173 | |
| 10:57:55 | gibi | the alembic_api.current does not return anything | |
| 10:58:02 | gibi | that function actually prints to stdout | |
| 11:02:29 | sean-k-mooney | hehe lovely | |
| 11:02:31 | sean-k-mooney | so 2 bugs | |
| 11:03:18 | sean-k-mooney | https://alembic.sqlalchemy.org/en/latest/api/commands.html#alembic.command.current right Display the current revision for a database. | |
| 11:04:02 | sean-k-mooney | we likely dont want to call the function form the command module | |
| 11:04:17 | sean-k-mooney | we likely want to get the saem from the engin or similar | |
| 11:10:12 | gibi | yepp | |
| 11:10:19 | gibi | https://alembic.sqlalchemy.org/en/latest/api/runtime.html#alembic.runtime.migration.MigrationContext | |
| 11:10:23 | gibi | I think that is the right way | |
| 11:10:40 | gibi | there is get_current_revision() on the MigrationContext object | |
| 11:10:43 | sean-k-mooney | https://alembic.sqlalchemy.org/en/latest/api/runtime.html#alembic.runtime.migration.MigrationContext.get_current_revision | |
| 11:10:45 | sean-k-mooney | ya | |
| 11:11:08 | sean-k-mooney | i saw that before when i first went looking for current | |
| 11:11:23 | sean-k-mooney | Return the current revision, usually that which is present in the alembic_version table in the database. | |
| 11:11:49 | sean-k-mooney | that ussaully is presumable becasue before you run the migration it may or may not match? | |
| 11:12:41 | sean-k-mooney | its likely good enough for our use case | |
| 11:14:01 | sean-k-mooney | zzzeek: if your around if we want to get teh current db version is get_current_revision() the correct function to call | |
| 11:57:54 | opendevreview | Merged openstack/placement master: Fix adding 'unknown' to the ConsumerTypeCache https://review.opendev.org/c/openstack/placement/+/807155 | |
| 12:11:41 | opendevreview | Balazs Gibizer proposed openstack/nova master: Reproduce bug/1943436 https://review.opendev.org/c/openstack/nova/+/808753 | |
| 12:13:07 | opendevreview | Balazs Gibizer proposed openstack/nova master: Fix nova-manage db version https://review.opendev.org/c/openstack/nova/+/808712 | |
| 12:13:23 | gibi | sean-k-mooney: ^^ | |
| 12:13:49 | gibi | I dropped your refactoring as the config object is not needed any more | |
| 12:14:00 | gibi | and the engine object was already created in scope | |
| 12:14:21 | gibi | bauzas, lyarwood: it is an RC critical fix ^^ | |
| 12:14:34 | bauzas | gibi: ack, will look | |
| 12:14:46 | sean-k-mooney | cool | |
| 12:15:23 | sean-k-mooney | ill take a look although it proably does not need the co-authoured by since you drop most of what i did | |
| 12:16:02 | bauzas | gibi: wow, thanks for finding the issue | |
| 12:16:43 | gibi | sean-k-mooney: you was the one that point out where the code failed so you earned the mention :) | |
| 12:17:17 | sean-k-mooney | yeah strack traces :) | |
| 12:17:31 | gibi | bauzas: I just tried to run nova-manage db version in my devstack this morning, and found things to for today :) | |
| 12:17:58 | bauzas | gibi: I just wonder why we had no find the issue with the existing tests | |
| 12:18:09 | bauzas | hadn't* | |
| 12:18:38 | sean-k-mooney | bauzas: because we would be monkey patching the calls to the alembic lib | |
| 12:19:05 | sean-k-mooney | so it would only show up in integration style test or maybe functional | |
| 12:19:11 | sean-k-mooney | not in unit test | |
| 12:20:02 | sean-k-mooney | bauzas: also i think gibi mentioned it only happens the second time you call sync | |
| 12:20:16 | sean-k-mooney | when its using alembic? | |
| 12:20:25 | gibi | it only fails if you db already under alembic control | |
| 12:20:43 | sean-k-mooney | right so with an empty db it wont fail | |
| 12:20:45 | gibi | so grenade did not detect it as it upgrades from legacy to alembic not alembic to alembiv | |
| 12:20:47 | bauzas | sean-k-mooney: we have functional tests for nova-manage, nope ? | |
| 12:20:56 | gibi | bauzas: not for db version | |
| 12:21:02 | bauzas | ah right shit | |
| 12:21:16 | gibi | becuase that is "simple" it just calls alembic ;) | |
| 12:54:40 | bauzas | mmm, looks like we no longer generate guru meditation reports using USR2 signal ? | |
| 12:54:51 | bauzas | it's just recycling the service | |
| 12:55:00 | bauzas | gibi: can you test ? | |
| 12:55:10 | bauzas | (context, other escalation meanwhile) | |
| 12:55:32 | sean-k-mooney | bauzas: it shoudl dump it into the log | |
| 12:56:14 | sean-k-mooney | bauzas: so when you do the kill --USR2 it should output the gmr to the logs | |
| 12:56:32 | sean-k-mooney | not to standard out of the terminal you run kill in | |
| 12:57:01 | sean-k-mooney | its possibel it coudl end up in the container standar out but im 99% sure it went to the logs | |
| 12:57:17 | bauzas | sean-k-mooney: gibi: no, it's recycling | |
| 12:57:22 | bauzas | sean-k-mooney: gibi: https://paste.opendev.org/show/809280/ | |
| 12:57:40 | bauzas | it should dump the stateinto the logs | |
| 12:57:51 | bauzas | but rather, it gets the signal and then dies | |
| 12:58:02 | sean-k-mooney | that is to the wsgi server | |
| 12:58:05 | bauzas | I guess this is uswsgi related | |
| 12:58:12 | sean-k-mooney | ya | |
| 12:58:18 | sean-k-mooney | have you tried it with nova-compute | |
| 12:58:52 | sean-k-mooney | for uwsgi i dont know if it ever worked but you might need to do it to the worker process | |
| 12:59:55 | bauzas | sean-k-mooney: I can run it against n-cpu for sure | |
| 13:00:06 | sean-k-mooney | that is what the customer need to do anyway | |
| 13:00:10 | sean-k-mooney | not nova-api | |
| 13:00:32 | bauzas | sean-k-mooney: sure but I wanted to test it against the API | |
| 13:00:38 | sean-k-mooney | bauzas: i dont know if we have ever tested GMR when the api is not runnign under python | |
| 13:00:54 | sean-k-mooney | bauzas: with nova-api directly it proably works | |
| 13:01:19 | sean-k-mooney | with nova-api under uwsgi i have never seen it documented how to make that work | |
| 13:01:30 | bauzas | sean-k-mooney: okay, this works with nova-compute service directly | |
| 13:01:41 | gibi | nova-compute gmr works for me | |
| 13:03:02 | gibi | nova-api doesnt | |
| 13:03:07 | bauzas | gibi: okay, so I guess it's PEBKAC as I issued the signal against the main PID on the wsgi server | |
| 13:03:14 | bauzas | ah | |
| 13:03:40 | bauzas | so, yeah, we should configure uwsgi to *not* handle this signal | |
| 13:03:54 | bauzas | a devstack change I guess | |
| 13:04:15 | gibi | nova-api works if I run it under python directly | |
| 13:05:35 | sean-k-mooney | SIGUSR2 -USR2 prints worker status | |
| 13:05:45 | sean-k-mooney | that is what uWSGI does by default | |
| 13:06:17 | bauzas | gibi: yeah, so this is because of the server | |
| 13:06:25 | sean-k-mooney | https://uwsgi-docs-additions.readthedocs.io/en/latest/Management.html#signals-for-controlling-uwsgi | |
| 13:06:56 | sean-k-mooney | we may or may not be able to change that | |
| 13:07:13 | bauzas | https://uwsgi-docs.readthedocs.io/en/latest/Management.html#signals-for-controlling-uwsgi | |
| 13:07:16 | bauzas | this | |
| 13:07:22 | bauzas | jinxed by sean-k-mooney | |
| 13:09:15 | opendevreview | Federico Ressi proposed openstack/nova master: Debug Nova APIs call failures https://review.opendev.org/c/openstack/nova/+/806683 | |
| 13:09:30 | bauzas | sean-k-mooney: looks like Stackalytics gives me a clue https://uwsgi-docs.readthedocs.io/en/latest/Options.html?#py-call-osafterfork | |
| 13:09:38 | opendevreview | Federico Ressi proposed openstack/nova master: Check Nova project changes with Tobiko scenario test cases https://review.opendev.org/c/openstack/nova/+/806853 | |
| 13:10:01 | sean-k-mooney | maybe | |
| 13:10:18 | bauzas | anyway, not an urgent problem but we should somehow track this | |
| 13:11:17 | sean-k-mooney | https://www.techatbloomberg.com/blog/configuring-uwsgi-production-deployment/ | |