| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-09-13 | |||
| 12:14:46 | sean-k-mooney | cool | |
| 12:15:23 | sean-k-mooney | ill take a look although it proably does not need the co-authoured by since you drop most of what i did | |
| 12:16:02 | bauzas | gibi: wow, thanks for finding the issue | |
| 12:16:43 | gibi | sean-k-mooney: you was the one that point out where the code failed so you earned the mention :) | |
| 12:17:17 | sean-k-mooney | yeah strack traces :) | |
| 12:17:31 | gibi | bauzas: I just tried to run nova-manage db version in my devstack this morning, and found things to for today :) | |
| 12:17:58 | bauzas | gibi: I just wonder why we had no find the issue with the existing tests | |
| 12:18:09 | bauzas | hadn't* | |
| 12:18:38 | sean-k-mooney | bauzas: because we would be monkey patching the calls to the alembic lib | |
| 12:19:05 | sean-k-mooney | so it would only show up in integration style test or maybe functional | |
| 12:19:11 | sean-k-mooney | not in unit test | |
| 12:20:02 | sean-k-mooney | bauzas: also i think gibi mentioned it only happens the second time you call sync | |
| 12:20:16 | sean-k-mooney | when its using alembic? | |
| 12:20:25 | gibi | it only fails if you db already under alembic control | |
| 12:20:43 | sean-k-mooney | right so with an empty db it wont fail | |
| 12:20:45 | gibi | so grenade did not detect it as it upgrades from legacy to alembic not alembic to alembiv | |
| 12:20:47 | bauzas | sean-k-mooney: we have functional tests for nova-manage, nope ? | |
| 12:20:56 | gibi | bauzas: not for db version | |
| 12:21:02 | bauzas | ah right shit | |
| 12:21:16 | gibi | becuase that is "simple" it just calls alembic ;) | |
| 12:54:40 | bauzas | mmm, looks like we no longer generate guru meditation reports using USR2 signal ? | |
| 12:54:51 | bauzas | it's just recycling the service | |
| 12:55:00 | bauzas | gibi: can you test ? | |
| 12:55:10 | bauzas | (context, other escalation meanwhile) | |
| 12:55:32 | sean-k-mooney | bauzas: it shoudl dump it into the log | |
| 12:56:14 | sean-k-mooney | bauzas: so when you do the kill --USR2 it should output the gmr to the logs | |
| 12:56:32 | sean-k-mooney | not to standard out of the terminal you run kill in | |
| 12:57:01 | sean-k-mooney | its possibel it coudl end up in the container standar out but im 99% sure it went to the logs | |
| 12:57:17 | bauzas | sean-k-mooney: gibi: no, it's recycling | |
| 12:57:22 | bauzas | sean-k-mooney: gibi: https://paste.opendev.org/show/809280/ | |
| 12:57:40 | bauzas | it should dump the stateinto the logs | |
| 12:57:51 | bauzas | but rather, it gets the signal and then dies | |
| 12:58:02 | sean-k-mooney | that is to the wsgi server | |
| 12:58:05 | bauzas | I guess this is uswsgi related | |
| 12:58:12 | sean-k-mooney | ya | |
| 12:58:18 | sean-k-mooney | have you tried it with nova-compute | |
| 12:58:52 | sean-k-mooney | for uwsgi i dont know if it ever worked but you might need to do it to the worker process | |
| 12:59:55 | bauzas | sean-k-mooney: I can run it against n-cpu for sure | |
| 13:00:06 | sean-k-mooney | that is what the customer need to do anyway | |
| 13:00:10 | sean-k-mooney | not nova-api | |
| 13:00:32 | bauzas | sean-k-mooney: sure but I wanted to test it against the API | |
| 13:00:38 | sean-k-mooney | bauzas: i dont know if we have ever tested GMR when the api is not runnign under python | |
| 13:00:54 | sean-k-mooney | bauzas: with nova-api directly it proably works | |
| 13:01:19 | sean-k-mooney | with nova-api under uwsgi i have never seen it documented how to make that work | |
| 13:01:30 | bauzas | sean-k-mooney: okay, this works with nova-compute service directly | |
| 13:01:41 | gibi | nova-compute gmr works for me | |
| 13:03:02 | gibi | nova-api doesnt | |
| 13:03:07 | bauzas | gibi: okay, so I guess it's PEBKAC as I issued the signal against the main PID on the wsgi server | |
| 13:03:14 | bauzas | ah | |
| 13:03:40 | bauzas | so, yeah, we should configure uwsgi to *not* handle this signal | |
| 13:03:54 | bauzas | a devstack change I guess | |
| 13:04:15 | gibi | nova-api works if I run it under python directly | |
| 13:05:35 | sean-k-mooney | SIGUSR2 -USR2 prints worker status | |
| 13:05:45 | sean-k-mooney | that is what uWSGI does by default | |
| 13:06:17 | bauzas | gibi: yeah, so this is because of the server | |
| 13:06:25 | sean-k-mooney | https://uwsgi-docs-additions.readthedocs.io/en/latest/Management.html#signals-for-controlling-uwsgi | |
| 13:06:56 | sean-k-mooney | we may or may not be able to change that | |
| 13:07:13 | bauzas | https://uwsgi-docs.readthedocs.io/en/latest/Management.html#signals-for-controlling-uwsgi | |
| 13:07:16 | bauzas | this | |
| 13:07:22 | bauzas | jinxed by sean-k-mooney | |
| 13:09:15 | opendevreview | Federico Ressi proposed openstack/nova master: Debug Nova APIs call failures https://review.opendev.org/c/openstack/nova/+/806683 | |
| 13:09:30 | bauzas | sean-k-mooney: looks like Stackalytics gives me a clue https://uwsgi-docs.readthedocs.io/en/latest/Options.html?#py-call-osafterfork | |
| 13:09:38 | opendevreview | Federico Ressi proposed openstack/nova master: Check Nova project changes with Tobiko scenario test cases https://review.opendev.org/c/openstack/nova/+/806853 | |
| 13:10:01 | sean-k-mooney | maybe | |
| 13:10:18 | bauzas | anyway, not an urgent problem but we should somehow track this | |
| 13:11:17 | sean-k-mooney | https://www.techatbloomberg.com/blog/configuring-uwsgi-production-deployment/ | |
| 13:12:11 | sean-k-mooney | This feature should be enabled by default because we expect processes to respond to signals that are sent to them. Without enabling this feature, the first developer to attempt to trap signals in a uWSGI-hosted service is going to be confused until they find this option. That could cause a delay of anywhere from 0 minutes to days, depending on how good someone is at Googling | |
| 13:12:13 | sean-k-mooney | or asking coworkers for their opinions. | |
| 13:12:17 | sean-k-mooney | ^ | |
| 13:15:13 | gibi | yupp that feels like a doc bug to suggest using py-call-osafterfork = true | |
| 13:16:15 | bauzas | I'm proud of my Google fu's then :) | |
| 13:17:13 | bauzas | I should ask for a promotion "Senior Principal Software Googler" I guess | |
| 13:18:22 | bauzas | I'm used to practice it for administrative and legal paperwork | |
| 13:18:45 | bauzas | anyway, back to the regression fix you asked me to look a | |
| 13:49:23 | noonedeadpunk | hey there! | |
| 13:49:54 | noonedeadpunk | I see weird behaviour of nova-manage when I'm trying to do db_sync when ca_file is provided for mysql connection string | |
| 13:50:25 | noonedeadpunk | s/ca_file/ssl_ca/ | |
| 13:50:52 | noonedeadpunk | So eventually that's stack trace https://paste.opendev.org/show/809281/ | |
| 13:51:21 | noonedeadpunk | And all other services treat same connection string correctly | |
| 13:51:43 | noonedeadpunk | So for me it feels like there's somewhere urlencode appllied or smth like that | |
| 13:52:16 | artom | noonedeadpunk, that smells like a thing we fixed recently... | |
| 13:52:20 | noonedeadpunk | If I add quotes in the connection line - this prevents file patch from being converted, but then wrong path is passed | |
| 13:52:47 | noonedeadpunk | oh, I'm running master but with head several weeks before... | |
| 13:52:55 | noonedeadpunk | will try to update sha used | |
| 13:53:04 | artom | noonedeadpunk, https://review.opendev.org/c/openstack/nova/+/805663/9 | |
| 13:53:38 | noonedeadpunk | well, I came to smth simmilar :) thanks | |
| 13:59:18 | noonedeadpunk | artom: works nicely! | |
| 13:59:27 | artom | \o/ | |
| 14:18:21 | gibi | interestingly multiple people independently reported bugs about nova/placement does not allow migrating VMs out from overallocated computes https://bugs.launchpad.net/nova/+bug/1943191 | |
| 14:19:09 | gibi | I think we have that situation for a long time, sice we use migration allocations | |
| 14:19:22 | gibi | hm, even before | |
| 14:19:27 | gibi | since we use placement actually | |
| 15:03:03 | opendevreview | Alban Lecorps proposed openstack/nova master: VMware: Support volumes backed by VStorageObject https://review.opendev.org/c/openstack/nova/+/808791 | |
| 15:09:44 | gibi | it is fun to see how bauzas consistently clicks on Review-Priority +1 instead of Workflow +1 then lyarwood goes and fix is up for him. this is real teamwork :) | |
| 15:10:04 | gibi | btw thanks for the reviews on the nova-manage db version fix | |
| 15:10:57 | lyarwood | ^_^ np | |
| 15:23:57 | dansmith | in bauzas' defense, it breaks a decade of muscle memory, so I sympathize | |
| 15:24:06 | dansmith | I usually catch myself, but not always | |
| 15:27:18 | bauzas | gibi: holy shit | |
| 15:27:59 | bauzas | dansmith: and yeah, it was that for at least 6 years for Nova and 8 years for another project ;) | |