| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-09-13 | |||
| 12:20:25 | gibi | it only fails if you db already under alembic control | |
| 12:20:43 | sean-k-mooney | right so with an empty db it wont fail | |
| 12:20:45 | gibi | so grenade did not detect it as it upgrades from legacy to alembic not alembic to alembiv | |
| 12:20:47 | bauzas | sean-k-mooney: we have functional tests for nova-manage, nope ? | |
| 12:20:56 | gibi | bauzas: not for db version | |
| 12:21:02 | bauzas | ah right shit | |
| 12:21:16 | gibi | becuase that is "simple" it just calls alembic ;) | |
| 12:54:40 | bauzas | mmm, looks like we no longer generate guru meditation reports using USR2 signal ? | |
| 12:54:51 | bauzas | it's just recycling the service | |
| 12:55:00 | bauzas | gibi: can you test ? | |
| 12:55:10 | bauzas | (context, other escalation meanwhile) | |
| 12:55:32 | sean-k-mooney | bauzas: it shoudl dump it into the log | |
| 12:56:14 | sean-k-mooney | bauzas: so when you do the kill --USR2 it should output the gmr to the logs | |
| 12:56:32 | sean-k-mooney | not to standard out of the terminal you run kill in | |
| 12:57:01 | sean-k-mooney | its possibel it coudl end up in the container standar out but im 99% sure it went to the logs | |
| 12:57:17 | bauzas | sean-k-mooney: gibi: no, it's recycling | |
| 12:57:22 | bauzas | sean-k-mooney: gibi: https://paste.opendev.org/show/809280/ | |
| 12:57:40 | bauzas | it should dump the stateinto the logs | |
| 12:57:51 | bauzas | but rather, it gets the signal and then dies | |
| 12:58:02 | sean-k-mooney | that is to the wsgi server | |
| 12:58:05 | bauzas | I guess this is uswsgi related | |
| 12:58:12 | sean-k-mooney | ya | |
| 12:58:18 | sean-k-mooney | have you tried it with nova-compute | |
| 12:58:52 | sean-k-mooney | for uwsgi i dont know if it ever worked but you might need to do it to the worker process | |
| 12:59:55 | bauzas | sean-k-mooney: I can run it against n-cpu for sure | |
| 13:00:06 | sean-k-mooney | that is what the customer need to do anyway | |
| 13:00:10 | sean-k-mooney | not nova-api | |
| 13:00:32 | bauzas | sean-k-mooney: sure but I wanted to test it against the API | |
| 13:00:38 | sean-k-mooney | bauzas: i dont know if we have ever tested GMR when the api is not runnign under python | |
| 13:00:54 | sean-k-mooney | bauzas: with nova-api directly it proably works | |
| 13:01:19 | sean-k-mooney | with nova-api under uwsgi i have never seen it documented how to make that work | |
| 13:01:30 | bauzas | sean-k-mooney: okay, this works with nova-compute service directly | |
| 13:01:41 | gibi | nova-compute gmr works for me | |
| 13:03:02 | gibi | nova-api doesnt | |
| 13:03:07 | bauzas | gibi: okay, so I guess it's PEBKAC as I issued the signal against the main PID on the wsgi server | |
| 13:03:14 | bauzas | ah | |
| 13:03:40 | bauzas | so, yeah, we should configure uwsgi to *not* handle this signal | |
| 13:03:54 | bauzas | a devstack change I guess | |
| 13:04:15 | gibi | nova-api works if I run it under python directly | |
| 13:05:35 | sean-k-mooney | SIGUSR2 -USR2 prints worker status | |
| 13:05:45 | sean-k-mooney | that is what uWSGI does by default | |
| 13:06:17 | bauzas | gibi: yeah, so this is because of the server | |
| 13:06:25 | sean-k-mooney | https://uwsgi-docs-additions.readthedocs.io/en/latest/Management.html#signals-for-controlling-uwsgi | |
| 13:06:56 | sean-k-mooney | we may or may not be able to change that | |
| 13:07:13 | bauzas | https://uwsgi-docs.readthedocs.io/en/latest/Management.html#signals-for-controlling-uwsgi | |
| 13:07:16 | bauzas | this | |
| 13:07:22 | bauzas | jinxed by sean-k-mooney | |
| 13:09:15 | opendevreview | Federico Ressi proposed openstack/nova master: Debug Nova APIs call failures https://review.opendev.org/c/openstack/nova/+/806683 | |
| 13:09:30 | bauzas | sean-k-mooney: looks like Stackalytics gives me a clue https://uwsgi-docs.readthedocs.io/en/latest/Options.html?#py-call-osafterfork | |
| 13:09:38 | opendevreview | Federico Ressi proposed openstack/nova master: Check Nova project changes with Tobiko scenario test cases https://review.opendev.org/c/openstack/nova/+/806853 | |
| 13:10:01 | sean-k-mooney | maybe | |
| 13:10:18 | bauzas | anyway, not an urgent problem but we should somehow track this | |
| 13:11:17 | sean-k-mooney | https://www.techatbloomberg.com/blog/configuring-uwsgi-production-deployment/ | |
| 13:12:11 | sean-k-mooney | This feature should be enabled by default because we expect processes to respond to signals that are sent to them. Without enabling this feature, the first developer to attempt to trap signals in a uWSGI-hosted service is going to be confused until they find this option. That could cause a delay of anywhere from 0 minutes to days, depending on how good someone is at Googling | |
| 13:12:13 | sean-k-mooney | or asking coworkers for their opinions. | |
| 13:12:17 | sean-k-mooney | ^ | |
| 13:15:13 | gibi | yupp that feels like a doc bug to suggest using py-call-osafterfork = true | |
| 13:16:15 | bauzas | I'm proud of my Google fu's then :) | |
| 13:17:13 | bauzas | I should ask for a promotion "Senior Principal Software Googler" I guess | |
| 13:18:22 | bauzas | I'm used to practice it for administrative and legal paperwork | |
| 13:18:45 | bauzas | anyway, back to the regression fix you asked me to look a | |
| 13:49:23 | noonedeadpunk | hey there! | |
| 13:49:54 | noonedeadpunk | I see weird behaviour of nova-manage when I'm trying to do db_sync when ca_file is provided for mysql connection string | |
| 13:50:25 | noonedeadpunk | s/ca_file/ssl_ca/ | |
| 13:50:52 | noonedeadpunk | So eventually that's stack trace https://paste.opendev.org/show/809281/ | |
| 13:51:21 | noonedeadpunk | And all other services treat same connection string correctly | |
| 13:51:43 | noonedeadpunk | So for me it feels like there's somewhere urlencode appllied or smth like that | |
| 13:52:16 | artom | noonedeadpunk, that smells like a thing we fixed recently... | |
| 13:52:20 | noonedeadpunk | If I add quotes in the connection line - this prevents file patch from being converted, but then wrong path is passed | |
| 13:52:47 | noonedeadpunk | oh, I'm running master but with head several weeks before... | |
| 13:52:55 | noonedeadpunk | will try to update sha used | |
| 13:53:04 | artom | noonedeadpunk, https://review.opendev.org/c/openstack/nova/+/805663/9 | |
| 13:53:38 | noonedeadpunk | well, I came to smth simmilar :) thanks | |
| 13:59:18 | noonedeadpunk | artom: works nicely! | |
| 13:59:27 | artom | \o/ | |
| 14:18:21 | gibi | interestingly multiple people independently reported bugs about nova/placement does not allow migrating VMs out from overallocated computes https://bugs.launchpad.net/nova/+bug/1943191 | |
| 14:19:09 | gibi | I think we have that situation for a long time, sice we use migration allocations | |
| 14:19:22 | gibi | hm, even before | |
| 14:19:27 | gibi | since we use placement actually | |
| 15:03:03 | opendevreview | Alban Lecorps proposed openstack/nova master: VMware: Support volumes backed by VStorageObject https://review.opendev.org/c/openstack/nova/+/808791 | |
| 15:09:44 | gibi | it is fun to see how bauzas consistently clicks on Review-Priority +1 instead of Workflow +1 then lyarwood goes and fix is up for him. this is real teamwork :) | |
| 15:10:04 | gibi | btw thanks for the reviews on the nova-manage db version fix | |
| 15:10:57 | lyarwood | ^_^ np | |
| 15:23:57 | dansmith | in bauzas' defense, it breaks a decade of muscle memory, so I sympathize | |
| 15:24:06 | dansmith | I usually catch myself, but not always | |
| 15:27:18 | bauzas | gibi: holy shit | |
| 15:27:59 | bauzas | dansmith: and yeah, it was that for at least 6 years for Nova and 8 years for another project ;) | |
| 15:28:10 | bauzas | wow, time flies | |
| 15:28:52 | gibi | yeah I also guessed that it was muscle memory | |
| 15:46:36 | bauzas | gibi: if I was a astronaut, I could then have created a problem with my spacecraft :p | |
| 15:48:06 | opendevreview | Merged openstack/nova master: Reproduce bug/1943436 https://review.opendev.org/c/openstack/nova/+/808753 | |
| 15:48:10 | gibi | I hope spacex freezing the UI layout at least for each individual flight :) | |
| 16:43:29 | opendevreview | Merged openstack/nova master: Fix nova-manage db version https://review.opendev.org/c/openstack/nova/+/808712 | |
| 16:53:36 | coreycb | hello, are there any near-term plans for new stable releases of nova? | |
| 17:02:43 | elodilles | coreycb: I'm not aware of plans but it is possible to release for wallaby, victoria and ussuri as well as there are merged patches | |
| 17:02:55 | lyarwood | coreycb: elodilles normally does these, I can't see any open but we can raise it tomorrow during the team meeting | |
| 17:03:06 | lyarwood | ah there we go | |
| 17:03:09 | lyarwood | :) | |
| 17:03:40 | elodilles | :) | |
| 17:33:00 | coreycb | elodilles: lyarwood: thanks for the response and raising it at the meeting | |