| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-05-06 | |||
| 14:29:46 | dansmith | ah, I see | |
| 14:30:16 | dansmith | seems like we could maybe just disable the pooling there, or just destroy the threadpool at the end of service.create before we return to allow the fork to be clean? | |
| 14:31:21 | gibi | disable pooling needs infra as the version check uses the generic scatter-gather. I would not duplicate that | |
| 14:31:42 | gibi | destroying the pool before the fork could work | |
| 14:31:51 | gibi | probably a bit cleaner than the re-init | |
| 14:31:54 | gibi | but | |
| 14:32:03 | gibi | I need to test it | |
| 14:32:46 | dansmith | for the disable pooling suggestion, I meant something like "use a temporary pool instead of the global one" - I know we need *a* pool there | |
| 14:33:34 | dansmith | you could also have a "sequential thread pool" that just runs each cell query synchronously in order instead of the threads maybe too | |
| 14:33:46 | dansmith | but yeah, just destroying after we're done with that check would be simple and clean, I suspect | |
| 14:34:11 | dansmith | I will miss the nova meeting today, btw. see you later | |
| 14:36:14 | sean-k-mooney | gibi: if notthing else it confirm that the issue is with how it forking and not reinitalisting it | |
| 14:36:48 | sean-k-mooney | gibi: dansmith i was on a metting but would markign the pool as thread local help | |
| 14:37:25 | sean-k-mooney | they would not be shared with the process correct. the only problem with that is if work on a thread pool wanted to add somethign to the same thread pool | |
| 14:37:56 | sean-k-mooney | my thinking is we should not have that patter in general | |
| 14:38:17 | sean-k-mooney | so if we mark the module level thread pool at threadlocal then on fork it will be initalsed to empty | |
| 14:38:39 | sean-k-mooney | and the first attempt to use it will init it in the child process without any shareing of state | |
| 14:39:24 | sean-k-mooney | having a way to disable scatter gahter for that first call on startup would also be an option | |
| 14:40:40 | sean-k-mooney | dansmith: by "sequential thread pool" you mean a sequtial executor right. like the one we used in tests | |
| 14:41:06 | dansmith | yes | |
| 14:41:55 | sean-k-mooney | ok futureist already provides that https://github.com/openstack/futurist/blob/master/futurist/_futures.py#L227 | |
| 14:42:14 | sean-k-mooney | so that would be an option for the early init | |
| 14:50:35 | sean-k-mooney | gibi: i comment with links to the thread local apporch but i have not tested that change it mich be worth trying with a DNM patch or locally | |
| 14:50:51 | sean-k-mooney | im just not sure if its the correct patthern to apply to this type of problem | |
| 14:51:26 | sean-k-mooney | i.e. can we convert all module global state to thread local storage if we have this problem or would that only work in this case | |
| 14:51:54 | sean-k-mooney | we dotn really want to have to have thread local db engine facades for example | |
| 14:54:31 | opendevreview | Stephen Finucane proposed openstack/placement master: setup: Remove pbr's wsgi_scripts https://review.opendev.org/c/openstack/placement/+/919582 | |
| 14:54:48 | stephenfin | sean-k-mooney: Think you can remove your -1 on that now? ^ | |
| 14:55:22 | sean-k-mooney | probaly we had that supprot in epoxy right | |
| 14:55:35 | sean-k-mooney | so we should be able to remove it now if we wanted too | |
| 14:58:08 | stephenfin | Yep | |
| 14:58:33 | sean-k-mooney | stephenfin: im still hesitent to remvoe this entrily for the simple reason that apache mod_wsgi does not suport using the module approch supproted by uwsgi/gunicorn | |
| 14:58:37 | sean-k-mooney | with that said | |
| 14:58:47 | sean-k-mooney | can the wsgi module be used directly | |
| 14:59:17 | sean-k-mooney | i.e. can you just poing mod_wsgi at https://github.com/openstack/placement/blob/master/placement/wsgi/api.py | |
| 14:59:49 | sean-k-mooney | that effectivly what was generated by pbr right | |
| 15:00:12 | sean-k-mooney | so for modwsgi you would just point to that in site packages ? | |
| 15:02:18 | sean-k-mooney | ok so not quite https://termbin.com/5n98 | |
| 15:02:35 | sean-k-mooney | but for the mod_wsgi usecase the answer is yes | |
| 15:02:53 | sean-k-mooney | care to call that out in the release note? | |
| 15:05:34 | sean-k-mooney | hum | |
| 15:05:36 | sean-k-mooney | python3 -c "import placement.wsgi.api; print(placement.wsgi.api.__file__);" | |
| 15:05:50 | sean-k-mooney | so because we dont card that with if __main__ | |
| 15:06:24 | sean-k-mooney | that import actully runs some of the code | |
| 15:07:16 | sean-k-mooney | stephenfin: anyway im not sure if we need to expalin how to locate /opt/stack/placement/placement/wsgi/api.py on the system | |
| 15:07:41 | sean-k-mooney | but instead of /opt/stack/data/venv/bin/placement-api they just need to use ^ | |
| 15:09:32 | sean-k-mooney | if this was not devstack that would be in /usr/lib/python3/dist-packages/ or /usr/lib/python3/site-packages/ | |
| 15:10:42 | Uggla | Nova meeting in ~50mn | |
| 15:49:43 | Uggla | Nova meeting in ~10mn, time for you to grab a cup of coffee. | |
| 15:53:25 | gibi | If I drink a coffee no then won't sleep until 2 in the morning | |
| 15:54:24 | gibi | sean-k-mooney: the thread local has a problem that if we have a periodic that wants to run a scatter-gather then the thread holding the periodic will see a different scatter-gather pool than the main one | |
| 15:55:03 | gibi | so I will look into the destroy pool before fork idea | |
| 15:57:56 | sean-k-mooney | i think there is a generic hook we can implemnt for that | |
| 15:58:10 | sean-k-mooney | we have some module levle reset functionaltiy that we use for mutable config | |
| 15:58:23 | sean-k-mooney | im thinking of thing like atexit() | |
| 15:58:44 | sean-k-mooney | there might be a pre/post fork hook we could register to do that | |
| 16:00:51 | gibi | https://github.com/sileht/cotyledon/blob/be444189de32a8c29c7107a9b02da44248a7e64a/cotyledon/_service_manager.py#L156 cotyledon has hooks | |
| 16:02:07 | opendevmeet | The meeting name has been set to 'nova' | |
| 16:02:07 | opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | |
| 16:02:07 | opendevmeet | Meeting started Tue May 6 16:02:07 2025 UTC and is due to finish in 60 minutes. The chair is Uggla. Information about MeetBot at http://wiki.debian.org/MeetBot. | |
| 16:02:07 | Uggla | #startmeeting nova | |
| 16:02:14 | bauzas | \o | |
| 16:02:17 | Uggla | Hello everyone | |
| 16:02:27 | Uggla | awaiting a moment for people to join. | |
| 16:02:50 | elodilles | o/ | |
| 16:02:54 | fwiesel | o/ | |
| 16:03:17 | gmaan | o/ | |
| 16:04:03 | Uggla | thanks bauzas for last week meeting. | |
| 16:04:11 | gibi | o/ | |
| 16:04:11 | bauzas | np | |
| 16:04:47 | Uggla | #topic Bugs (stuck/critical) | |
| 16:04:53 | Uggla | #info No Critical bug | |
| 16:05:01 | Uggla | #topic Gate status | |
| 16:05:11 | Uggla | #link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs | |
| 16:05:19 | Uggla | #link https://etherpad.opendev.org/p/nova-ci-failures-minimal | |
| 16:05:33 | Uggla | #link https://zuul.openstack.org/builds?project=openstack%2Fnova&project=openstack%2Fplacement&branch=stable%2F*&branch=master&pipeline=periodic-weekly&skip=0 Nova&Placement periodic jobs status | |
| 16:05:39 | Uggla | #info Please look at the gate failures and file a bug report with the gate-failure tag. | |
| 16:05:44 | Uggla | #info Please try to provide meaningful comment when you recheck | |
| 16:06:13 | Uggla | If I understood well, the gate was blocked by this https://review.opendev.org/c/openstack/nova/+/948392 last week. | |
| 16:06:27 | gibi | jepp | |
| 16:06:30 | gibi | fix is landed | |
| 16:06:31 | Uggla | It is landed now. | |
| 16:06:48 | Uggla | It the gate looks good. Please tell me if I'm wrong. | |
| 16:06:54 | gibi | I have hit by https://bugs.launchpad.net/glance/+bug/2109428 multiple times recently. Not a blocker but definitly a source of rechecks | |
| 16:06:54 | sean-k-mooney | the cyborg and barbican jobs are still broken i think | |
| 16:07:05 | gibi | sean-k-mooney: yepp those non-votings are broken | |
| 16:07:08 | gibi | bugs are filed | |
| 16:07:25 | sean-k-mooney | i may try and find time to go fix them | |
| 16:07:35 | sean-k-mooney | they are both broken on the lack of a pyproject.toml | |
| 16:07:40 | gibi | yepp | |
| 16:07:47 | Uggla | good to know thx gibi | |
| 16:07:51 | gibi | https://bugs.launchpad.net/barbican/+bug/2109584 | |
| 16:07:54 | gibi | https://bugs.launchpad.net/openstack-cyborg/+bug/2109583 | |
| 16:07:56 | sean-k-mooney | if i can fix it without having to install the service locally i might give it a try | |
| 16:08:01 | sean-k-mooney | i can refence thos bugs | |
| 16:08:07 | gibi | thanks | |
| 16:08:18 | gmaan | I think we need to make pyproject.toml for all projects in openstack otherwise they will break slowly at some point | |
| 16:08:30 | gibi | gmaan: I agree | |
| 16:09:21 | Uggla | should we track this ? | |
| 16:09:41 | sean-k-mooney | gmaan: yes we will | |