| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-05-05 | |||
| 14:57:24 | opendevreview | Balazs Gibizer proposed openstack/nova master: Make nova.utils.pass_context private https://review.opendev.org/c/openstack/nova/+/948188 | |
| 14:57:24 | opendevreview | Balazs Gibizer proposed openstack/nova master: Rename DEFAULT_GREEN_POOL to DEFAULT_EXECUTOR https://review.opendev.org/c/openstack/nova/+/948086 | |
| 14:57:25 | opendevreview | Balazs Gibizer proposed openstack/nova master: Make the default executor configurable https://review.opendev.org/c/openstack/nova/+/948087 | |
| 14:57:26 | opendevreview | Balazs Gibizer proposed openstack/nova master: Print ThreadPool statistics https://review.opendev.org/c/openstack/nova/+/948340 | |
| 14:57:26 | opendevreview | Balazs Gibizer proposed openstack/nova master: WIP: allow service to start with threading https://review.opendev.org/c/openstack/nova/+/948311 | |
| 14:57:28 | opendevreview | Balazs Gibizer proposed openstack/nova master: DNM:Run nova-next with n-sch in threading mode https://review.opendev.org/c/openstack/nova/+/948450 | |
| 15:16:52 | opendevreview | Kamil Sambor proposed openstack/nova master: Replace eventlet semaphores with threading one https://review.opendev.org/c/openstack/nova/+/948437 | |
| #openstack-nova - 2025-05-06 | |||
| 05:23:15 | opendevreview | Michael Still proposed openstack/nova master: libvirt: Add extra spec for sound device. https://review.opendev.org/c/openstack/nova/+/926126 | |
| 05:23:16 | opendevreview | Michael Still proposed openstack/nova master: Protect older compute managers from sound model requests. https://review.opendev.org/c/openstack/nova/+/940770 | |
| 05:23:16 | opendevreview | Michael Still proposed openstack/nova master: libvirt: Add extra specs for USB redirection. https://review.opendev.org/c/openstack/nova/+/927354 | |
| 08:06:10 | zigo | haleyb: Thanks. | |
| 08:06:50 | zigo | haleyb: Can you give me your email address and the one of Billy (wolsen) please ? | |
| 08:07:14 | zigo | haleyb: I would welcome you guys to join #debian-openstack, and #debian-openstack-commits, which is were I discuss OpenStack packaging. | |
| 09:45:56 | opendevreview | Arnaud Morin proposed openstack/nova master: Fix limit when instances are stuck in build_requests https://review.opendev.org/c/openstack/nova/+/947804 | |
| 12:06:53 | opendevreview | Elod Illes proposed openstack/osc-placement stable/2025.1: Add bindep.txt for ubunutu 24.04 support https://review.opendev.org/c/openstack/osc-placement/+/948867 | |
| 12:18:18 | gibi | sean-k-mooney[m] dansmith FYI: the first real complication with threading https://review.opendev.org/c/openstack/oslo.service/+/945720/comments/fb5d6632_f0eaf102 | |
| 12:20:02 | elodilles | hi stable maintainers, may i get a +2+W for this simple clean cherry pick that fixes osc-placement's stable/2025.1 gate? o:) https://review.opendev.org/c/openstack/osc-placement/+/948867 | |
| 12:21:39 | sean-k-mooney | gibi: that not really related to threading | |
| 12:22:14 | sean-k-mooney | that related to multi processing and how cotyledon is handling workers | |
| 12:22:31 | gibi | sean-k-mooney: the python doc says do not mix threading with os.fork | |
| 12:22:32 | gibi | we do | |
| 12:22:44 | gibi | and I see the bad results of it | |
| 12:23:04 | sean-k-mooney | well we were not but we are now | |
| 12:23:07 | gibi | if I force os.spawn as suggested then I see the pickling error due to a lambda in oslo.config | |
| 12:23:57 | sean-k-mooney | so even without the eventlet removal work | |
| 12:24:36 | sean-k-mooney | if we enabel the cotyledon backend it would fail in the same way | |
| 12:25:07 | sean-k-mooney | at least in the places we are spawnign thread today if they interacted with a thread pool | |
| 12:26:35 | sean-k-mooney | im not if we really shoudl be using worker process instead of worker threads | |
| 12:27:34 | gibi | we have limited number of thread pools and those are started in the worker process I guess making it no problem. Now that our default ThreadPoolExecutor is also threaded, and that is starting in the master process, os.fork become a problem | |
| 12:28:21 | gibi | I will see if the only pickling error is the oslo.config lamda, and if I can patch that out, but if not then we have a nice complication in our hand. | |
| 12:28:44 | sean-k-mooney | well my point is i think this is perhaps a bug on oslo.service | |
| 12:29:04 | sean-k-mooney | since the new backend does not behave the same as the old processlauncher | |
| 12:29:06 | gibi | it is not a bug it is a behavior of os.fork. The child inherits the parent's state | |
| 12:29:24 | sean-k-mooney | although maybe it is the same https://github.com/openstack/oslo.service/blob/master/oslo_service/backend/eventlet/service.py#L556 | |
| 12:30:18 | sean-k-mooney | gibi: sure but the fork is ment ot happen before we ever create any of the executors | |
| 12:30:36 | gibi | nope | |
| 12:30:44 | gibi | let me link it to you... | |
| 12:31:01 | sean-k-mooney | the thread pools are not initalised until there first use right? | |
| 12:31:11 | gibi | right | |
| 12:31:19 | gibi | this is the point of the fork https://github.com/openstack/nova/blob/a5bcaf69b1a80d4d02fe092900471a6e7a28e292/nova/cmd/scheduler.py#L51 | |
| 12:31:21 | sean-k-mooney | which happen after the fork if there is one | |
| 12:31:32 | gibi | but Service.create() is before that | |
| 12:31:45 | gibi | and that already depends on our executors and therefore initialize them | |
| 12:31:50 | sean-k-mooney | which threadpool is causign the error | |
| 12:32:23 | sean-k-mooney | is it the oslo one or one of the ones we are creating | |
| 12:32:39 | gibi | in the scheduler startup both the default and the scatter gather pool is initialized before the fork by Service().create | |
| 12:33:02 | sean-k-mooney | ok i would not expect either to be sued at this point | |
| 12:33:26 | sean-k-mooney | the service creat is going to hit cell0 | |
| 12:33:34 | sean-k-mooney | for the schduler at least | |
| 12:33:49 | gibi | Service.create calls https://github.com/openstack/nova/blob/a5bcaf69b1a80d4d02fe092900471a6e7a28e292/nova/service.py#L258 which calls scatter-gather | |
| 12:34:19 | sean-k-mooney | ah i see | |
| 12:34:53 | gibi | the default pool is a bit more complicated but also used for the Schedulers of async init | |
| 12:35:03 | sean-k-mooney | ok so we have a few option to adress that i guess. | |
| 12:35:19 | sean-k-mooney | we could move all this init logic later into the workers | |
| 12:35:48 | sean-k-mooney | you menthioned something about not usign os.fork and using spawn? | |
| 12:36:02 | gibi | sean-k-mooney: it depends on if we can kill the master from the worker in a consistent way. the raise_if_old_compute should stop the master process | |
| 12:36:21 | gibi | sean-k-mooney: we can force os.spawn | |
| 12:36:37 | gibi | but that hits a pickling error when it spanws the worker | |
| 12:36:40 | sean-k-mooney | do we have the equivlent for an thread join for processes | |
| 12:37:03 | gibi | the interface exists I'm not sure about the semantic | |
| 12:37:09 | sean-k-mooney | i.e. can we have the master wait for all the child process to exit | |
| 12:37:24 | gibi | also current default behavior is that if the worker dies the master creates a new worker | |
| 12:37:43 | sean-k-mooney | i would not expect that but ok | |
| 12:38:12 | sean-k-mooney | i assume that handeld in oslo.service? | |
| 12:38:31 | gibi | either oslo.service or cotyledon | |
| 12:38:34 | gibi | one of those | |
| 12:38:51 | gibi | I saw the logic once so I can find it again if needed | |
| 12:38:59 | elodilles | (thanks gibi for the +2+W o/) | |
| 12:39:15 | gibi | but bottom line, in some cases we want to respawn the worker, but in other cases we want to kill the master | |
| 12:39:27 | gibi | so we need an active information flow from worker to master | |
| 12:39:37 | sean-k-mooney | i do not really expect either ot take actions like that. if things die i expect that to propagate up and systemd or whatever process is maanging nova would hanel that | |
| 12:39:41 | gibi | to influence the logic, if we move the version check to the worker | |
| 12:39:56 | gibi | systemd handles the master process | |
| 12:39:58 | opendevreview | Arnaud Morin proposed openstack/nova master: Fix limit when instances are stuck in build_requests https://review.opendev.org/c/openstack/nova/+/947804 | |
| 12:40:13 | gibi | so if master dies the systemd restarts that | |
| 12:40:49 | gibi | but if a single worker dies killing the rest of the workers and then let the master die to let systemd restart it seems dangerous without graceful shutdown | |
| 12:40:54 | sean-k-mooney | right and i think the master process shoudl just spawn the chile process and wiat for them to exit. any recreation of the child process i woudl expect to be handled in nova | |
| 12:41:09 | sean-k-mooney | its tricky however. | |
| 12:41:22 | sean-k-mooney | is this one of the know open issue for the eventlet remove | |
| 12:41:27 | sean-k-mooney | *removal | |
| 12:43:12 | gibi | I can a) try to patch oslo.config to support os.spawn or b) try to re-init the executors in the worker if I can detect that I'm a worker c) move the version check to the worker and try to find a way to signal the master no to respawn but to exit | |
| 12:44:33 | gibi | I linked the issue to the #openstack-eventlet-removal as well so maybe Herve will have some ideas too | |
| 12:44:53 | sean-k-mooney | looking at the eventlet oslo service code | |
| 12:45:03 | sean-k-mooney | the respawn logic was hardcoded | |
| 12:45:38 | sean-k-mooney | so while i dont really expect that to happen it seams like its how it has alwasy worked | |
| 12:45:41 | stephenfin | dansmith: Could you clarify what we're trying to say about nova-metadata-wsgi under local/global deployments here? https://docs.openstack.org/nova/latest/admin/cells.html#nova-metadata-api-service | |
| 12:45:50 | sean-k-mooney | gibi: we do have another option | |
| 12:46:15 | sean-k-mooney | which is to not use oslo to create teh works but to spwan them on a process pool ourselves | |
| 12:47:17 | stephenfin | Are we simply saying that `[api] local_metadata_per_cell` should be false for global and true for local? Because with the removal of the eventlet server, there's no way to run the metadata API in the same service as the compute (REST) API now so "standalone" doesn't make much sense | |
| 12:47:19 | sean-k-mooney | gibi: that would allow us to get back futures form the process pools and we can then catch the excption and decied what to do based on that | |
| 12:47:35 | stephenfin | (because it's always "standalone") | |
| 12:47:44 | gibi | yeah that is option d). But I guess other projects will have similar problems so a nova local solution is not a nice one | |
| 12:48:04 | sean-k-mooney | gibi: well option e woudl be do d in oslo | |
| 12:48:11 | gibi | sean-k-mooney: :) | |
| 12:48:42 | gibi | stephenfin: you are probably correct | |
| 12:48:52 | sean-k-mooney | gibi: neutron may have some ideas ro we could look to one of the service that has been usign coytolon or what ever it is for years | |
| 12:49:07 | gibi | yeah | |
| 12:49:53 | sean-k-mooney | stephenfin: local metadata per cell should be set to true if you have confiured the metadata agent deploy by neutron to use the local nova metadta api endpoint | |
| 12:50:35 | sean-k-mooney | stephenfin: but i think what the doc is saying is | |
| 12:50:58 | stephenfin | > The nova metadata API service must not be run as a standalone service, using the nova-metadata-wsgi service, in this case. | |