| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-05-01 | |||
| 13:56:06 | MengyangZhang[m]1 | yes, cinder is ok with this | |
| 18:30:27 | opendevreview | Dan Smith proposed openstack/nova master: Example NVMe cleaning script for one-time-use https://review.opendev.org/c/openstack/nova/+/948446 | |
| 20:57:55 | opendevreview | Dan Smith proposed openstack/nova master: Example NVMe cleaning script for one-time-use https://review.opendev.org/c/openstack/nova/+/948446 | |
| 22:09:56 | opendevreview | melanie witt proposed openstack/nova-specs master: Re-propose vTPM live migration https://review.opendev.org/c/openstack/nova-specs/+/947542 | |
| #openstack-nova - 2025-05-02 | |||
| 11:26:23 | opendevreview | Mengyang Zhang proposed openstack/nova master: Add Burst Length Support to Cinder QoS https://review.opendev.org/c/openstack/nova/+/939490 | |
| 16:44:26 | sean-k-mooney | gibi: bauzas can ye mark https://launchpad.net/nova/bobcat as obsolete now that the eol tag has been applied and the branch deleted | |
| #openstack-nova - 2025-05-05 | |||
| 14:06:35 | opendevreview | Balazs Gibizer proposed openstack/nova master: [quota]Refactor group counting to scatter-gather https://review.opendev.org/c/openstack/nova/+/948064 | |
| 14:08:00 | gibi | dansmith: ^^ added test coverage as requested | |
| 14:16:44 | dansmith | gibi: ack, will look in a bit | |
| 14:22:14 | gibi | thanks | |
| 14:25:29 | opendevreview | Balazs Gibizer proposed openstack/nova master: WIP:Translate scatter-gather to futurist https://review.opendev.org/c/openstack/nova/+/947966 | |
| 14:30:39 | opendevreview | Balazs Gibizer proposed openstack/nova master: WIP:Translate scatter-gather to futurist https://review.opendev.org/c/openstack/nova/+/947966 | |
| 14:44:56 | opendevreview | Balazs Gibizer proposed openstack/nova master: WIP:Translate scatter-gather to futurist https://review.opendev.org/c/openstack/nova/+/947966 | |
| 14:53:36 | opendevreview | Dan Smith proposed openstack/nova master: Example NVMe cleaning script for one-time-use https://review.opendev.org/c/openstack/nova/+/948446 | |
| 14:57:21 | opendevreview | Balazs Gibizer proposed openstack/nova master: Use futurist for _get_default_green_pool() https://review.opendev.org/c/openstack/nova/+/948072 | |
| 14:57:22 | opendevreview | Balazs Gibizer proposed openstack/nova master: Replace utils.spawn_n with spawn https://review.opendev.org/c/openstack/nova/+/948076 | |
| 14:57:22 | opendevreview | Balazs Gibizer proposed openstack/nova master: Add spawn_on https://review.opendev.org/c/openstack/nova/+/948079 | |
| 14:57:23 | opendevreview | Balazs Gibizer proposed openstack/nova master: Move ComputeManager to use spawn_on https://review.opendev.org/c/openstack/nova/+/948186 | |
| 14:57:23 | opendevreview | Balazs Gibizer proposed openstack/nova master: Move ConductorManager to use spawn_on https://review.opendev.org/c/openstack/nova/+/948187 | |
| 14:57:24 | opendevreview | Balazs Gibizer proposed openstack/nova master: Make nova.utils.pass_context private https://review.opendev.org/c/openstack/nova/+/948188 | |
| 14:57:24 | opendevreview | Balazs Gibizer proposed openstack/nova master: Rename DEFAULT_GREEN_POOL to DEFAULT_EXECUTOR https://review.opendev.org/c/openstack/nova/+/948086 | |
| 14:57:25 | opendevreview | Balazs Gibizer proposed openstack/nova master: Make the default executor configurable https://review.opendev.org/c/openstack/nova/+/948087 | |
| 14:57:26 | opendevreview | Balazs Gibizer proposed openstack/nova master: Print ThreadPool statistics https://review.opendev.org/c/openstack/nova/+/948340 | |
| 14:57:26 | opendevreview | Balazs Gibizer proposed openstack/nova master: WIP: allow service to start with threading https://review.opendev.org/c/openstack/nova/+/948311 | |
| 14:57:28 | opendevreview | Balazs Gibizer proposed openstack/nova master: DNM:Run nova-next with n-sch in threading mode https://review.opendev.org/c/openstack/nova/+/948450 | |
| 15:16:52 | opendevreview | Kamil Sambor proposed openstack/nova master: Replace eventlet semaphores with threading one https://review.opendev.org/c/openstack/nova/+/948437 | |
| #openstack-nova - 2025-05-06 | |||
| 05:23:15 | opendevreview | Michael Still proposed openstack/nova master: libvirt: Add extra spec for sound device. https://review.opendev.org/c/openstack/nova/+/926126 | |
| 05:23:16 | opendevreview | Michael Still proposed openstack/nova master: Protect older compute managers from sound model requests. https://review.opendev.org/c/openstack/nova/+/940770 | |
| 05:23:16 | opendevreview | Michael Still proposed openstack/nova master: libvirt: Add extra specs for USB redirection. https://review.opendev.org/c/openstack/nova/+/927354 | |
| 08:06:10 | zigo | haleyb: Thanks. | |
| 08:06:50 | zigo | haleyb: Can you give me your email address and the one of Billy (wolsen) please ? | |
| 08:07:14 | zigo | haleyb: I would welcome you guys to join #debian-openstack, and #debian-openstack-commits, which is were I discuss OpenStack packaging. | |
| 09:45:56 | opendevreview | Arnaud Morin proposed openstack/nova master: Fix limit when instances are stuck in build_requests https://review.opendev.org/c/openstack/nova/+/947804 | |
| 12:06:53 | opendevreview | Elod Illes proposed openstack/osc-placement stable/2025.1: Add bindep.txt for ubunutu 24.04 support https://review.opendev.org/c/openstack/osc-placement/+/948867 | |
| 12:18:18 | gibi | sean-k-mooney[m] dansmith FYI: the first real complication with threading https://review.opendev.org/c/openstack/oslo.service/+/945720/comments/fb5d6632_f0eaf102 | |
| 12:20:02 | elodilles | hi stable maintainers, may i get a +2+W for this simple clean cherry pick that fixes osc-placement's stable/2025.1 gate? o:) https://review.opendev.org/c/openstack/osc-placement/+/948867 | |
| 12:21:39 | sean-k-mooney | gibi: that not really related to threading | |
| 12:22:14 | sean-k-mooney | that related to multi processing and how cotyledon is handling workers | |
| 12:22:31 | gibi | sean-k-mooney: the python doc says do not mix threading with os.fork | |
| 12:22:32 | gibi | we do | |
| 12:22:44 | gibi | and I see the bad results of it | |
| 12:23:04 | sean-k-mooney | well we were not but we are now | |
| 12:23:07 | gibi | if I force os.spawn as suggested then I see the pickling error due to a lambda in oslo.config | |
| 12:23:57 | sean-k-mooney | so even without the eventlet removal work | |
| 12:24:36 | sean-k-mooney | if we enabel the cotyledon backend it would fail in the same way | |
| 12:25:07 | sean-k-mooney | at least in the places we are spawnign thread today if they interacted with a thread pool | |
| 12:26:35 | sean-k-mooney | im not if we really shoudl be using worker process instead of worker threads | |
| 12:27:34 | gibi | we have limited number of thread pools and those are started in the worker process I guess making it no problem. Now that our default ThreadPoolExecutor is also threaded, and that is starting in the master process, os.fork become a problem | |
| 12:28:21 | gibi | I will see if the only pickling error is the oslo.config lamda, and if I can patch that out, but if not then we have a nice complication in our hand. | |
| 12:28:44 | sean-k-mooney | well my point is i think this is perhaps a bug on oslo.service | |
| 12:29:04 | sean-k-mooney | since the new backend does not behave the same as the old processlauncher | |
| 12:29:06 | gibi | it is not a bug it is a behavior of os.fork. The child inherits the parent's state | |
| 12:29:24 | sean-k-mooney | although maybe it is the same https://github.com/openstack/oslo.service/blob/master/oslo_service/backend/eventlet/service.py#L556 | |
| 12:30:18 | sean-k-mooney | gibi: sure but the fork is ment ot happen before we ever create any of the executors | |
| 12:30:36 | gibi | nope | |
| 12:30:44 | gibi | let me link it to you... | |
| 12:31:01 | sean-k-mooney | the thread pools are not initalised until there first use right? | |
| 12:31:11 | gibi | right | |
| 12:31:19 | gibi | this is the point of the fork https://github.com/openstack/nova/blob/a5bcaf69b1a80d4d02fe092900471a6e7a28e292/nova/cmd/scheduler.py#L51 | |
| 12:31:21 | sean-k-mooney | which happen after the fork if there is one | |
| 12:31:32 | gibi | but Service.create() is before that | |
| 12:31:45 | gibi | and that already depends on our executors and therefore initialize them | |
| 12:31:50 | sean-k-mooney | which threadpool is causign the error | |
| 12:32:23 | sean-k-mooney | is it the oslo one or one of the ones we are creating | |
| 12:32:39 | gibi | in the scheduler startup both the default and the scatter gather pool is initialized before the fork by Service().create | |
| 12:33:02 | sean-k-mooney | ok i would not expect either to be sued at this point | |
| 12:33:26 | sean-k-mooney | the service creat is going to hit cell0 | |
| 12:33:34 | sean-k-mooney | for the schduler at least | |
| 12:33:49 | gibi | Service.create calls https://github.com/openstack/nova/blob/a5bcaf69b1a80d4d02fe092900471a6e7a28e292/nova/service.py#L258 which calls scatter-gather | |
| 12:34:19 | sean-k-mooney | ah i see | |
| 12:34:53 | gibi | the default pool is a bit more complicated but also used for the Schedulers of async init | |
| 12:35:03 | sean-k-mooney | ok so we have a few option to adress that i guess. | |
| 12:35:19 | sean-k-mooney | we could move all this init logic later into the workers | |
| 12:35:48 | sean-k-mooney | you menthioned something about not usign os.fork and using spawn? | |
| 12:36:02 | gibi | sean-k-mooney: it depends on if we can kill the master from the worker in a consistent way. the raise_if_old_compute should stop the master process | |
| 12:36:21 | gibi | sean-k-mooney: we can force os.spawn | |
| 12:36:37 | gibi | but that hits a pickling error when it spanws the worker | |
| 12:36:40 | sean-k-mooney | do we have the equivlent for an thread join for processes | |
| 12:37:03 | gibi | the interface exists I'm not sure about the semantic | |
| 12:37:09 | sean-k-mooney | i.e. can we have the master wait for all the child process to exit | |
| 12:37:24 | gibi | also current default behavior is that if the worker dies the master creates a new worker | |
| 12:37:43 | sean-k-mooney | i would not expect that but ok | |
| 12:38:12 | sean-k-mooney | i assume that handeld in oslo.service? | |
| 12:38:31 | gibi | either oslo.service or cotyledon | |
| 12:38:34 | gibi | one of those | |
| 12:38:51 | gibi | I saw the logic once so I can find it again if needed | |
| 12:38:59 | elodilles | (thanks gibi for the +2+W o/) | |
| 12:39:15 | gibi | but bottom line, in some cases we want to respawn the worker, but in other cases we want to kill the master | |
| 12:39:27 | gibi | so we need an active information flow from worker to master | |
| 12:39:37 | sean-k-mooney | i do not really expect either ot take actions like that. if things die i expect that to propagate up and systemd or whatever process is maanging nova would hanel that | |
| 12:39:41 | gibi | to influence the logic, if we move the version check to the worker | |
| 12:39:56 | gibi | systemd handles the master process | |
| 12:39:58 | opendevreview | Arnaud Morin proposed openstack/nova master: Fix limit when instances are stuck in build_requests https://review.opendev.org/c/openstack/nova/+/947804 | |
| 12:40:13 | gibi | so if master dies the systemd restarts that | |
| 12:40:49 | gibi | but if a single worker dies killing the rest of the workers and then let the master die to let systemd restart it seems dangerous without graceful shutdown | |
| 12:40:54 | sean-k-mooney | right and i think the master process shoudl just spawn the chile process and wiat for them to exit. any recreation of the child process i woudl expect to be handled in nova | |
| 12:41:09 | sean-k-mooney | its tricky however. | |
| 12:41:22 | sean-k-mooney | is this one of the know open issue for the eventlet remove | |
| 12:41:27 | sean-k-mooney | *removal | |
| 12:43:12 | gibi | I can a) try to patch oslo.config to support os.spawn or b) try to re-init the executors in the worker if I can detect that I'm a worker c) move the version check to the worker and try to find a way to signal the master no to respawn but to exit | |