Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2025-05-06
12:35:03 sean-k-mooney ok so we have a few option to adress that i guess.
12:35:19 sean-k-mooney we could move all this init logic later into the workers
12:35:48 sean-k-mooney you menthioned something about not usign os.fork and using spawn?
12:36:02 gibi sean-k-mooney: it depends on if we can kill the master from the worker in a consistent way. the raise_if_old_compute should stop the master process
12:36:21 gibi sean-k-mooney: we can force os.spawn
12:36:37 gibi but that hits a pickling error when it spanws the worker
12:36:40 sean-k-mooney do we have the equivlent for an thread join for processes
12:37:03 gibi the interface exists I'm not sure about the semantic
12:37:09 sean-k-mooney i.e. can we have the master wait for all the child process to exit
12:37:24 gibi also current default behavior is that if the worker dies the master creates a new worker
12:37:43 sean-k-mooney i would not expect that but ok
12:38:12 sean-k-mooney i assume that handeld in oslo.service?
12:38:31 gibi either oslo.service or cotyledon
12:38:34 gibi one of those
12:38:51 gibi I saw the logic once so I can find it again if needed
12:38:59 elodilles (thanks gibi for the +2+W o/)
12:39:15 gibi but bottom line, in some cases we want to respawn the worker, but in other cases we want to kill the master
12:39:27 gibi so we need an active information flow from worker to master
12:39:37 sean-k-mooney i do not really expect either ot take actions like that. if things die i expect that to propagate up and systemd or whatever process is maanging nova would hanel that
12:39:41 gibi to influence the logic, if we move the version check to the worker
12:39:56 gibi systemd handles the master process
12:39:58 opendevreview Arnaud Morin proposed openstack/nova master: Fix limit when instances are stuck in build_requests https://review.opendev.org/c/openstack/nova/+/947804
12:40:13 gibi so if master dies the systemd restarts that
12:40:49 gibi but if a single worker dies killing the rest of the workers and then let the master die to let systemd restart it seems dangerous without graceful shutdown
12:40:54 sean-k-mooney right and i think the master process shoudl just spawn the chile process and wiat for them to exit. any recreation of the child process i woudl expect to be handled in nova
12:41:09 sean-k-mooney its tricky however.
12:41:22 sean-k-mooney is this one of the know open issue for the eventlet remove
12:41:27 sean-k-mooney *removal
12:43:12 gibi I can a) try to patch oslo.config to support os.spawn or b) try to re-init the executors in the worker if I can detect that I'm a worker c) move the version check to the worker and try to find a way to signal the master no to respawn but to exit
12:44:33 gibi I linked the issue to the #openstack-eventlet-removal as well so maybe Herve will have some ideas too
12:44:53 sean-k-mooney looking at the eventlet oslo service code
12:45:03 sean-k-mooney the respawn logic was hardcoded
12:45:38 sean-k-mooney so while i dont really expect that to happen it seams like its how it has alwasy worked
12:45:41 stephenfin dansmith: Could you clarify what we're trying to say about nova-metadata-wsgi under local/global deployments here? https://docs.openstack.org/nova/latest/admin/cells.html#nova-metadata-api-service
12:45:50 sean-k-mooney gibi: we do have another option
12:46:15 sean-k-mooney which is to not use oslo to create teh works but to spwan them on a process pool ourselves
12:47:17 stephenfin Are we simply saying that `[api] local_metadata_per_cell` should be false for global and true for local? Because with the removal of the eventlet server, there's no way to run the metadata API in the same service as the compute (REST) API now so "standalone" doesn't make much sense
12:47:19 sean-k-mooney gibi: that would allow us to get back futures form the process pools and we can then catch the excption and decied what to do based on that
12:47:35 stephenfin (because it's always "standalone")
12:47:44 gibi yeah that is option d). But I guess other projects will have similar problems so a nova local solution is not a nice one
12:48:04 sean-k-mooney gibi: well option e woudl be do d in oslo
12:48:11 gibi sean-k-mooney: :)
12:48:42 gibi stephenfin: you are probably correct
12:48:52 sean-k-mooney gibi: neutron may have some ideas ro we could look to one of the service that has been usign coytolon or what ever it is for years
12:49:07 gibi yeah
12:49:53 sean-k-mooney stephenfin: local metadata per cell should be set to true if you have confiured the metadata agent deploy by neutron to use the local nova metadta api endpoint
12:50:35 sean-k-mooney stephenfin: but i think what the doc is saying is
12:50:58 stephenfin > The nova metadata API service must not be run as a standalone service, using the nova-metadata-wsgi service, in this case.
12:51:00 sean-k-mooney if you do that you need to deploy the dedicated nova-metadata-api wsgi app
12:51:08 sean-k-mooney im not sure why the combined one could not work however
12:51:13 stephenfin There's no combined one
12:51:24 stephenfin Not in WSGI. Only eventlet (now removed)
12:51:25 sean-k-mooney oh then that would be why i guess
12:51:31 sean-k-mooney ack
12:52:08 gibi yepp the combined one was eventlet only
12:52:40 gibi in the nova-operator we always start the metadata wsgi as separate pod either on top or on cell level
12:53:13 sean-k-mooney ok i guess we were just tryign to say if you want to run per cell metadata you need to deploy addtional metadata api (at least 1 per cell)
12:53:27 stephenfin gibi: tbc, when you say on top or cell-level you're refererring to the global and local deployment topologies described in that doc, yeah?
12:53:36 gibi yeapp
12:53:39 sean-k-mooney yep
12:53:42 stephenfin okay, sweet
12:53:57 gibi stephenfin: do you see a need for clarifying things in our doc?
12:54:18 stephenfin I think so. The sentence I quoted above is confusing IMO
12:54:21 stephenfin > The nova metadata API service must not be run as a standalone service, using the nova-metadata-wsgi service, in this case.
12:54:47 stephenfin To me, that says "you cannot use nova-metadata-wsgi" service if deploying in a global configuration
12:55:06 sean-k-mooney that was not ture even before we remvoe the combined one
12:55:10 sean-k-mooney but now it can be simplifed
12:55:16 stephenfin and since we no longer provide nova-metadata (i.e. the eventlet one), that would suggest global deployments weren't possible anymore
12:55:20 sean-k-mooney we only have the wsgi service on master
12:55:25 sean-k-mooney and only the split endpoints
12:55:27 stephenfin yeah, hence my concern
12:55:37 stephenfin but it sounds like it's just worded weirdly
12:55:39 opendevreview Merged openstack/osc-placement stable/2025.1: Add bindep.txt for ubunutu 24.04 support https://review.opendev.org/c/openstack/osc-placement/+/948867
12:55:48 sean-k-mooney we used to have seperate console script for nova-api and nvoa-api-metadata
12:55:51 stephenfin and I can rephrase like so
12:56:07 sean-k-mooney i think gibi dropped the refences to the console scripts when they were remvoed
12:56:28 stephenfin > The ``api.local_metadata_per_cell`` option must be set to ``False``
12:56:31 sean-k-mooney but keepd the verbage related to the pbr wsgi scripts
12:56:35 stephenfin Or drop the sentence entirely
12:56:40 gibi yeah I tried to cleanup the doc when the eventlet server is removed but probably missed things
12:57:02 sean-k-mooney it need to be set based on teh toplogy
12:57:55 gibi yeah, if the metadata is deployed globally (top level) the local_metadata_per_cell needs to be false, but when deployed metadata to each cell then it need to be true
12:58:15 sean-k-mooney it tecnically optiona to set this to true for the per cell deployment by the way. it default to false
12:58:37 sean-k-mooney meanign if you deploy per cell metadata they can lookup metdata for other cells if they have access to the api db to do so
12:58:47 gibi sean-k-mooney: it depends, the the local metadata has cell db access then sure it can be set to false
12:58:57 gibi yeah that
12:59:28 gibi sean-k-mooney: cotyledon has the worker respawn implemented here https://github.com/sileht/cotyledon/blob/be444189de32a8c29c7107a9b02da44248a7e64a/cotyledon/_service_manager.py#L254-L256
12:59:39 sean-k-mooney https://docs.openstack.org/nova/latest/configuration/config.html#api.local_metadata_per_cell is basically a configuration to prevnet that cross cell lookup
13:00:08 gibi yepp
13:14:09 gibi option b) aka re-initing the executor after fork works, based on the process name stored on the executor compared to the current process name. But this is ugly and nova only solution. Fortunately we will move all our spawns to executors so at least it applicable easily for now
13:38:46 opendevreview Merged openstack/osc-placement stable/2025.1: Update .gitreview for stable/2025.1 https://review.opendev.org/c/openstack/osc-placement/+/943757
14:02:24 dansmith gibi: that move to spawn soon is surprising to me, as I suspect it will be hella slow, especially for openstack code
14:02:53 dansmith gibi: I'm not sure where we care about process workers, other than the api services in standalone mode.. in wsgi mode we shouldn't be forking at all, right?
14:02:54 gibi we don't spawn much, just at the start to get the worker processes (or when they die)
14:03:31 gibi dansmith: this is scheduler. The default behavior of oslo.service is to fork the workers
14:03:31 dansmith oh I guess conductor needs process workers too
14:03:46 gibi conductor too
14:04:00 dansmith do we need to do that for scheduler though? we did under eventlet for parallelism, but I'm not sure either do going forward
14:05:10 gibi threading in python 3.12 still limited by the GIL, so having a way to spawn processes as scaling make sense as that way we can saturate more CPU cores if needed
14:06:02 dansmith sure, but there are lots of multithreaded python programs providing reasonable performance with the GIL :)

Earlier   Later