BUG: Using modin with APT-installed MPI on linux doesn't work

[sfc-gh-mvashishtha]

Modin version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest released version of Modin.

• I have confirmed this bug exists on the main branch of Modin. (In order to do this you can follow this guide.)

Reproducible Example

sudo apt install libmpich-dev
python -m pip install -e "modin[mpi]"
MODIN_ENGINE=unidist UNIDIST_BACKEND=mpi mpiexec -n 1 python -c "import modin.pandas as pd; print(pd.DataFrame([1,2,3]))"

Issue Description

MPI fails to start and gives errors like Error in spawn call. Example: https://github.com/modin-project/modin/actions/runs/12753981920/job/35546821512?pr=7420#logs

Expected Behavior

modin[mpi] should run without error.

Error Logs

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/runner/work/modin/modin/modin/logging/logger_decorator.py", line 144, in run_and_log
    return obj(*args, **kwargs)
  File "/home/runner/work/modin/modin/modin/pandas/dataframe.py", line 203, in __init__
    distributed_frame = from_non_pandas(data, index, columns, dtype)
  File "/home/runner/work/modin/modin/modin/pandas/io.py", line 972, in from_non_pandas
    new_qc = FactoryDispatcher.from_non_pandas(df, index, columns, dtype)
  File "/home/runner/work/modin/modin/modin/core/execution/dispatching/factories/dispatcher.py", line 177, in from_non_pandas
    return cls.get_factory()._from_non_pandas(*args, **kwargs)
  File "/home/runner/work/modin/modin/modin/core/execution/dispatching/factories/dispatcher.py", line 115, in get_factory
    Engine.subscribe(_update_engine)
  File "/home/runner/work/modin/modin/modin/config/pubsub.py", line 295, in subscribe
    callback(cls)
  File "/home/runner/work/modin/modin/modin/pandas/__init__.py", line 133, in _update_engine
    initialize_unidist()
  File "/home/runner/work/modin/modin/modin/core/execution/unidist/common/utils.py", line 41, in initialize_unidist
    unidist.init()
  File "/opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages/unidist/api.py", line 92, in init
    init_backend()
  File "/opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages/unidist/core/base/utils.py", line 28, in init_backend
    initialize_mpi()
  File "/opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages/unidist/core/backends/mpi/utils.py", line 12, in initialize_mpi
    init()
  File "/opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages/unidist/core/backends/mpi/core/controller/api.py", line [234](https://github.com/modin-project/modin/actions/runs/12753981920/job/35546821512?pr=7420#step:6:235), in init
    intercomm = MPI.COMM_SELF.Spawn(
  File "src/mpi4py/MPI.src/Comm.pyx", line 2544, in mpi4py.MPI.Intracomm.Spawn
mpi4py.MPI.Exception: Other MPI error, error stack:
internal_Comm_spawn(77171)..: MPI_Comm_spawn(command=/opt/hostedtoolcache/Python/3.9.21/x64/bin/python, argv=0x7fd906bd26d0, maxprocs=5, info=0x9c000000, 0, MPI_COMM_SELF, intercomm=0x7fd906bd2680, array_of_errcodes=(nil)) failed
MPID_Comm_spawn_multiple(85): Error in spawn call

Installed Versions

I don't know; I don't have a linux machine, and we saw this issue in CI on ubuntu: https://github.com/modin-project/modin/actions/runs/12753981920/job/35546821512?pr=7420#logs

[anmyachev]

Thanks @sfc-gh-mvashishtha!

[anmyachev]

Related commit in mpi4py: mpi4py/mpi4py@1e4e7f2