Python3 installation update (#289)

* Fixes #277; MultiNest source corrected. * Fixes #283; Change back to requiring PyMultiNest installation from source. * Fixes xpsi environment name, and small text modifications. * Fixes #274; Mentioned that blackbody extension needed for the Post-Processing tutorial. * Fixes #275; Broken link to Modeling tutorial fixed in Synthetic data example. * Update CALMIP installation instructions resolving Issue #288 * Fixing typos and links in documentation * Links corrected to Riley+2019 in Python docstrings. --------- Co-authored-by: Sebastien Guillot <[email protected]>
xpsi-group · Feb 16, 2023 · 6557cf7 · 6557cf7
1 parent fae9075
commit 6557cf7
Show file tree

Hide file tree

Showing 13 changed files with 148 additions and 76 deletions.
diff --git a/CHANGELOG.rst b/CHANGELOG.rst
@@ -72,7 +72,7 @@ Changed
 Deprecated
 ^^^^^^^^^^
 
-*The Python2 version of X-PSI (v1.2.1) is now considered deprecated, although documentation and tutorials are still available.
+* The Python2 version of X-PSI (v1.2.1) is now considered deprecated, although documentation and tutorials are still available.
 
 Removed
 ^^^^^^^

diff --git a/docs/source/HPC_systems.rst b/docs/source/HPC_systems.rst
@@ -310,7 +310,7 @@ For example job scripts, see the Helios example in :ref:`example_job`.
 
 .. _CALMIPsystem:
 
-CALMIP (not updated for python3 yet)
+CALMIP
 ------------------------------------
 
 `CALMIP <https://www.calmip.univ-toulouse.fr>`_ is the supercomputer of `Université Fédérale de Toulouse <https://www.univ-toulouse.fr>`_
@@ -323,60 +323,105 @@ In your ``$HOME`` file system, from the login node, start by loading the necessa
 .. code-block:: bash
 
     module purge
-    module load python/2.7.14
+    module load conda
     module load cmake
     module load intel/18.2.199
     module load intelmpi/18.2
     module load gsl/2.5-icc
 
-Then, install/update the required python packages:
+Then, create the conda environnnement and Install python packages with conda (or pip):
 
 .. code-block:: bash
 
-    pip install emcee==3.0.2  --user
-    pip install --upgrade numpy --user
-    pip install --upgrade Cython --user
+    conda create -n xpsi --clone base
+    conda activate xpsi
+    conda install numpy scipy matplotlib wrapt
+    conda install cython
+    conda install h5py
+    conda install -c conda-forge fgivenx
     pip install schwimmbad --user
 
+Point to the Intel compilers
 
-Install MPI4PY in your ``$HOME``:
+.. code-block:: bash
+
+    export FC=ifort
+    export CC=icc
+    export CXX=icpc
+
+Install mpi4py in your ``$HOME`` (e.g. in ``~/Softwares``):
 
 .. code-block:: bash
 
-    wget https://bitbucket.org/mpi4py/mpi4py/downloads/mpi4py-3.0.0.tar.gz
-    tar -xvf mpi4py-3.0.0.tar.gz
-    cd mpi4py-3.0.0
-    python setup.py install --user
+    mkdir Softwares
+    cd Softwares
+    wget https://bitbucket.org/mpi4py/mpi4py/downloads/mpi4py-3.1.4.tar.gz
+    tar zxvf mpi4py-3.1.4.tar.gz
+    cd mpi4py-3.1.4
+    python setup.py build
+    python setup.py install
+    # Test on login node:
+    mpiexec -n 4 python demo/helloworld.py
 
-Download the MultiNest package in your ``$HOME``:
+
+Download and Install the MultiNest package in your ``$HOME`` (e.g. in ``~/Softwares``:
 
 .. code-block:: bash
 
-    git clone https://github.com/farhanferoz/MultiNest.git ~/multinest
-    cd ~/multinest/MultiNest_v3.11_CMake/multinest
+    cd ~/Softwares
+    git clone https://github.com/farhanferoz/MultiNest.git  ./MultiNest
+    cd MultiNest/MultiNest_v3.12_CMake/multinest/
     mkdir build
     cd build
+    cmake -DCMAKE_INSTALL_PREFIX=~/Softwares/MultiNest \
+                -DCMAKE_{C,CXX}_FLAGS="-O3 -xCORE-AVX512 -mkl" \
+                -DCMAKE_Fortran_FLAGS="-O3 -xCORE-AVX512 -mkl" \
+                -DCMAKE_C_COMPILER=mpiicc    \
+                -DCMAKE_CXX_COMPILER=mpiicpc \
+                -DCMAKE_Fortran_COMPILER=mpiifort  ..
+    make
 
+    ## Check that librairies have been compiled and are present
+    ls ../lib
 
-Compile MultiNest in your ``$HOME``, following recommendation from CALMIP support:
+Install pymultinest in your ``$HOME`` (e.g. in ``~/Softwares``:
 
 .. code-block:: bash
 
-    cmake -DCMAKE_INSTALL_PREFIX=~/multiNest \
-            -DCMAKE_{C,CXX}_FLAGS="-O3 -xCORE-AVX512 -mkl" \
-            -DCMAKE_Fortran_FLAGS="-O3 -xCORE-AVX512 -mkl" \
-            -DCMAKE_C_COMPILER=mpiicc    \
-            -DCMAKE_CXX_COMPILER=mpiicpc \
-            -DCMAKE_Fortran_COMPILER=mpiifort  ..
-    make
+    cd ~/Softwares
+    git clone https://github.com/JohannesBuchner/PyMultiNest.git ./pymultinest
+    cd pymultinest
+    python setup.py install
+
+    # Add MultiNest to Library Path to test PyMultiNest (action to do for every job to run)
+    export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HOME/Softwares/MultiNest/MultiNest_v3.12_CMake/multinest/lib
+
+    # Test pymultinest
+    mpiexec -n 2 python pymultinest_demo.py
+
+
+Clone and Install X-PSI in ~/Softwares/
+
+.. code-block:: bash
+
+    cd ~/Softwares
+    git clone https://github.com/xpsi-group/xpsi.git
+    cd xpsi/
+    LDSHARED="icc -shared" CC=icc python setup.py install
+
+    # Test installation
+    cd ~/
+    python -c "import xpsi"
+
+    ## Ignore the warnings about GetDist, NestCheck, CornerPlotter
+    ##  which are only for PostProcessing (not usually performed on HPC systems).
+
 
 Set up your library paths:
 
 .. code-block:: bash
 
-    export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HOME/multiNest/MultiNest_v3.12_CMake/multinest/lib
-    export PYTHONPATH=$HOME/.local/lib/python2.7/site-packages/:$PYTHONPATH
+    export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HOME/Softwares/MultiNest/MultiNest_v3.12_CMake/multinest/lib
     export LD_PRELOAD=$MKLROOT/lib/intel64/libmkl_core.so:$MKLROOT/lib/intel64/libmkl_sequential.so
 
-
 Note that the ``module`` commands, and the library path ``commands`` above will have to be added in your SBATCH script (see :ref:`example_job`) to execute a run.
diff --git a/docs/source/Module_generator_tutorial.rst b/docs/source/Module_generator_tutorial.rst
@@ -5,8 +5,8 @@ Module generator tutorial
 
 To develop models in X-PSI, users are recommended to to create custom derived classes for the different X-PSI model
 components (e.g. ``Prior``, ``Photosphere``, etc.) that require further modification. Examples of such `Custom` classes
-can be found in :ref:`./Modeling.ipynb`. These classes are preferably to be written as distinct modules within a project
-directory rather than in a notebook as is done in :ref:`./Modeling.ipynb`. The modules can then be imported by a `Main`
+can be found in :doc:`Modeling<Modeling>`. These classes are preferably to be written as distinct modules within a project
+directory rather than in a notebook as is done in :doc:`Modeling<Modeling>`. The modules can then be imported by a `Main`
 script, which can be executed using an MPI command in a shell to exploit parallelism for resource-intensive likelihood
 evaluations.
 

diff --git a/docs/source/Post-processing.ipynb b/docs/source/Post-processing.ipynb
@@ -11,7 +11,8 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "In this tutorial we demonstrate the functionality and posterior operations supported by the [PostProcessing](postprocessing.rst) module by operating with the example posterior samples found in ``examples/examples_fast/Outputs``. A previous version of this Post-processing tutorial operating with the posterior samples reported by [Riley et al. (2019)](https://ui.adsabs.harvard.edu/abs/2019ApJ...887L..21R/abstract) is given [here](https://github.com/xpsi-group/xpsi/blob/cff82f299da93ca97c84ffb34390ab077cc001b3/docs/source/postprocessing_tutorial.ipynb) with more post-processing examples (however, using model files which are not supported anymore)."
+    "In this tutorial we demonstrate the functionality and posterior operations supported by the [PostProcessing](postprocessing.rst) module by operating with the example posterior samples found in ``examples/examples_fast/Outputs``. A previous version of this Post-processing tutorial operating with the posterior samples reported by [Riley et al. (2019)](https://ui.adsabs.harvard.edu/abs/2019ApJ...887L..21R/abstract) is given [here](https://github.com/xpsi-group/xpsi/blob/cff82f299da93ca97c84ffb34390ab077cc001b3/docs/source/postprocessing_tutorial.ipynb) with more post-processing examples (however, using model files which are not supported anymore).\n",
+    "To run this tutorial, you should also install X-PSI with the default (blackbody) atmosphere extension module, which is compiled when installing X-PSI with default settings."
    ]
   },
   {

diff --git a/docs/source/hotregion.rst b/docs/source/hotregion.rst
@@ -19,8 +19,8 @@ Representation of the ``HotRegion`` class instantiation options ``omit`` and ``c
 Representation of a ``HotRegion`` instance configuration, when ``omit`` or ``cede`` are set to ``True``. A description of these parameters is also presented below. If ``is_antiphased = True``, the parameter ``phase_shift`` is added to :math:`\pi` radians. For example, if ``phase_shift = 0``, the omission or the superseding region (respectively on the left and right side of the image) is in anti-phase with the observer meridian (i.e., its azimuth is :math:`\pi`).
 
 .. autoclass:: xpsi.HotRegion.HotRegion
-    :members: _HotRegion__construct_cellMesh, _HotRegion__compute_rays,
-              _HotRegion__compute_cellParamVecs, embed,
+    :members: __construct_cellMesh, __compute_rays,
+              __compute_cellParamVecs, embed,
               psi, integrate, num_rays, sqrt_num_cells, leaves, phases,
               set_phases, phases_in_cycles, print_settings, num_cells,
               cede, concentric

diff --git a/docs/source/install.rst b/docs/source/install.rst
@@ -11,7 +11,7 @@ X-PSI is an open-source software package that is available on `GitHub
     git clone https://github.com/xpsi-group/xpsi.git </path/to/xpsi>
 
 In this page, we lay down the instructions for installing X-PSI and all the
-necessary pre-requisites on your local self-administered system.
+necessary prerequisites on your local self-administered system.
 
 .. note::
 
@@ -21,7 +21,7 @@ necessary pre-requisites on your local self-administered system.
 
 .. _dev_env:
 
-Prerequisite Python packages
+Prerequisite Python Packages
 ----------------------------
 
 X-PSI was originally developed in Python 2.7 and was ported to Python 3 as of 
@@ -39,7 +39,7 @@ contents are:
 
 .. code-block:: bash
 
-    name: xpsi
+    name: xpsi_py3
     channels:
         - defaults
     dependencies:
@@ -64,12 +64,12 @@ To create a virtual environment from this file:
 
      conda env create -f <path/to/xpsi>/basic_environment.yml
 
-If Conda does not solve the environment dependencies, you may need to create
+If conda does not solve the environment dependencies, you may need to create
 an environment manually via
 
 .. code-block:: bash
 
-     conda create -n xpsi
+     conda create -n xpsi_py3
 
 and then install the core dependencies listed in ``basic_environment.yml`` via
 conda.
@@ -78,25 +78,32 @@ Activate the environment as:
 
 .. code-block:: bash
 
-    conda activate xpsi
+    conda activate xpsi_py3
 
 .. note::
 
     **ALL THE FOLLOWING STEPS SHOULD BE PERFORMED IN THIS NEWLY CREATED
     ENVIRONMENT.** Pay special attention to reactivate the environment if you
     ever have to restart the kernel.
 
-Next, install the packages
-`pymultinest <https://johannesbuchner.github.io/PyMultiNest/>`_ and
-`mpi4py <http://cython.readthedocs.io/en/latest>`_ which are required for 
+Next, install
+`mpi4py <https://bitbucket.org/mpi4py/mpi4py/downloads/>`_ which is required for 
 nested sampling:
 
 .. code-block:: bash
 
-    conda install -c conda-forge mpi4py pymultinest
-    
-Note that pyMultiNest is only a wrapper and requires MultiNest to run properly.
-See below for installation instructions (:ref:`multinest`).
+    conda install -c conda-forge mpi4py
+
+
+We also need `PyMultiNest <https://github.com/JohannesBuchner/PyMultiNest>`_
+(the interface to the MultiNest library) for nested sampling.
+However, `conda install -c conda-forge pymultinest` might try
+to install dependencies in the environment,
+including binaries for MPI, BLAS/LAPACK, and a Fortran compiler,
+all in order to install MultiNest. Moreover, the MultiNest version
+listed is a minor release too low to satisfy all our needs.
+Thus, see the PyMultiNest instructions below.
+
 Then, install optional packages
 `getdist <https://getdist.readthedocs.io/en/latest/>`_,
 `h5py <https://docs.h5py.org/en/stable/index.html>`_,
@@ -117,11 +124,12 @@ In addition, some optional miscellaneous packages are:
 
 .. _nonpython:
 
-Prerequisite Non-Python Packages
---------------------------------
+Prerequisite Non-Python Packages and PyMultiNest
+------------------------------------------------
 
-X-PSI has several dependencies that are not Python packages. Build and
-install guidelines are given below.
+X-PSI has several dependencies that are not Python packages,
+or which are Python packages but need to be installed from source (PyMultiNest).
+Build and install guidelines are given below.
 
 GSL
 ^^^
@@ -178,22 +186,20 @@ system and X-PSI can be installed locally without sampling functionality, it is
 advisable to install MultiNest on your personal machine to gain experience in
 application to inexpensive test problems. In addition, to leverage some
 capabilities of sample post-processing software you 
-`require MultiNest <https://github.com/JohannesBuchner/MultiNest>`_ ``v3.12``. 
+`require MultiNest <https://github.com/farhanferoz/MultiNest>`_ ``v3.12``.
 To build the MultiNest library, you require an MPI-wrapped Fortran compiler
 (e.g.,  `openmpi-mpifort <https://anaconda.org/conda-forge/openmpi-mpifort>`_
 from Open MPI).
 
 .. note::
 
-    The following assumes you have installed PyMultiNest and mpi4py. If you
+    The following assumes you have installed mpi4py. If you
     have not already installed it through the ``environment.yml`` file, you may
-    do so e.g. via ``conda install -c conda-forge pymultinest mpi4py``.
+    do so e.g. via ``conda install -c conda-forge mpi4py``.
 
-We follow
-`this guide <https://johannesbuchner.github.io/PyMultiNest/install.html>`_ for 
-installation of MultiNest. Prerequisites for MultiNest are c and fortran 
-compilers (e.g. ``gcc``, ``gfortran``), ``cmake``, ``blas``, ``lapack``, and
-``atlas``:
+Prerequisites for MultiNest are c and fortran
+compilers (e.g. ``gcc`` and ``gfortran``), ``cmake``, ``blas``, ``lapack``, and
+``atlas``. In case missing them, they can be installed by:
 
 .. code-block:: bash
 
@@ -204,12 +210,25 @@ then navigate to it and build:
 
 .. code-block:: bash
 
-    git clone https://github.com/JohannesBuchner/MultiNest
-    cd MultiNest/build
-    cmake ..
+    git clone https://github.com/farhanferoz/MultiNest.git <path/to/clone>/multinest
+    cd <path/to/clone>/multinest/MultiNest_v3.12_CMake/multinest/
+    mkdir build
+    cd build
+    CC=gcc FC=mpif90 CXX=g++ cmake -DCMAKE_{C,CXX}_FLAGS="-O3 -march=native -funroll-loops" -DCMAKE_Fortran_FLAGS="-O3 -march=native -funroll-loops" ..
     make
+    ls ../lib/
+
+Now you need the Python interface to MultiNest:
+
+.. code-block:: bash
+
+    git clone https://github.com/JohannesBuchner/PyMultiNest.git <path/to/clone>/pymultinest
+    cd <path/to/clone>/pymultinest
+    python setup.py install [--user]
 
-Next, you need PyMultinest to interface with MultiNest. To do so, add the
+The package will be installed in your conda environment, if the environment is activated.
+In that case, the optional ``--user`` flag should be omitted.
+We also need PyMultinest to interface with MultiNest. To do so, add the
 following line to ``~/.bashrc``:
 
 .. code-block:: bash
@@ -226,12 +245,10 @@ which should import without any errors. If you get ``ERROR:   Could not load
 MultiNest library "libmultinest.so"``, that means either MultiNest was not
 succesfully installed or could not be found.  While X-PSI will run properly,
 the nested-sampling capabilities (requiring MultiNest) will crash. The user can
-use EMCEE as the back-up sampler (see example in :doc:`Modeling<Modeling>`).
+use emcee as the back-up sampler (see example in :doc:`Modeling<Modeling>`).
 Note however that the post-processing turorials have only been implemented
 for the outputs of MultiNest.
 
-The package will be installed in your Conda environment (if activated).
-
 
 X-PSI
 -----
@@ -243,7 +260,7 @@ Finally, to build and install from the X-PSI clone root, execute:
     CC=<path/to/compiler/executable> python setup.py install [--user]
 
 The ``--user`` flag is optional and specifies where the package is installed;
-if you want to install the package in a virtual environment, omit this flag.
+if you want to install the package in a virtual environment (as recommended), omit this flag.
 
 For ``icc``, you may need to prepend this command with
 ``LDSHARED="icc -shared"``. This ensures that both the compiler and linker
@@ -257,14 +274,14 @@ will not change for runtime, we state the runtime linking instructions at
 compilation in the ``setup.py`` script.
 
 A quick check of the X-PSI installation can be done with ``import xpsi``, which
-should print to screen the following:
+should print to screen something like the following:
 
 .. code-block:: bash
 
     /=============================================\
     | X-PSI: X-ray Pulse Simulation and Inference |
     |---------------------------------------------|
-    |                Version: 1.2.1               |
+    |                Version: 2.0.0               |
     |---------------------------------------------|
     |      https://xpsi-group.github.io/xpsi      |
     \=============================================/
@@ -273,6 +290,12 @@ should print to screen the following:
     Imported nestcheck version: 0.2.1
 
 
+.. note::
+
+   Importing X-PSI should not be done in the X-PSI root directory (where the ``setup.py`` file locates).
+   Otherwise, a following type of error is expected:
+   ``ImportError: cannot import name 'set_phase_interpolant' from 'xpsi.tools' (unknown location)``
+
 For a more complete verification of the X-PSI installation, you can execute
 the following: