Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

document when conda support breaks down (aka GLIBC errors) #21

Open
jr3cermak opened this issue May 11, 2022 · 3 comments
Open

document when conda support breaks down (aka GLIBC errors) #21

jr3cermak opened this issue May 11, 2022 · 3 comments
Assignees
Labels
inwork should appear in a release soon

Comments

@jr3cermak
Copy link
Collaborator

The pinned versions of various pythonic items for gridtools still works. Conda support begins to break down when authors recompile prior versions with newer underlying operating systems. Pinned versions of compiled python modules will begin to fail on compute clusters with very old GLIBC libraries. Documentation to follow.

@jr3cermak jr3cermak self-assigned this May 11, 2022
jr3cermak added a commit that referenced this issue May 11, 2022
 - Issue #21: set `tile` variable to `NC_CHAR` dtype
 - Issue #21: initial fix for mosaic and exchange file handling for land vs
   no land points in ocean grid.
jr3cermak added a commit that referenced this issue May 12, 2022
 - add tests to ensure issue #21 stays resolved
 - when no land points are in the ocean grid, skip writing atmos/land exchange file
@jr3cermak
Copy link
Collaborator Author

jr3cermak commented May 17, 2022

Reloading an environment that was previously stated as working may yield a result like this:

(gridtest) jrcermakiii@chinook03:~/src/gridtools/examples$ python mkGridsExample01.py 
Traceback (most recent call last):
  File "mkGridsExample01.py", line 12, in <module>
    from gridtools.gridutils import GridUtils
  File "/import/AKWATERS/jrcermakiii/src/gridtools/gridtools/gridutils.py", line 5, in <module>
    import xarray as xr
  File "/import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/xarray/__init__.py", line 1, in <module>
    from . import testing, tutorial, ufuncs
  File "/import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/xarray/testing.py", line 8, in <module>
    from xarray.core import duck_array_ops, formatting, utils
  File "/import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/xarray/core/duck_array_ops.py", line 13, in <module>
    import pandas as pd
  File "/import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/__init__.py", line 22, in <module>
    from pandas.compat import (
  File "/import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/compat/__init__.py", line 15, in <module>
    from pandas.compat.numpy import (
  File "/import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/compat/numpy/__init__.py", line 7, in <module>
    from pandas.util.version import Version
  File "/import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/util/__init__.py", line 1, in <module>
    from pandas.util._decorators import (  # noqa
  File "/import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/util/_decorators.py", line 14, in <module>
    from pandas._libs.properties import cache_readonly  # noqa
  File "/import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/__init__.py", line 13, in <module>
    from pandas._libs.interval import Interval
ImportError: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/interval.cpython-37m-x86_64-linux-gnu.so)

This is an indication that the underlying system (or cluster) you are using is falling out of date. The supported python versions of modules managed by conda are under constant integration (CI) are are rebuilt using newer systems and hence are linked to more recent versions of GLIBC. On my particular system, the underlying GLIBC is version 2.12. One or more of the loaded python modules now requires GLIBC 2.14. Unless you have a backup of the conda environment, there is no way to recover the original operating environment. You will likely have to resort to conventional manual installation of modules to restore the environment.

To see what your conda installation is using for GLIBC, use conda info. The version of glibc should be noted in the virtual packages.

$ conda info

     active environment : gridtest
    active env location : /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest
            shell level : 1
       user config file : /home/jrcermakiii/.condarc
 populated config files : /home/jrcermakiii/.condarc
          conda version : 4.9.2
    conda-build version : not installed
         python version : 3.9.1.final.0
       virtual packages : __glibc=2.12=0
                          __unix=0=0
                          __archspec=1=x86_64
       base environment : /import/AKWATERS/jrcermakiii/local/miniconda3  (writable)
           channel URLs : https://conda.anaconda.org/pyviz/linux-64
                          https://conda.anaconda.org/pyviz/noarch
                          https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /import/AKWATERS/jrcermakiii/local/miniconda3/pkgs
                          /home/jrcermakiii/.conda/pkgs
       envs directories : /import/AKWATERS/jrcermakiii/local/miniconda3/envs
                          /home/jrcermakiii/.conda/envs
               platform : linux-64
             user-agent : conda/4.9.2 requests/2.25.0 CPython/3.9.1 Linux/2.6.32-754.35.1.el6.61015g0000.x86_64 centos/6.10 glibc/2.12
                UID:GID : 3739:2161
             netrc file : None
           offline mode : False

@jr3cermak
Copy link
Collaborator Author

jr3cermak commented May 17, 2022

The second failure mode is updating the base environment for conda. This message is typically seen when a new conda version is available.

==> WARNING: A newer version of conda exists. <==
  current version: 4.9.2
  latest version: 4.12.0

Please update conda by running

    $ conda update -n base -c defaults conda

However, due to the age of your operating system, doing the above can render your conda unusable even though the above command will succeed! You may see the following error messages after updating for one or more of the conda commands:

$ conda create -n gridtest python==3.7.10
Collecting package metadata (current_repodata.json): failed

CondaHTTPError: HTTP 000 CONNECTION FAILED for url <https://conda.anaconda.org/pyviz/linux-64/current_repodata.json>
Elapsed: -

An HTTP error occurred when trying to retrieve this URL.
HTTP errors are often intermittent, and a simple retry will get you on your way.
'https://conda.anaconda.org/pyviz/linux-64'

Again, there is no easy way to recover from this incident unless you have a backup of your conda installation directory. The only way to recover is to reinstall conda with a last known working version and reload your individual conda environments.

If conda is broken beyond repair, there may be a way to salvage your environments. If you
were fortunate to keep the miniconda shell installer archive (Miniconda3-py39_4.9.2-Linux-x86_64.sh),
then you may be in luck.

Process:

  • Deactivate any exiting environments
  • Rename the existing miniconda root to a backup
  • Remove any .bashrc references
  • Re-login to clear any lingering shell enviroment references
  • Re-install miniconda to create a new miniconda root
  • Re-login to initialize the new shell environment
  • Change directory into the backup root envs directory
  • Copy (cp -a) environments you wish to salvage into the newly created miniconda root envs directory
  • Attempt to activate the copied environment
  • If all checks out, erase the backup miniconda root

@jr3cermak
Copy link
Collaborator Author

We have also discovered that this GLIBC issue is currently confined to one particular package: pandas. The fix is to load the conda environment as usual for gridtools. Manually uninstall and reinstall pandas via pip.

$ pip uninstall pandas
Found existing installation: pandas 1.3.5
Uninstalling pandas-1.3.5:
  Would remove:
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas-1.3.5-py3.7.egg-info
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/*
  Would not remove (might be manually added):
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/algos.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/arrays.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/groupby.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/hashing.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/hashtable.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/index.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/indexing.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/internals.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/interval.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/join.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/json.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/lib.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/missing.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/ops.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/ops_dispatch.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/parsers.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/properties.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/reduction.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/reshape.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/sparse.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/testing.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/tslib.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/tslibs/base.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/tslibs/ccalendar.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/tslibs/conversion.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/tslibs/dtypes.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/tslibs/fields.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/tslibs/nattype.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/tslibs/np_datetime.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/tslibs/offsets.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/tslibs/parsing.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/tslibs/period.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/tslibs/strptime.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/tslibs/timedeltas.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/tslibs/timestamps.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/tslibs/timezones.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/tslibs/tzconversion.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/tslibs/vectorized.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/window/aggregations.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/window/indexers.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/writers.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/io/sas/_sas.cpython-37m-x86_64-linux-gnu.so
Proceed (Y/n)? y
  Successfully uninstalled pandas-1.3.5


(gridtest) jrcermakiii@chinook03:~/src/gridtools/docs/conda$ pip install pandas==1.3.5
Collecting pandas==1.3.5
  Downloading pandas-1.3.5.tar.gz (4.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.7/4.7 MB 7.6 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: python-dateutil>=2.7.3 in /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages (from pandas==1.3.5) (2.8.2)
Requirement already satisfied: numpy>=1.17.3 in /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages (from pandas==1.3.5) (1.21.6)
Requirement already satisfied: pytz>=2017.3 in /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages (from pandas==1.3.5) (2022.1)
Requirement already satisfied: six>=1.5 in /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages (from python-dateutil>=2.7.3->pandas==1.3.5) (1.16.0)
Building wheels for collected packages: pandas
  Building wheel for pandas (pyproject.toml) ... done
  Created wheel for pandas: filename=pandas-1.3.5-cp37-cp37m-linux_x86_64.whl size=29515586 sha256=f5ace4f1391d0a3f8d58490453859e86cdd194c47247b879cade9e05bebbb4e2
  Stored in directory: /import/home/jrcermakiii/.cache/pip/wheels/8b/a0/9b/00c6a08ec7b87bf8d7bf4d9ddde7dd7da7c6e822041c580287
Successfully built pandas
Installing collected packages: pandas
Successfully installed pandas-1.3.5

@jr3cermak jr3cermak added the inwork should appear in a release soon label Jun 1, 2022
jr3cermak added a commit that referenced this issue Jun 3, 2022
 - Fix tutorial rendering on github
   - `local_instllation_tutorial.ipynb`
   - `gridtoolAppTutorial.ipynb`
 - Update gridtools manual
   - `source/tutorials/jupyterMOM6.rst`: API updates to `computeBathymetricRoughness()`
 - Update API documentation
   - `bathyutils.py:computeBathymetricRoughness()`: By default, the user should
     supply an extended grid prior to diagnosis of roughness.  The user should
     then clip the result to return the grid to the original dimensions.
 - Update conda documentation
   - Reolve issue #21
 - Add commonly encountered problem to workarounds for jinja2.
 - Add more metadata to written grid files.
   - `grid_angleCalcMethod`
   - `grid_resolution_x`, `grid_resolution_x_Units`
   - `grid_resolution_y`, `grid_resolution_y_Units`
   - `grid_gridResolution`, `grid_gridResolutionUnits`
   - `grid_gridResolutionX`, `grid_gridResolutionXUnits`
   - `grid_gridResolutionY`, `grid_gridResolutionYUnits`
 - Example updates
   - Make note to improve mkGridIterative with updated materials.
   - `mkGridsExample4.ipynb`: improve metadata and comparison of the two
     methods.  Apply `angle_dx` fix for this grid.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
inwork should appear in a release soon
Projects
None yet
Development

No branches or pull requests

1 participant