trouble installing notebook locally with Python 3.12

[bendichter]

When I run this cell locally:

import warnings
warnings.filterwarnings('ignore')

try:
    from databook_utils.dandi_utils import dandi_stream_open
except:
    !git clone https://github.com/AllenInstitute/openscope_databook.git
    %cd openscope_databook
    %pip install -e .

I get

fatal: destination path 'openscope_databook' already exists and is not an empty directory.
/Users/bendichter/dev/openscope_databook/docs/visualization/openscope_databook
Obtaining file:///Users/bendichter/dev/openscope_databook/docs/visualization/openscope_databook
  Preparing metadata (setup.py) ... done
Collecting ophys-nway-matching@ git+https://github.com/AllenInstitute/ophys_nway_matching@545504ab55922717ab623f8ede2c521a60aa1458 (from databook_utils==1.1.0)
  Cloning https://github.com/AllenInstitute/ophys_nway_matching (to revision 545504ab55922717ab623f8ede2c521a60aa1458) to /private/var/folders/cg/mhhypnc14wn9kz5ydllymyc00000gp/T/pip-install-7c2_46b9/ophys-nway-matching_1ff1791cdcc842a0b673739f7813baba
  Running command git clone --filter=blob:none --quiet https://github.com/AllenInstitute/ophys_nway_matching /private/var/folders/cg/mhhypnc14wn9kz5ydllymyc00000gp/T/pip-install-7c2_46b9/ophys-nway-matching_1ff1791cdcc842a0b673739f7813baba
  Running command git rev-parse -q --verify 'sha^545504ab55922717ab623f8ede2c521a60aa1458'
  Running command git fetch -q https://github.com/AllenInstitute/ophys_nway_matching 545504ab55922717ab623f8ede2c521a60aa1458
  Running command git checkout -q 545504ab55922717ab623f8ede2c521a60aa1458
  Resolved https://github.com/AllenInstitute/ophys_nway_matching to commit 545504ab55922717ab623f8ede2c521a60aa1458
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting ssm@ git+https://github.com/lindermanlab/ssm (from databook_utils==1.1.0)
  Cloning https://github.com/lindermanlab/ssm to /private/var/folders/cg/mhhypnc14wn9kz5ydllymyc00000gp/T/pip-install-7c2_46b9/ssm_3f89b73d756c4cb5b39da4e289fd1291
  Running command git clone --filter=blob:none --quiet https://github.com/lindermanlab/ssm /private/var/folders/cg/mhhypnc14wn9kz5ydllymyc00000gp/T/pip-install-7c2_46b9/ssm_3f89b73d756c4cb5b39da4e289fd1291
  Resolved https://github.com/lindermanlab/ssm to commit 6c856ad3967941d176eb348bcd490cfaaa08ba60
  Preparing metadata (setup.py) ... done
Collecting ndx-harvey-swac@ git+https://github.com/sytseng/ndx-harvey-swac (from databook_utils==1.1.0)
  Cloning https://github.com/sytseng/ndx-harvey-swac to /private/var/folders/cg/mhhypnc14wn9kz5ydllymyc00000gp/T/pip-install-7c2_46b9/ndx-harvey-swac_c38c99efeae047b897cce4a7de10c222
  Running command git clone --filter=blob:none --quiet https://github.com/sytseng/ndx-harvey-swac /private/var/folders/cg/mhhypnc14wn9kz5ydllymyc00000gp/T/pip-install-7c2_46b9/ndx-harvey-swac_c38c99efeae047b897cce4a7de10c222
  Resolved https://github.com/sytseng/ndx-harvey-swac to commit c82b84cbf7c9d3e8c320c439dab3bf2fd48f0ae8
  Preparing metadata (setup.py) ... done
Collecting ccfwidget==0.5.3 (from databook_utils==1.1.0)
  Using cached ccfwidget-0.5.3-py3-none-any.whl.metadata (614 bytes)
Collecting cebra (from databook_utils==1.1.0)
  Using cached cebra-0.4.0-py2.py3-none-any.whl.metadata (5.8 kB)
Requirement already satisfied: cython in /opt/miniconda3/lib/python3.12/site-packages (from databook_utils==1.1.0) (3.0.10)
Collecting dandi==0.61.2 (from databook_utils==1.1.0)
  Using cached dandi-0.61.2-py3-none-any.whl.metadata (7.6 kB)
Collecting elephant==0.12.0 (from databook_utils==1.1.0)
  Using cached elephant-0.12.0.tar.gz (2.7 MB)
  Preparing metadata (setup.py) ... done
Collecting h5py>=3.8 (from databook_utils==1.1.0)
  Using cached h5py-3.11.0-cp312-cp312-macosx_11_0_arm64.whl.metadata (2.5 kB)
Collecting ipympl==0.9.4 (from databook_utils==1.1.0)
  Using cached ipympl-0.9.4-py3-none-any.whl.metadata (8.7 kB)
Collecting ipywidgets==7.8.1 (from databook_utils==1.1.0)
  Using cached ipywidgets-7.8.1-py2.py3-none-any.whl.metadata (1.9 kB)
Collecting itkwidgets==0.32.6 (from databook_utils==1.1.0)
  Using cached itkwidgets-0.32.6-py2.py3-none-any.whl.metadata (20 kB)
Requirement already satisfied: jupyter==1.0.0 in /opt/miniconda3/lib/python3.12/site-packages (from databook_utils==1.1.0) (1.0.0)
Collecting matplotlib==3.6.2 (from databook_utils==1.1.0)
  Using cached matplotlib-3.6.2.tar.gz (35.8 MB)
  Preparing metadata (setup.py) ... done
Collecting mpl-interactions==0.22.0 (from databook_utils==1.1.0)
  Using cached mpl_interactions-0.22.0-py3-none-any.whl.metadata (10 kB)
Collecting neo==0.13.0 (from databook_utils==1.1.0)
  Using cached neo-0.13.0-py3-none-any.whl.metadata (8.8 kB)
Collecting numpy==1.23.5 (from databook_utils==1.1.0)
  Using cached numpy-1.23.5.tar.gz (10.7 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [33 lines of output]
      Traceback (most recent call last):
        File "/opt/miniconda3/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/opt/miniconda3/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/miniconda3/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 112, in get_requires_for_build_wheel
          backend = _build_backend()
                    ^^^^^^^^^^^^^^^^
        File "/opt/miniconda3/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 77, in _build_backend
          obj = import_module(mod_path)
                ^^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/miniconda3/lib/python3.12/importlib/__init__.py", line 90, in import_module
          return _bootstrap._gcd_import(name[level:], package, level)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
        File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
        File "<frozen importlib._bootstrap>", line 1310, in _find_and_load_unlocked
        File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
        File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
        File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
        File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
        File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
        File "<frozen importlib._bootstrap_external>", line 995, in exec_module
        File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
        File "/private/var/folders/cg/mhhypnc14wn9kz5ydllymyc00000gp/T/pip-build-env-jah8dexv/overlay/lib/python3.12/site-packages/setuptools/__init__.py", line 16, in <module>
          import setuptools.version
        File "/private/var/folders/cg/mhhypnc14wn9kz5ydllymyc00000gp/T/pip-build-env-jah8dexv/overlay/lib/python3.12/site-packages/setuptools/version.py", line 1, in <module>
          import pkg_resources
        File "/private/var/folders/cg/mhhypnc14wn9kz5ydllymyc00000gp/T/pip-build-env-jah8dexv/overlay/lib/python3.12/site-packages/pkg_resources/__init__.py", line 2172, in <module>
          register_finder(pkgutil.ImpImporter, find_on_path)
                          ^^^^^^^^^^^^^^^^^^^
      AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
Note: you may need to restart the kernel to use updated packages.

It looks like the installation is not working because I am trying to use Python 3.12 instead of 3.8. Python 3.8 will reach end of life in October 2024, so I would recommend updating this. Also, I don't see much documentation for the fact that this notebook requires Python 3.8, aside from the runtime.txt file. I am not familiar with that convention. Maybe it is Jupyter Book-specific, but anyway, I feel this should be updated or at least documented better.

Also, this cell will download the entire 1-GB package again, even though I already cloned it.

[bendichter]

I would also recommend that you build a test matrix across operating systems and Python versions in your CI to get ahead of these types of issues

[rcpeene]

@bendichter This is a great idea. Would you mind making a separate issue and refer to some implementation examples/ideas?

[rcpeene]

Spoke with @bendichter in person. We can specify multiple python versions in our build test, but it would multiply the amount of time required to run the build test. Our testing instance is not free so it may also be more expensive. @jeromelecoq what do you think? It may not provide much more benefit than updating the Databook's versioning when necessary.