Can't run `poetry install --with lint,typing,test,test_integration` due to numpy problem #19271

walterheck · 2024-03-19T14:14:45Z

walterheck
Mar 19, 2024

Wanted to try and see if I could contribute a simple feature, but I'm not very experienced in Python. I followed the instructions to contribute here, it tells me to run poetry install --with lint,typing,test,test_integration.

That fails for me on a mac like so:

> poetry install --with lint,typing,test,test_integration
Creating virtualenv langchain-community-VdMVZEGV-py3.12 in /Users/walterheck/Library/Caches/pypoetry/virtualenvs
Installing dependencies from lock file

Package operations: 93 installs, 0 updates, 0 removals

  - Installing pycparser (2.21)
  - Installing attrs (23.2.0)
  - Installing certifi (2024.2.2)
  - Installing cffi (1.16.0)
  - Installing charset-normalizer (3.3.2)
  - Installing idna (3.6)
  - Installing six (1.16.0)
  - Installing urllib3 (2.0.7)
  - Installing cryptography (42.0.5)
  - Installing frozenlist (1.4.1)
  - Installing multidict (6.0.5)
  - Installing protobuf (4.25.3)
  - Installing python-dateutil (2.9.0.post0)
  - Installing requests (2.31.0)
  - Installing sniffio (1.3.1)
  - Installing typing-extensions (4.10.0)
  - Installing aiosignal (1.3.1)
  - Installing anyio (3.7.1)
  - Installing click (8.1.7)
  - Installing filelock (3.13.1)
  - Installing fsspec (2024.2.0)
  - Installing h11 (0.14.0)
  - Installing jsonpointer (2.4)
  - Installing numpy (1.24.4): Failed

  ChefBuildError

  Backend 'setuptools.build_meta:__legacy__' is not available.

  Traceback (most recent call last):
    File "/opt/homebrew/Cellar/poetry/1.8.2/libexec/lib/python3.12/site-packages/pyproject_hooks/_in_process/_in_process.py", line 77, in _build_backend
      obj = import_module(mod_path)
            ^^^^^^^^^^^^^^^^^^^^^^^
    File "/opt/homebrew/Cellar/[email protected]/3.12.2_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module
      return _bootstrap._gcd_import(name[level:], package, level)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
    File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
    File "<frozen importlib._bootstrap>", line 1310, in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
    File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
    File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
    File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
    File "<frozen importlib._bootstrap_external>", line 995, in exec_module
    File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
    File "/private/var/folders/ky/pc308ybn2bgcm3v0wwf7dysw0000gn/T/tmp6e397mgd/.venv/lib/python3.12/site-packages/setuptools/__init__.py", line 10, in <module>
      import distutils.core
  ModuleNotFoundError: No module named 'distutils'


  at /opt/homebrew/Cellar/poetry/1.8.2/libexec/lib/python3.12/site-packages/poetry/installation/chef.py:164 in _prepare
      160│
      161│                 error = ChefBuildError("\n\n".join(message_parts))
      162│
      163│             if error is not None:
    → 164│                 raise error from None
      165│
      166│             return path
      167│
      168│     def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:

Note: This error originates from the build backend, and is likely not a problem with poetry but with numpy (1.24.4) not supporting PEP 517 builds. You can verify this by running 'pip wheel --no-cache-dir --use-pep517 "numpy (==1.24.4)"'.

  - Installing orjson (3.9.15)
  - Installing packaging (23.2)
  - Installing pydantic (1.10.14)
  - Installing pytz (2024.1)
  - Installing pyyaml (6.0.1)
  - Installing tqdm (4.66.2)
  - Installing tzdata (2024.1)
  - Installing yarl (1.9.4)

Any idea what I might do to fix this? Googling around gives inconclusive suggestions :)

maximeperrindev · 2024-03-19T14:30:39Z

maximeperrindev
Mar 19, 2024

@walterheck
Can you try to run poetry run pip install numpy

0 replies

walterheck · 2024-03-19T17:40:16Z

walterheck
Mar 19, 2024
Author

that works, but it install 1.26.4, a later version:

> poetry run pip install numpy
Collecting numpy
  Downloading numpy-1.26.4-cp312-cp312-macosx_11_0_arm64.whl.metadata (61 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.1/61.1 kB 1.9 MB/s eta 0:00:00
Downloading numpy-1.26.4-cp312-cp312-macosx_11_0_arm64.whl (13.7 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13.7/13.7 MB 20.7 MB/s eta 0:00:00
Installing collected packages: numpy
Successfully installed numpy-1.26.4

if I then run poetry install --with lint,typing,test,test_integration again, it tries to downgrade because the poetry lock has 1.24.4 in it:

> poetry install --with lint,typing,test,test_integration
Installing dependencies from lock file

Package operations: 61 installs, 1 update, 0 removals

  - Downgrading numpy (1.26.4 -> 1.24.4): Failed

  ChefBuildError

  Backend 'setuptools.build_meta:__legacy__' is not available.

  Traceback (most recent call last):
    File "/opt/homebrew/Cellar/poetry/1.8.2/libexec/lib/python3.12/site-packages/pyproject_hooks/_in_process/_in_process.py", line 77, in _build_backend
      obj = import_module(mod_path)
            ^^^^^^^^^^^^^^^^^^^^^^^
    File "/opt/homebrew/Cellar/[email protected]/3.12.2_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module
      return _bootstrap._gcd_import(name[level:], package, level)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
    File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
    File "<frozen importlib._bootstrap>", line 1310, in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
    File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
    File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
    File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
    File "<frozen importlib._bootstrap_external>", line 995, in exec_module
    File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
    File "/private/var/folders/ky/pc308ybn2bgcm3v0wwf7dysw0000gn/T/tmpw392fxm0/.venv/lib/python3.12/site-packages/setuptools/__init__.py", line 10, in <module>
      import distutils.core
  ModuleNotFoundError: No module named 'distutils'


  at /opt/homebrew/Cellar/poetry/1.8.2/libexec/lib/python3.12/site-packages/poetry/installation/chef.py:164 in _prepare
      160│
      161│                 error = ChefBuildError("\n\n".join(message_parts))
      162│
      163│             if error is not None:
    → 164│                 raise error from None
      165│
      166│             return path
      167│
      168│     def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:

Note: This error originates from the build backend, and is likely not a problem with poetry but with numpy (1.24.4) not supporting PEP 517 builds. You can verify this by running 'pip wheel --no-cache-dir --use-pep517 "numpy (==1.24.4)"'.

Installing 1.24.4 specifically also fails:

> poetry run pip install numpy==1.24.4
Collecting numpy==1.24.4
  Downloading numpy-1.24.4.tar.gz (10.9 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.9/10.9 MB 18.1 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
ERROR: Exception:
Traceback (most recent call last):
  File "/Users/walterheck/Library/Caches/pypoetry/virtualenvs/langchain-community-VdMVZEGV-py3.12/lib/python3.12/site-packages/pip/_internal/cli/base_command.py", line 180, in exc_logging_wrapper
    status = run_func(*args)
             ^^^^^^^^^^^^^^^
  File "/Users/walterheck/Library/Caches/pypoetry/virtualenvs/langchain-community-VdMVZEGV-py3.12/lib/python3.12/site-packages/pip/_internal/cli/req_command.py", line 245, in wrapper
    return func(self, options, args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/walterheck/Library/Caches/pypoetry/virtualenvs/langchain-community-VdMVZEGV-py3.12/lib/python3.12/site-packages/pip/_internal/commands/install.py", line 377, in run
    requirement_set = resolver.resolve(
                      ^^^^^^^^^^^^^^^^^
  File "/Users/walterheck/Library/Caches/pypoetry/virtualenvs/langchain-community-VdMVZEGV-py3.12/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 95, in resolve
    result = self._result = resolver.resolve(
                            ^^^^^^^^^^^^^^^^^
  File "/Users/walterheck/Library/Caches/pypoetry/virtualenvs/langchain-community-VdMVZEGV-py3.12/lib/python3.12/site-packages/pip/_vendor/resolvelib/resolvers.py", line 546, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/walterheck/Library/Caches/pypoetry/virtualenvs/langchain-community-VdMVZEGV-py3.12/lib/python3.12/site-packages/pip/_vendor/resolvelib/resolvers.py", line 397, in resolve
    self._add_to_criteria(self.state.criteria, r, parent=None)
  File "/Users/walterheck/Library/Caches/pypoetry/virtualenvs/langchain-community-VdMVZEGV-py3.12/lib/python3.12/site-packages/pip/_vendor/resolvelib/resolvers.py", line 173, in _add_to_criteria
    if not criterion.candidates:
           ^^^^^^^^^^^^^^^^^^^^
  File "/Users/walterheck/Library/Caches/pypoetry/virtualenvs/langchain-community-VdMVZEGV-py3.12/lib/python3.12/site-packages/pip/_vendor/resolvelib/structs.py", line 156, in __bool__
    return bool(self._sequence)
           ^^^^^^^^^^^^^^^^^^^^
  File "/Users/walterheck/Library/Caches/pypoetry/virtualenvs/langchain-community-VdMVZEGV-py3.12/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 155, in __bool__
    return any(self)
           ^^^^^^^^^
  File "/Users/walterheck/Library/Caches/pypoetry/virtualenvs/langchain-community-VdMVZEGV-py3.12/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 143, in <genexpr>
    return (c for c in iterator if id(c) not in self._incompatible_ids)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/walterheck/Library/Caches/pypoetry/virtualenvs/langchain-community-VdMVZEGV-py3.12/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 47, in _iter_built
    candidate = func()
                ^^^^^^
  File "/Users/walterheck/Library/Caches/pypoetry/virtualenvs/langchain-community-VdMVZEGV-py3.12/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 182, in _make_candidate_from_link
    base: Optional[BaseCandidate] = self._make_base_candidate_from_link(
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/walterheck/Library/Caches/pypoetry/virtualenvs/langchain-community-VdMVZEGV-py3.12/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 228, in _make_base_candidate_from_link
    self._link_candidate_cache[link] = LinkCandidate(
                                       ^^^^^^^^^^^^^^
  File "/Users/walterheck/Library/Caches/pypoetry/virtualenvs/langchain-community-VdMVZEGV-py3.12/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 290, in __init__
    super().__init__(
  File "/Users/walterheck/Library/Caches/pypoetry/virtualenvs/langchain-community-VdMVZEGV-py3.12/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 156, in __init__
    self.dist = self._prepare()
                ^^^^^^^^^^^^^^^
  File "/Users/walterheck/Library/Caches/pypoetry/virtualenvs/langchain-community-VdMVZEGV-py3.12/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 222, in _prepare
    dist = self._prepare_distribution()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/walterheck/Library/Caches/pypoetry/virtualenvs/langchain-community-VdMVZEGV-py3.12/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 301, in _prepare_distribution
    return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/walterheck/Library/Caches/pypoetry/virtualenvs/langchain-community-VdMVZEGV-py3.12/lib/python3.12/site-packages/pip/_internal/operations/prepare.py", line 525, in prepare_linked_requirement
    return self._prepare_linked_requirement(req, parallel_builds)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/walterheck/Library/Caches/pypoetry/virtualenvs/langchain-community-VdMVZEGV-py3.12/lib/python3.12/site-packages/pip/_internal/operations/prepare.py", line 640, in _prepare_linked_requirement
    dist = _get_prepared_distribution(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/walterheck/Library/Caches/pypoetry/virtualenvs/langchain-community-VdMVZEGV-py3.12/lib/python3.12/site-packages/pip/_internal/operations/prepare.py", line 71, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(
  File "/Users/walterheck/Library/Caches/pypoetry/virtualenvs/langchain-community-VdMVZEGV-py3.12/lib/python3.12/site-packages/pip/_internal/distributions/sdist.py", line 54, in prepare_distribution_metadata
    self._install_build_reqs(finder)
  File "/Users/walterheck/Library/Caches/pypoetry/virtualenvs/langchain-community-VdMVZEGV-py3.12/lib/python3.12/site-packages/pip/_internal/distributions/sdist.py", line 124, in _install_build_reqs
    build_reqs = self._get_build_requires_wheel()
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/walterheck/Library/Caches/pypoetry/virtualenvs/langchain-community-VdMVZEGV-py3.12/lib/python3.12/site-packages/pip/_internal/distributions/sdist.py", line 101, in _get_build_requires_wheel
    return backend.get_requires_for_build_wheel()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/walterheck/Library/Caches/pypoetry/virtualenvs/langchain-community-VdMVZEGV-py3.12/lib/python3.12/site-packages/pip/_internal/utils/misc.py", line 745, in get_requires_for_build_wheel
    return super().get_requires_for_build_wheel(config_settings=cs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/walterheck/Library/Caches/pypoetry/virtualenvs/langchain-community-VdMVZEGV-py3.12/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_impl.py", line 166, in get_requires_for_build_wheel
    return self._call_hook('get_requires_for_build_wheel', {
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/walterheck/Library/Caches/pypoetry/virtualenvs/langchain-community-VdMVZEGV-py3.12/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_impl.py", line 321, in _call_hook
    raise BackendUnavailable(data.get('traceback', ''))
pip._vendor.pyproject_hooks._impl.BackendUnavailable: Traceback (most recent call last):
  File "/Users/walterheck/Library/Caches/pypoetry/virtualenvs/langchain-community-VdMVZEGV-py3.12/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 77, in _build_backend
    obj = import_module(mod_path)
          ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.12.2_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1310, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/private/var/folders/ky/pc308ybn2bgcm3v0wwf7dysw0000gn/T/pip-build-env-23iwblc7/overlay/lib/python3.12/site-packages/setuptools/__init__.py", line 10, in <module>
    import distutils.core
ModuleNotFoundError: No module named 'distutils'

I presume it's because distutils on python 3.12 is deprecated: https://docs.python.org/3.11/library/distutils.html

Looking for direction on what I can do best?

0 replies

walterheck · 2024-03-19T17:40:57Z

walterheck
Mar 19, 2024
Author

I guess the langchain dependency for numpy needs to be updated to 1.26.4?

0 replies

maximeperrindev · 2024-03-20T07:31:21Z

maximeperrindev
Mar 20, 2024

@walterheck yeah i'm using python 3.9. That probably explains why it is working for me.
You can downgrade your python aas a workaround for now. I'll check if I need to update the poetry.lock later

0 replies

PeerBoerner1 · 2024-05-17T17:24:14Z

PeerBoerner1
May 17, 2024

I had the same problem on my mac (MacBook Air M3, Sonoma 14.5) and Python 3.12. I used homebrew to install [email protected] and then forced poetry to use this version as well (poetry env use python3.11). All worked after that. Here's the link for how to force Poetry environments: https://python-poetry.org/docs/managing-environments/#switching-between-environments

The core issue seems to be that distutils was removed from python 3.12 (https://www.reddit.com/r/learnpython/comments/17ciqnl/modulenotfounderror_no_module_named_distutils_on/). Going back a version solves this.

I tried getting distutils installed with Python 3.12 but was unsuccessful. That's work for another day.

Use Python 3.11 and force Poetry to do that same. That should solve the issue.

1 reply

khoadaniel Sep 9, 2024

Thanks a ton!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Can't run `poetry install --with lint,typing,test,test_integration` due to numpy problem #19271

{{title}}

Replies: 5 comments 1 reply

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

Can't run poetry install --with lint,typing,test,test_integration due to numpy problem #19271

walterheck Mar 19, 2024

Replies: 5 comments · 1 reply

maximeperrindev Mar 19, 2024

walterheck Mar 19, 2024 Author

walterheck Mar 19, 2024 Author

maximeperrindev Mar 20, 2024

PeerBoerner1 May 17, 2024

khoadaniel Sep 9, 2024

Can't run `poetry install --with lint,typing,test,test_integration` due to numpy problem #19271

walterheck
Mar 19, 2024

Replies: 5 comments 1 reply

maximeperrindev
Mar 19, 2024

walterheck
Mar 19, 2024
Author

walterheck
Mar 19, 2024
Author

maximeperrindev
Mar 20, 2024

PeerBoerner1
May 17, 2024