Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kedro[test]==0.18.13 fails even though requiring its depndencies do not #12768

Open
1 task done
notatallshaw opened this issue Jun 15, 2024 · 6 comments · May be fixed by #13001
Open
1 task done

kedro[test]==0.18.13 fails even though requiring its depndencies do not #12768

notatallshaw opened this issue Jun 15, 2024 · 6 comments · May be fixed by #13001
Labels
C: dependency resolution About choosing which dependencies to install type: bug A confirmed bug or unintended behavior
Milestone

Comments

@notatallshaw
Copy link
Member

Description

On Linux Python 3.9 installing kedro[test]==0.18.13 fails with resolution impossible

However if you directly specify it's requirements as extracted from the metadata (https://files.pythonhosted.org/packages/50/01/76d44fd50471cd1bd9899e161c07444026c954c01a9012b2c3a8f8a9e1c5/kedro-0.18.13-py3-none-any.whl.metadata) e.g.:

requirements.txt
anyconfig <0.14,>=0.10
attrs >=21.3
build
cachetools ~=5.3
click <9.0
cookiecutter <3.0,>=2.1.1
dynaconf <4.0,>=3.1.2
fsspec <2024.1,>=2021.4
gitpython ~=3.0
importlib-resources >=1.3
jmespath <2.0,>=0.9.5
more-itertools <11,>=9
omegaconf ~=2.3
parse ~=1.19.0
pip-tools <8,>=6.5
pluggy <1.3,>=1.0
PyYAML <7.0,>=4.2
rich <14.0,>=12.0
rope <2.0,>=0.21
setuptools >=65.5.1
toml ~=0.10
toposort ~=1.5
importlib-metadata <5.0,>=3.6 ; python_version < "3.8"
importlib-metadata >=3.6 ; python_version >= "3.8"
bandit <2.0,>=1.6.2 
behave ==1.2.6 
biopython ~=1.73 
blacken-docs ==1.9.2 
black ~=22.0 
compress-pickle[lz4] ~=2.1.0 
coverage[toml] 
dask[complete] ~=2021.10 
dill ~=0.3.1 
filelock <4.0,>=3.4.0 
geopandas <1.0,>=0.6.0 
hdfs <3.0,>=2.5.8 
holoviews >=1.13.0 
import-linter[toml] ==1.8.0 
isort ~=5.0 
Jinja2 <3.1.0 
joblib >=0.14 
jupyterlab-server <2.16.0,>=2.11.1 
jupyterlab <3.6.0,~=3.0 
jupyter ~=1.0 
lxml ~=4.6 
memory-profiler <1.0,>=0.50.0 
networkx ~=2.4 
opencv-python ~=4.5.5.64 
openpyxl <4.0,>=3.0.3 
pandas ~=1.3 
Pillow ~=9.0 
plotly <6.0,>=4.8.0 
pre-commit <3.0,>=2.9.2 
pylint <3.0,>=2.17.0 
pyproj ~=3.0 
pytest-cov ~=3.0 
pytest-mock <2.0,>=1.7.1 
pytest-xdist[psutil] ~=2.2.1 
pytest ~=7.2 
redis ~=4.1 
requests-mock ~=1.6 
requests ~=2.20 
s3fs <0.5,>=0.3.0 
scikit-learn <2,>=1.0.2 
scipy >=1.7.3 
semver 
SQLAlchemy ~=1.2 
triad <1.0,>=0.6.7 
trufflehog ~=2.1 
xlsxwriter ~=1.0 
tensorflow ~=2.0 ; (platform_system != "Darwin" or platform_machine != "arm64")
tables ~=3.6 ; (platform_system != "Windows")
tensorflow-macos ~=2.0 ; (platform_system == "Darwin" and platform_machine == "arm64")
tables ~=3.6.0 ; (platform_system == "Windows" and python_version < "3.8")
tables ~=3.8.0 ; (platform_system == "Windows" and python_version >= "3.8")
matplotlib <3.4,>=3.0.3 ; (python_version < "3.10")
moto ==1.3.7 ; (python_version < "3.10")
delta-spark ~=1.2.1 ; (python_version < "3.11")
pandas-gbq <0.18.0,>=0.12.0 ; (python_version < "3.11")
pyarrow >=1.0 ; (python_version < "3.11")
pyspark <3.4,>=2.2 ; (python_version < "3.11")
ipython <8.0,>=7.31.1 ; (python_version < "3.8")
adlfs <=2022.2,>=2021.7.1 ; (python_version == "3.7")
gcsfs <=2023.1,>=2021.4 ; (python_version == "3.7")
matplotlib <3.6,>=3.5 ; (python_version >= "3.10")
moto ==4.1.12 ; (python_version >= "3.10")
delta-spark >=1.2.1 ; (python_version >= "3.11")
pandas-gbq >=0.18.0 ; (python_version >= "3.11")
pyarrow >=7.0 ; (python_version >= "3.11")
pyspark >=3.4 ; (python_version >= "3.11")
adlfs ~=2023.1 ; (python_version >= "3.8")
gcsfs <2023.3,>=2023.1 ; (python_version >= "3.8")
ipython ~=8.10 ; (python_version >= "3.8")

Then it installs fine. Also uv and older vesions of pip can install kedro[test]==0.18.13.

Expected behavior

kedro[test]==0.18.13 installs fine

pip version

24.0

Python version

3.9

OS

Linux

How to Reproduce

  1. python -m pip install --ignore-installed --dry-run "kedro[test]==0.18.13"

Output

ERROR: Cannot install dask[complete]==2021.12.0 and kedro[test]==0.18.13 because these package versions have conflicting dependencies.

The conflict is caused by:
    kedro[test] 0.18.13 depends on dask~=2021.10; extra == "test"
    dask[complete] 2021.12.0 depends on dask 2021.12.0 (from https://files.pythonhosted.org/packages/15/6d/99c63be3ea8a4a651d845addeea1f1b3bb8e5c6730bc26cfb6176631adf7/dask-2021.12.0-py3-none-any.whl (from https://pypi.org/simple/dask/) (requires-python:>=3.7))

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip to attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

Code of Conduct

@notatallshaw notatallshaw added S: needs triage Issues/PRs that need to be triaged type: bug A confirmed bug or unintended behavior labels Jun 15, 2024
@notatallshaw
Copy link
Member Author

notatallshaw commented Jun 15, 2024

Probably realted to #12317, but other requirement examples are fixed by sarugaku/resolvelib#152 so I wanted to track this one, and it's peculiarities, seperately.

@notatallshaw
Copy link
Member Author

notatallshaw commented Jun 15, 2024

Here is a relatively minimal reproducer example that's easier to prove is an issue:

Create a pyproject.toml like so:

[build-system]
requires = ["setuptools"] 
build-backend = "setuptools.build_meta"

[project]
name = "reproducer"
authors = [
    {name = "reproducer"}
]
version = "0.0.1"
description = "reproducer"
dependencies = [
    "adlfs~=2023.1",
    "dask[complete]~=2021.10",
    "gcsfs>=2023.1, <2023.3",
]

Run python -m pip install --ignore-installed --dry-run . and get the error:

ERROR: Cannot install dask[complete]==2021.12.0 and reproducer==0.0.1 because these package versions have conflicting dependencies.

The conflict is caused by:
    reproducer 0.0.1 depends on dask~=2021.10
    dask[complete] 2021.12.0 depends on dask 2021.12.0 (from https://files.pythonhosted.org/packages/15/6d/99c63be3ea8a4a651d845addeea1f1b3bb8e5c6730bc26cfb6176631adf7/dask-2021.12.0-py3-none-any.whl (from https://pypi.org/simple/dask/) (requires-python:>=3.7))

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip to attempt to solve the dependency conflict

Now run python -m pip install --ignore-installed --dry-run "adlfs~=2023.1" "dask[complete]~=2021.10" "gcsfs>=2023.1, <2023.3" and it completes:

Would install Jinja2-3.1.4 MarkupSafe-2.1.5 PyJWT-2.8.0 PyYAML-6.0.1 adlfs-2023.8.0 aiohttp-3.9.5 aiosignal-1.3.1 async-timeout-4.0.3 attrs-23.2.0 azure-core-1.30.2 azure-datalake-store-0.0.53 azure-identity-1.16.1 azure-storage-blob-12.20.0 bokeh-3.4.1 cachetools-5.3.3 certifi-2024.6.2 cffi-1.16.0 charset-normalizer-3.3.2 click-8.1.7 cloudpickle-3.0.0 contourpy-1.2.1 cryptography-42.0.8 dask-2021.12.0 decorator-5.1.1 distributed-2021.12.0 frozenlist-1.4.1 fsspec-2023.1.0 gcsfs-2023.1.0 google-api-core-2.19.0 google-auth-2.30.0 google-auth-oauthlib-1.2.0 google-cloud-core-2.4.1 google-cloud-storage-2.17.0 google-crc32c-1.5.0 google-resumable-media-2.7.1 googleapis-common-protos-1.63.1 idna-3.7 isodate-0.6.1 locket-1.0.0 msal-1.28.1 msal-extensions-1.1.0 msgpack-1.0.8 multidict-6.0.5 numpy-1.26.4 oauthlib-3.2.2 packaging-24.1 pandas-2.2.2 partd-1.4.2 pillow-10.3.0 portalocker-2.8.2 proto-plus-1.23.0 protobuf-4.25.3 psutil-5.9.8 pyasn1-0.6.0 pyasn1_modules-0.4.0 pycparser-2.22 python-dateutil-2.9.0.post0 pytz-2024.1 requests-2.32.3 requests-oauthlib-2.0.0 rsa-4.9 setuptools-70.0.0 six-1.16.0 sortedcontainers-2.4.0 tblib-3.0.0 toolz-0.12.1 tornado-6.4.1 typing_extensions-4.12.2 tzdata-2024.1 urllib3-2.2.1 xyzservices-2024.6.0 yarl-1.9.4 zict-3.0.0

@uranusjr uranusjr changed the title kedro[test]==0.18.13 fails even though requiring it's depndencies does not kedro[test]==0.18.13 fails even though requiring its depndencies do not Jun 17, 2024
@pradyunsg pradyunsg added C: dependency resolution About choosing which dependencies to install and removed S: needs triage Issues/PRs that need to be triaged labels Jun 20, 2024
@pradyunsg
Copy link
Member

@notatallshaw IIUC, this has been fixed in resolvelib and we need to pull in that fix here?

@notatallshaw
Copy link
Member Author

notatallshaw commented Jun 25, 2024

No, this is an example of a requirement not fixed by the fallback implemented in resolvelib.

Either, backjumping needs to be removed (or made optional) to fix this, or we go with sarugaku/resolvelib#134 (comment) and document the provider needs to behave in a certain way, and then we will need to update the pip provider to comply.

I am working on a PR on the resolvelib side to document this behavior, and give an example with a test provider, and then we'll see where we go from there.

@notatallshaw
Copy link
Member Author

This will be fixed once resolvelib releases and it is vendored: sarugaku/resolvelib#155

@notatallshaw
Copy link
Member Author

Updating to pip 25.0 milestone as I reason here: sarugaku/resolvelib#159 (comment)

@notatallshaw notatallshaw modified the milestones: 24.3, 25.0 Oct 10, 2024
@notatallshaw notatallshaw linked a pull request Oct 18, 2024 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C: dependency resolution About choosing which dependencies to install type: bug A confirmed bug or unintended behavior
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants