Could more be cached in the backtracking process?

[jerbob92]

Let me start with: I'm not sure if this is actually in pip-compile or in pip itself, so please let me know if this should be requested somewhere else.

What's the problem this feature will solve?

We use Dependabot to find package updates in our projects. Dependabot would basically run:

pip-compile --allow-unsafe --output-file=requirements.txt -P {dependency-name} requirements.in

For every dependency in the requirements.txt file.

Since the change from the legacy resolver to the backtracking resolver, this process takes ages (multiple hours for most, 10+ hours for some).

Describe the solution you'd like

What I noticed is that even though a lot is cached already (package version lookups, wheels), it still does quite some work for every package in the backtracking process like copying the wheel, extracting the wheel, looking at the metadata.

The legacy resolver seemed to cache everything much "harder", and finishes almost instantly when things are cached.

Alternative Solutions

Couldn't find an alternative.

Additional context

N/A

[webknjaz]

Perhaps we need to link Pip's docs from ours.. Does https://pip.pypa.io/en/stable/topics/dependency-resolution/#possible-ways-to-reduce-backtracking help?

I'd try to compare pip install vs. pip-compile to bisect the problem..

[jerbob92]

Thanks, but that doesn't say anything about improving the performance of backtracking itself.

I don't mind that it's backtracking, it's just that backtracking itself is very slow, but the reason why it is (or needs to be) slow is unclear to me.

Will do some tests with pip install tomorrow!

[jerbob92]

I did some tests with pip itself and it's not better indeed. I have searched a bit and there seems to be multiple initiatives to make the backtracking resolver faster: pypa/pip#12921

Closing this one :)