--retries has no effect during streaming downloads #12383
Labels
Comments
asottile-sentry
added
S: needs triage
|
this is absolutely not the correct patch but it does satisfy my requirements for now diff --git a/src/pip/_internal/network/download.py b/src/pip/_internal/network/download.py
index d1d43541e..4919f3e0a 100644
--- a/src/pip/_internal/network/download.py
+++ b/src/pip/_internal/network/download.py
@@ -7,6 +7,7 @@ import os
from typing import Iterable, Optional, Tuple
from pip._vendor.requests.models import CONTENT_CHUNK_SIZE, Response
+from pip._vendor.tenacity import retry, stop_after_attempt
from pip._internal.cli.progress_bars import get_download_progress_renderer
from pip._internal.exceptions import NetworkConnectionError
@@ -128,6 +129,7 @@ class Downloader:
self._session = session
self._progress_bar = progress_bar
+ @retry(reraise=True, stop=stop_after_attempt(5))
def __call__(self, link: Link, location: str) -> Tuple[str, str]:
"""Download the file given by link into location."""
try: |
asottile-sentry
added a commit
to getsentry/sentry
that referenced
this issue
Nov 20, 2023
pypa/pip#12383 (comment) this is a terrible hack -- essentially: - write a small `.pth` file to monkeypatch pip for any calls into the virtualenv site-packages - add a retrier to `Downloader.__call__` - instrument all of our virtualenv creation with it
k-fish
pushed a commit
to getsentry/sentry
that referenced
this issue
Nov 21, 2023
pypa/pip#12383 (comment) this is a terrible hack -- essentially: - write a small `.pth` file to monkeypatch pip for any calls into the virtualenv site-packages - add a retrier to `Downloader.__call__` - instrument all of our virtualenv creation with it
armenzg
pushed a commit
to getsentry/sentry
that referenced
this issue
Nov 27, 2023
pypa/pip#12383 (comment) this is a terrible hack -- essentially: - write a small `.pth` file to monkeypatch pip for any calls into the virtualenv site-packages - add a retrier to `Downloader.__call__` - instrument all of our virtualenv creation with it
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
a few things up front since I've found a few related issues, including one that's almost certainly an exact duplicate however the advice there doesn't seem like an exact match to my case:
in my case I am seeing this from github actions both against public pypi and against an "internal" but public-facing wheeling mirror at https://pypi.devinfra.sentry.io/simple . I'm using the default timeout (15 seconds) which for most connections is fast enough (typical network speeds between GHA and public pypi or our pypi server are about 50MBps -- easily downloading even the bulkiest packages in a second or two). unfortunately GitHub's network is significantly flaky -- and simple retries would help immensely for these downloads (even if they started over from the beginning as #4796)
there was rationale given in the duplicate above here that retrying
ReadTimeOutErrors would lead to excessively long waiting -- however pip seems to already retry them in cases which aren't streamed responses -- here's an example where I've artificially reduced the public pypi timeout low enough and you can see the retries onReadTimeOutError:worst case I can retry the entire pip installation -- but this seems like quite a heavy hammer for what should essentially be a bunch of file downloads
related: it seems
--timeoutalso doesn't affect streamed downloads:Expected behavior
I expect a retry to occur when streaming a response (wheel / sdist / archive download) when the connection is stalled due to intermittent network failures leading to ReadTimeOut
pip version
22.1.2 -- also reproduced on latest (23.3.1)
Python version
3.8.16
OS
ubuntu 22.04 (ubuntu-latest in GHA)
How to Reproduce
I am using this requirements file -- however I can reproduce it against public pypi with enough attempts (github actions network is quite flaky unfortunately!)
effectively I'm running
pip install -r requirements-dev-frozen.txtOutput
going to hide this one because it's a bit long
the relevant error is:
$ pip install -r requirements-dev-frozen.txt
Code of Conduct
The text was updated successfully, but these errors were encountered: