Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GH-84850: Remove urllib.request.URLopener and FancyURLopener #125739

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

GH-84850: Remove urllib.request.URLopener and FancyURLopener #125739

wants to merge 5 commits into from
+44 −987

Conversation

barneygale
Copy link
Contributor

@barneygale barneygale commented Oct 20, 2024 •
edited
Loading

These classes have been deprecated since Python 3.3. They originate in Python 2's urllib, whereas most of the rest of this module comes from urllib2.

📚 Documentation preview 📚: https://cpython-previews--125739.org.readthedocs.build/

@barneygale
pythonGH-84850: Remove urllib.request.URLopener and FancyURLopener
2b3f775 
These classes have been deprecated since Python 3.3.
@bedevere-app bedevere-app bot mentioned this pull request Oct 20, 2024
Open
@vstinner
Copy link
Member

vstinner commented Oct 20, 2024 •
edited
Loading

A code search on PyPI top 8,000 projects (at 2024-10-16) on URLopener finds 21 projects:

  • arelle_release (2.31.6)
  • bandit (1.7.10)
  • cinemagoer (2023.5.1)
  • eventlet (0.37.0)
  • fissix (24.4.24)
  • flake8_tidy_imports (4.10.0)
  • future (1.0.0)
  • jplephem (2.22)
  • m2crypto (0.42.0)
  • newrelic (10.2.0)
  • pscript (0.7.7)
  • py-backwards (0.7)
  • pygments (2.18.0)
  • reportlab (4.2.5)
  • ruff (0.6.9)
  • tf_keras (2.17.0)
  • tf_keras_nightly-2.19.0.dev2024101609
  • wagon (1.0.1)
  • wandb (0.18.3)
  • wxPython (4.2.2)
  • yt (4.3.1)

I didn't check the code to see if imports are conditionals on Python <= 3.3 or not. I ignored six.py copies and .pyi files.

PYPI-2024-10-16/arelle_release-2.31.6.tar.gz: arelle_release-2.31.6/arelle/WebCache.py: class WebCacheUrlOpener(request.FancyURLopener):
PYPI-2024-10-16/bandit-1.7.10.tar.gz: bandit-1.7.10/bandit/blacklists/calls.py: |      |                     | - urllib.URLopener                 |           |
PYPI-2024-10-16/bandit-1.7.10.tar.gz: bandit-1.7.10/bandit/blacklists/calls.py: |      |                     | - urllib.request.URLopener         |           |
PYPI-2024-10-16/bandit-1.7.10.tar.gz: bandit-1.7.10/bandit/blacklists/calls.py: |      |                     | - urllib.FancyURLopener            |           |
PYPI-2024-10-16/bandit-1.7.10.tar.gz: bandit-1.7.10/bandit/blacklists/calls.py: |      |                     | - urllib.request.FancyURLopener    |           |
PYPI-2024-10-16/bandit-1.7.10.tar.gz: bandit-1.7.10/bandit/blacklists/calls.py: |      |                     |   .URLopener                       |           |
PYPI-2024-10-16/bandit-1.7.10.tar.gz: bandit-1.7.10/bandit/blacklists/calls.py: |      |                     |   .FancyURLopener                  |           |
PYPI-2024-10-16/bandit-1.7.10.tar.gz: bandit-1.7.10/bandit/blacklists/calls.py: "urllib.request.URLopener",
PYPI-2024-10-16/bandit-1.7.10.tar.gz: bandit-1.7.10/bandit/blacklists/calls.py: "urllib.request.FancyURLopener",
PYPI-2024-10-16/bandit-1.7.10.tar.gz: bandit-1.7.10/examples/urlopen.py: opener = urllib.request.URLopener()
PYPI-2024-10-16/bandit-1.7.10.tar.gz: bandit-1.7.10/examples/urlopen.py: opener = urllib.request.FancyURLopener()
PYPI-2024-10-16/cinemagoer-2023.5.1.tar.gz: cinemagoer-2023.5.1/imdb/parser/http/__init__.py: class IMDbURLopener:
PYPI-2024-10-16/cinemagoer-2023.5.1.tar.gz: cinemagoer-2023.5.1/imdb/parser/http/__init__.py: self.urlOpener = IMDbURLopener(*arguments, **keywords)
PYPI-2024-10-16/cinemagoer-2023.5.1.tar.gz: cinemagoer-2023.5.1/tests/conftest.py: from imdb.parser.http import IMDbURLopener
PYPI-2024-10-16/cinemagoer-2023.5.1.tar.gz: cinemagoer-2023.5.1/tests/conftest.py: retrieve_unicode_orig = IMDbURLopener.retrieve_unicode
PYPI-2024-10-16/cinemagoer-2023.5.1.tar.gz: cinemagoer-2023.5.1/tests/conftest.py: IMDbURLopener.retrieve_unicode = retrieve_unicode_cached
PYPI-2024-10-16/cinemagoer-2023.5.1.tar.gz: cinemagoer-2023.5.1/tests/conftest.py: IMDbURLopener.retrieve_unicode = retrieve_unicode_orig
PYPI-2024-10-16/eventlet-0.37.0.tar.gz: eventlet-0.37.0/eventlet/green/urllib/request.py: URLopener.open_ftp = patcher.patch_function(URLopener.open_ftp, *to_patch_in_functions)
PYPI-2024-10-16/fissix-24.4.24.tar.gz: fissix-24.4.24/fissix/fixes/fix_urllib.py: "URLopener",
PYPI-2024-10-16/fissix-24.4.24.tar.gz: fissix-24.4.24/fissix/fixes/fix_urllib.py: "FancyURLopener",
PYPI-2024-10-16/flake8_tidy_imports-4.10.0.tar.gz: flake8_tidy_imports-4.10.0/src/flake8_tidy_imports/__init__.py: "urllib.FancyURLopener": (
PYPI-2024-10-16/flake8_tidy_imports-4.10.0.tar.gz: flake8_tidy_imports-4.10.0/src/flake8_tidy_imports/__init__.py: "urllib.URLopener": (
PYPI-2024-10-16/future-1.0.0.tar.gz: future-1.0.0/src/future/moves/urllib/request.py: #                     URLopener,
PYPI-2024-10-16/future-1.0.0.tar.gz: future-1.0.0/src/future/moves/urllib/request.py: #                     FancyURLopener,
PYPI-2024-10-16/future-1.0.0.tar.gz: future-1.0.0/src/libpasteurize/fixes/fix_imports2.py: u'ContentTooShortError', u'FancyURLopener',u'URLopener',
PYPI-2024-10-16/future-1.0.0.tar.gz: future-1.0.0/tests/test_future/test_urllib.py: # Shortcut for testing FancyURLopener
PYPI-2024-10-16/future-1.0.0.tar.gz: future-1.0.0/tests/test_future/test_urllib.py: opener = urllib_request.FancyURLopener(proxies=proxies)
PYPI-2024-10-16/future-1.0.0.tar.gz: future-1.0.0/tests/test_future/test_urllib.py: ('FancyURLopener style of invoking requests is deprecated.',
PYPI-2024-10-16/future-1.0.0.tar.gz: future-1.0.0/tests/test_future/test_urllib.py: opener = urllib_request.FancyURLopener()
PYPI-2024-10-16/future-1.0.0.tar.gz: future-1.0.0/tests/test_future/test_urllib.py: def test_URLopener_deprecation(self):
PYPI-2024-10-16/future-1.0.0.tar.gz: future-1.0.0/tests/test_future/test_urllib.py: urllib_request.URLopener()
PYPI-2024-10-16/future-1.0.0.tar.gz: future-1.0.0/tests/test_future/test_urllib.py: class URLopener_Tests(unittest.TestCase):
PYPI-2024-10-16/future-1.0.0.tar.gz: future-1.0.0/tests/test_future/test_urllib.py: """Testcase to test the open method of URLopener class."""
PYPI-2024-10-16/future-1.0.0.tar.gz: future-1.0.0/tests/test_future/test_urllib.py: class DummyURLopener(urllib_request.URLopener):
PYPI-2024-10-16/future-1.0.0.tar.gz: future-1.0.0/tests/test_future/test_urllib.py: ('DummyURLopener style of invoking requests is deprecated.',
PYPI-2024-10-16/future-1.0.0.tar.gz: future-1.0.0/tests/test_future/test_urllib.py: self.assertEqual(DummyURLopener().open(
PYPI-2024-10-16/future-1.0.0.tar.gz: future-1.0.0/tests/test_future/test_urllib.py: self.assertEqual(DummyURLopener().open(
PYPI-2024-10-16/future-1.0.0.tar.gz: future-1.0.0/tests/test_future/test_urllib_toplevel.py: # Shortcut for testing FancyURLopener
PYPI-2024-10-16/future-1.0.0.tar.gz: future-1.0.0/tests/test_future/test_urllib_toplevel.py: opener = urllib_request.FancyURLopener(proxies=proxies)
PYPI-2024-10-16/future-1.0.0.tar.gz: future-1.0.0/tests/test_future/test_urllib_toplevel.py: ('FancyURLopener style of invoking requests is deprecated.',
PYPI-2024-10-16/future-1.0.0.tar.gz: future-1.0.0/tests/test_future/test_urllib_toplevel.py: opener = urllib_request.FancyURLopener()
PYPI-2024-10-16/future-1.0.0.tar.gz: future-1.0.0/tests/test_future/test_urllib_toplevel.py: def test_URLopener_deprecation(self):
PYPI-2024-10-16/future-1.0.0.tar.gz: future-1.0.0/tests/test_future/test_urllib_toplevel.py: urllib_request.URLopener()
PYPI-2024-10-16/future-1.0.0.tar.gz: future-1.0.0/tests/test_future/test_urllib_toplevel.py: class URLopener_Tests(unittest.TestCase):
PYPI-2024-10-16/future-1.0.0.tar.gz: future-1.0.0/tests/test_future/test_urllib_toplevel.py: """Testcase to test the open method of URLopener class."""
PYPI-2024-10-16/future-1.0.0.tar.gz: future-1.0.0/tests/test_future/test_urllib_toplevel.py: class DummyURLopener(urllib_request.URLopener):
PYPI-2024-10-16/future-1.0.0.tar.gz: future-1.0.0/tests/test_future/test_urllib_toplevel.py: ('DummyURLopener style of invoking requests is deprecated.',
PYPI-2024-10-16/future-1.0.0.tar.gz: future-1.0.0/tests/test_future/test_urllib_toplevel.py: self.assertEqual(DummyURLopener().open(
PYPI-2024-10-16/future-1.0.0.tar.gz: future-1.0.0/tests/test_future/test_urllib_toplevel.py: self.assertEqual(DummyURLopener().open(
PYPI-2024-10-16/future-1.0.0.tar.gz: future-1.0.0/tests/test_future/test_urllibnet.py: open_url = urllib_request.FancyURLopener().open(URL)
PYPI-2024-10-16/jplephem-2.22.tar.gz: jplephem-2.22/jplephem/excerpter.py: from urllib.request import URLopener
PYPI-2024-10-16/jplephem-2.22.tar.gz: jplephem-2.22/jplephem/excerpter.py: from urllib import URLopener
PYPI-2024-10-16/jplephem-2.22.tar.gz: jplephem-2.22/jplephem/excerpter.py: self.opener = URLopener()
PYPI-2024-10-16/m2crypto-0.42.0.tar.gz: m2crypto-0.42.0/src/M2Crypto/m2urllib.py: URLopener.open_https = open_https
PYPI-2024-10-16/newrelic-10.2.0.tar.gz: newrelic-10.2.0/newrelic/hooks/external_urllib.py: if hasattr(module, 'URLopener'):
PYPI-2024-10-16/newrelic-10.2.0.tar.gz: newrelic-10.2.0/newrelic/hooks/external_urllib.py: wrap_function_wrapper(module, 'URLopener.open',
PYPI-2024-10-16/newrelic-10.2.0.tar.gz: newrelic-10.2.0/tests/external_httplib/test_urllib.py: opener = urllib.URLopener()
PYPI-2024-10-16/newrelic-10.2.0.tar.gz: newrelic-10.2.0/tests/external_httplib/test_urllib.py: opener = urllib.URLopener()
PYPI-2024-10-16/newrelic-10.2.0.tar.gz: newrelic-10.2.0/tests/external_httplib/test_urllib.py: opener = urllib.URLopener()
PYPI-2024-10-16/newrelic-10.2.0.tar.gz: newrelic-10.2.0/tests/external_httplib/test_urllib.py: opener = urllib.URLopener()
PYPI-2024-10-16/newrelic-10.2.0.tar.gz: newrelic-10.2.0/tests/external_httplib/test_urllib.py: opener = urllib.URLopener()
PYPI-2024-10-16/newrelic-10.2.0.tar.gz: newrelic-10.2.0/tests/external_httplib/test_urllib.py: opener = urllib.URLopener()
PYPI-2024-10-16/pscript-0.7.7.tar.gz: pscript-0.7.7/translate_to_legacy.py: 'ContentTooShortError', 'FancyURLopener', 'URLopener',
PYPI-2024-10-16/py-backwards-0.7.tar.gz: py-backwards-0.7/py_backwards/transformers/six_moves.py: MovedAttribute("URLopener", "urllib", "urllib.request"),
PYPI-2024-10-16/py-backwards-0.7.tar.gz: py-backwards-0.7/py_backwards/transformers/six_moves.py: MovedAttribute("FancyURLopener", "urllib", "urllib.request"),
PYPI-2024-10-16/pygments-2.18.0.tar.gz: pygments-2.18.0/pygments/lexers/_sourcemod_builtins.py: from urllib.request import FancyURLopener
PYPI-2024-10-16/pygments-2.18.0.tar.gz: pygments-2.18.0/pygments/lexers/_sourcemod_builtins.py: class Opener(FancyURLopener):
PYPI-2024-10-16/reportlab-4.2.5.tar.gz: reportlab-4.2.5/src/reportlab/lib/utils.py: #copied here from urllib.URLopener.open_data because
PYPI-2024-10-16/ruff-0.6.9.tar.gz: ruff-0.6.9/crates/ruff_linter/src/rules/flake8_bandit/rules/suspicious_function_call.rs: // URLOpen (`URLopener`, `FancyURLopener`)
PYPI-2024-10-16/ruff-0.6.9.tar.gz: ruff-0.6.9/crates/ruff_linter/src/rules/flake8_bandit/rules/suspicious_function_call.rs: ["urllib", "request", "URLopener" | "FancyURLopener"] |
PYPI-2024-10-16/tf_keras-2.17.0.tar.gz: tf_keras-2.17.0/tf_keras/src/utils/data_utils.py: Under Python 2, `urlretrieve` relies on `FancyURLopener` from legacy
PYPI-2024-10-16/tf_keras_nightly-2.19.0.dev2024101609.tar.gz: tf_keras_nightly-2.19.0.dev2024101609/tf_keras/src/utils/data_utils.py: Under Python 2, `urlretrieve` relies on `FancyURLopener` from legacy
PYPI-2024-10-16/wagon-1.0.1.tar.gz: wagon-1.0.1/wagon.py: from urllib.request import URLopener
PYPI-2024-10-16/wagon-1.0.1.tar.gz: wagon-1.0.1/wagon.py: from urllib import URLopener
PYPI-2024-10-16/wagon-1.0.1.tar.gz: wagon-1.0.1/wagon.py: f = URLopener()
PYPI-2024-10-16/wandb-0.18.3.tar.gz: wandb-0.18.3/wandb/vendor/pygments/lexers/_sourcemod_builtins.py: from urllib import FancyURLopener
PYPI-2024-10-16/wandb-0.18.3.tar.gz: wandb-0.18.3/wandb/vendor/pygments/lexers/_sourcemod_builtins.py: from urllib.request import FancyURLopener
PYPI-2024-10-16/wandb-0.18.3.tar.gz: wandb-0.18.3/wandb/vendor/pygments/lexers/_sourcemod_builtins.py: class Opener(FancyURLopener):
PYPI-2024-10-16/wxPython-4.2.2.tar.gz: wxPython-4.2.2/wx/lib/pubsub/py2and3.py: MovedAttribute("URLopener", "urllib", "urllib.request"),
PYPI-2024-10-16/wxPython-4.2.2.tar.gz: wxPython-4.2.2/wx/lib/pubsub/py2and3.py: MovedAttribute("FancyURLopener", "urllib", "urllib.request"),
PYPI-2024-10-16/yt-4.3.1.tar.gz: yt-4.3.1/yt/funcs.py: class MyURLopener(urllib.request.FancyURLopener):
PYPI-2024-10-16/yt-4.3.1.tar.gz: yt-4.3.1/yt/funcs.py: fn, h = MyURLopener().retrieve(url, filename)

cc @hugovk

@barneygale
Copy link
Contributor Author

Thanks Victor. From that list, I think about 6 packages are using URLopener in anger, while the remainder are naming coincidences, linter rules, or vendoring of one of affected packages.

@barneygale
Copy link
Contributor Author

barneygale commented Oct 22, 2024 •
edited
Loading

I believe only four packages would break for users if we removed these classes: eventlet, jplephem, wagon, and yt. It looks like eventlet is best fixed with a Python version conditional, so we can't fix that til this patch lands. I've opened issues for the other packages:

I'll work on PRs if the package authors seem interested.

@barneygale
Copy link
Contributor Author

I've submitted PRs against the three projects, one of which has been accepted already. They're all simple changes - in Python 2 terms, we're upgrading from urllib to urllib2.

Given that 3.14 is still around a year away, and only 3 projects in 8,000 are affected and possible to remedy at this stage, I submit that we don't need to wait for the other two PRs. Thoughts?

@hugovk
Copy link
Member

hugovk commented Oct 25, 2024

Given that 3.14 is still around a year away, and only 3 projects in 8,000 are affected and possible to remedy at this stage, I submit that we don't need to wait for the other two PRs. Thoughts?

Agreed, this is definitely on the low side.

barneygale
barneygale commented Oct 25, 2024
View reviewed changes
Lib/test/test_urllib.py
finally:
self.unfakeftp()

def test_userpass_inurl(self):
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test and the one below are removed due to GH-78809

@barneygale barneygale mentioned this pull request Oct 30, 2024
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@vstinner vstinner Awaiting requested review from vstinner

Assignees
No one assigned
Labels
awaiting core review
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@barneygale @vstinner @hugovk