Use Simple Repository API for --skip-existing

[calebj]

Just playing around with this idea, because it would let twine skip attempted uploads on all repositories that implement PEP 503 (which should be most if not all of them), rather than handling an error response. There is also an adopted JSON spec, PEP 691, but that has less support outside of PyPI.

The pip index versions command grabs versions for the current python version and platform, but the command is marked experimental. It appears to support both JSON and HTML.

I use pypiserver for private packages. pip index versions works on it without any changes. It doesn't pay any attention to the Accept: header and only returns HTML.

What would be the best way to add this to twine? I think twine should default to fetching the simple file list and exclude matching filenames; if the repository doesn't provide a list, it should fall back on the current behavior of ignoring failed duplicate uploads. Working with the JSON list would be simpler, but that should be implemented and tested only after HTML is working.

Can others chime in about what other repositories they use, and whether there would be an issue with using the simple interface?

[woodruffw]

because it would let twine skip attempted uploads on all repositories that implement PEP 503 (which should be most if not all of them), rather than handling an error response.

Please correct me if I'm misunderstanding, but I think this assumes that the index updates the simple index (whether 503 or 691) immediately once a file is added, which is not guaranteed (and in practice isn't true for PyPI, thanks to extensive CDN caching).

In practice I think this will be reliable for existing files that have been present for a while, but not reliable for PyPI or TestPyPI when the file was recently added (e.g. in a CI/CD workflow that partially uploaded the release's files but failed halfway through).

[calebj]

It's not an assumption of mine, I can't speak for other repositories but pypiserver only caches downloads and PyPI has had issues with it before. The fallback will catch a stale index, albeit after the request is made.

Twine already uses the PyPI specific JSON API (example, docs), and a peek at the headers on that endpoint shows cache-control: max-age=900, public. The limit for that is ironically higher than the simple index endpoint, which has cache-control: max-age=600, public for both HTML and JSON.

Regarding PyPI: pypi/warehouse#12214 shows past problems with both the custom-JSON and simple index APIs. Twine might be able to bypass that by using cache-control: no-cache in the request, provided the CDN layer respects it.

[sigmavirus24]

There's also an assumption that any proxies between a non-PyPI index that's private and the index aren't caching proxies with dogger caching semantics