feat: use python3.11+ for cibuildwheel driver

[mayeut]

As discussed in various issues, I think that cibuildwheel is in position where it might make sense to follow SPEC 0 schedule rather than the CPython schedule. This would allow to take advantage of new Python features faster (or without/ with less back ports).

The idea is that this is a developer tool, and CI and developers tend to be able to get newer versions of Python, so dropping old versions of Python will not affect them too much, and we get big benefits in maintenance. You only need to have Python 3.11 or newer on your CI, and locally if you want to debug wheel production.

For producing Python wheels, we do and still will support the widest range of basically any PyPA project. Currently 3.6+.

This draft PR is:

• to gather feedback on such a change
• to check that indeed, every CI we support is ready.
• preview what would change
• if we want to follow-up with this, then, it might be a good thing to warn users ahead to let them update there workflows if need be (mostly for CI that do not come with python pre-installed).

The idea would be to switch to SPEC 0 in 2024Q4 after Python 3.8 EOL & Python 3.13 GA, thus dropping 3.8, 3.9 & 3.10.

To be clear to anyone coming across this, we're talking about the version of Python that is used to run cibuildwheel. That is distinct from the versions of Python that we build wheels for, which would be unchanged by this.

[joerick]

I support this change. There is very little reason to hang on to older Pythons on the host side, when we can benefit from new features.

To be clear to anyone coming across this, we're talking about the version of Python that is used to run cibuildwheel. That is distinct from the versions of Python that we build wheels for, which would be unchanged by this.

[EwoutH]

I think it's very logical for a project like cibuildwheel to follow SPEC 0 on the host side.

[henryiii]

Let's get 3.13.0 in, make a patch release, then I think this will be ready to merge?

[mayeut]

then I think this will be ready to merge?

I was waiting for 3.13.0 final availability in GHA but maybe we don't care that much about that (or removing the allow-prereleases: true in tests)

[henryiii]

Don't necessarily need to remove the allow-prereleases. The only time Python pre-releases is for upcoming Python versions, which you opt-into anyway. Maybe "3.x" or version ranges could be confused, but otherwise there's no harm in keeping it around. And it's really only our tests, users don't care or need GHA's 3.13.

[henryiii]

(and, 3.13.0 is available AFAICT, it's just not shipped in manifests or runner images yet. But also I'm not talking about merging the instant we release, either ;) )

[Czaki]

looks good to me

[henryiii]

Should we add a warning to cibuildwheel running on Python 3.8-3.10 stating that the host Python needs to be upgraded to keep working with the next version of cibuildwheel? Unlike a normal Python package, you don't want to get silently stuck on an older version of cibuildwheel.

[joerick]

Yeah, that's a good idea. I just checked and a decent number of projects just write pip install cibuildwheel into their build scripts - those people would miss out on cibuildwheel upgrades if their python is oldish.

[joerick]

We're talking in #2047 about holding this until a v3.0 release, but we'll want to get #2052 and perhaps other changes in before then, so I've added a label to that effect.

[jezdez]

Since you're asking for feedback, count me in as worried about adopting a faster-paced deprecation for a PyPA project, which traditionally focuses on all Python ecosystems, not just the Scientific Python ecosystem, which SPEC 0 was approved for.

Would not merging this prevent innovation? Could you share more about the rationale please?

[mayeut]

@jezdez, I added the comment bit from @joerick in the description:

To be clear to anyone coming across this, we're talking about the version of Python that is used to run cibuildwheel. That is distinct from the versions of Python that we build wheels for, which would be unchanged by this.

[henryiii]

I wonder if we should avoid mentioning SPEC 0 too much, I’ve been avoiding mentioning it. I believe the idea is that this is a developer tool, and CI and developers tend to be able to get newer versions of Python, so dropping old versions of Python will not affect them too much, and we get big benefits in maintenance. You only need to have Python 3.11 or newer on your CI, and locally if you want to debug wheel production.

For producing Python wheels, we do and still will support the widest range of basically any PyPA project except manylinux. Currently 3.6+.

Do you have a specific example of a CI service that does not have Python 3.11 or newer?

[jezdez]

I wonder if we should avoid mentioning SPEC 0 too much, I’ve been avoiding mentioning it.

To be honest, that might do the trick since it doesn't add much clarity to downstream developers, given cibuildwheel's foundational position in the Python ecosystem. SPEC 0 is as you know pretty impactful for the SciPy community and a major achievement for the coordinated maintenance of a core set of packages.

I believe the idea is that this is a developer tool, and CI and developers tend to be able to get newer versions of Python, so dropping old versions of Python will not affect them too much, and we get big benefits in maintenance. You only need to have Python 3.11 or newer on your CI, and locally if you want to debug wheel production.

That's the key information that wasn't clear from the PR description above and I worried this would accelerate the deprecation of supported Python versions when building wheel files.

For producing Python wheels, we do and still will support the widest range of basically any PyPA project except manylinux. Currently 3.6+.

Excellent, thanks for confirming.

Do you have a specific example of a CI service that does not have Python 3.11 or newer?

No, this is moot now that I better understand the motivation for the change.