Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MAINT automate the pypi release process with CI and trusted publishing #548

Merged
merged 1 commit into from
Jan 14, 2025

Conversation

ogrisel
Copy link
Contributor

@ogrisel ogrisel commented Jan 14, 2025

I would like to ease the release process by automating it while improving security by leveraging the transparency (provenance tracking) and integrity checks (via signatures) of the trusted publishers' setup.

I am following the official guide here:

https://packaging.python.org/en/latest/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows/

The expected results of that workflow are:

  • regular publishing to test.pypi.org whenever we push to master to be able to test that the workflow works well;
  • automated github and pypi.org releases whenever we push a tag.

@ogrisel
Copy link
Contributor Author

ogrisel commented Jan 14, 2025

I will configure the trusted publisher on both pypi.org and testpypi.org.

@ogrisel
Copy link
Contributor Author

ogrisel commented Jan 14, 2025

I did the required config. Let's merge to check if this works as expected on test.pypi.org.

@ogrisel ogrisel merged commit 73598f6 into cloudpipe:master Jan 14, 2025
25 checks passed
@ogrisel ogrisel deleted the pypi-release-automation branch January 14, 2025 14:54
@ogrisel
Copy link
Contributor Author

ogrisel commented Jan 14, 2025

My "pending" trusted publisher config on testpypi.org does not seem to work as expected:

https://github.com/cloudpipe/cloudpickle/actions/runs/12770251295/job/35594676819

I will try to create the cloudpickle project manually on test.pypi.org and reconfigure the trusted publisher on that project once it exists instead of using the pending config.

@ogrisel
Copy link
Contributor Author

ogrisel commented Jan 14, 2025

I did that and configured the trusted publisher here: https://test.pypi.org/manage/project/cloudpickle/settings/publishing/

Let's try again by pushing an empty commit to master.

@ogrisel
Copy link
Contributor Author

ogrisel commented Jan 14, 2025

I got another, but different error this time:

https://github.com/cloudpipe/cloudpickle/actions/runs/12770664554/job/35596079248

Transparency log entry created with index: 162316554
Uploading distributions to https://test.pypi.org/legacy/
Uploading cloudpickle-3.2.0.dev0-py3-none-any.whl
WARNING  Error during upload. Retry with the --verbose option for more details. 
ERROR    HTTPError: 400 Bad Request from https://test.pypi.org/legacy/          
         Bad Request    

@ogrisel
Copy link
Contributor Author

ogrisel commented Jan 14, 2025

Another fail with a similar error message:

https://github.com/cloudpipe/cloudpickle/actions/runs/12770941887/job/35596981931

@ogrisel
Copy link
Contributor Author

ogrisel commented Jan 14, 2025

With verbose output we get:

Uploading cloudpickle-3.2.0.dev0-py3-none-any.whl
INFO     Response from https://test.pypi.org/legacy/:                           
         400 Bad Request                                                        
INFO     <html>                                                                 
          <head>                                                                
           <title>400 File already exists                                       
         ('cloudpickle-3.2.0.dev0-py3-none-any.whl', with blake2_256 hash       
         '74e6a08c764494daf6ed2ae115ca79af2d3859493e3e05c04e3aaf038d2e4696').   
         See https://test.pypi.org/help/#file-name-reuse for more               
         information.</title>                                                   
          </head>                                                               
          <body>                                                                
           <h1>400 File already exists                                          
         ('cloudpickle-3.2.0.dev0-py3-none-any.whl', with blake2_256 hash       
         '74e6a08c764494daf6ed2ae115ca79af2d3859493e3e05c04e3aaf038d2e4696').   
         See https://test.pypi.org/help/#file-name-reuse for more               
         information.</h1>                                                      
           The server could not comply with the request since it is either      
         malformed or otherwise incorrect.<br/><br/>                            
         File already exists                                                    
         (&#x27;cloudpickle-3.2.0.dev0-py3-none-any.whl&#x27;, with blake2_256  
         hash                                                                   
         &#x27;74e6a08c764494daf6ed2ae115ca79af2d3859493e3e05c04e3aaf038d2e4696&
         #x27;). See https://test.pypi.org/help/#file-name-reuse for more       
         information.                                                           
                                                                                
                                                                                
          </body>                                                               
         </html>                                                                
ERROR    HTTPError: 400 Bad Request from https://test.pypi.org/legacy/          
         Bad Request

unfortunately, flit does not support git-based dev versioning:

pypa/flit#257

So either we switch to an alternative to flit that does, or we do not attempt to upload files with same names but different digests to testpypi.

The skip-existing flag should work:

https://github.com/pypa/gh-action-pypi-publish?tab=readme-ov-file#tolerating-release-package-file-duplicates

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant