Build WMCore services and publish them to pypy, usinig to tag pushes #11318

khurtado · 2022-10-05T13:11:45Z

Notes:
Packages supported: [wmagent, wmagent-devtools, wmcore, reqmon, reqmgr2, reqmgr2ms, global-workqueue, acdcserver]

t0-reqmon is the same as reqmon (when it comes to building the python package), so we are not building this
Ref:
https://github.com/cms-sw/cmsdist/blob/comp_gcc630/t0_reqmon.spec
https://github.com/cms-sw/cmsdist/blob/comp_gcc630/reqmon.spec
t0 spec relies on 2 github repositories with packages that apparently have the same name. They are already uploading a t0 python package on pypy here: https://pypi.org/project/t0/
https://github.com/cms-sw/cmsdist/blob/comp_gcc630/t0.spec
I'm not sure if we need to rename our package component or not, but this should be a different GH issue

Status

tested in a fork.
We need to upload the pypy secret token inside the WMCore repository for this to work

Description

These changes create a set of github actions to automatically build and publish WMCore services ([wmagent, wmagent-devtools, wmcore, reqmon, reqmgr2, reqmgr2ms, global-workqueue, acdcserver]) to pypy: https://pypi.org/user/cms-oc-dmwm/

Is it backward compatible (if not, which system it affects?)

YES

External dependencies / deployment changes

#11324

cmsdmwmbot · 2022-10-05T13:29:28Z

Jenkins results:

Python3 Unit tests: succeeded
- 1 tests no longer failing
Python3 Pylint check: succeeded
Pylint py3k check: succeeded
Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13607/artifact/artifacts/PullRequestReport.html

cmsdmwmbot · 2022-10-05T13:37:25Z

Jenkins results:

Python3 Unit tests: failed
- 2 new failures
- 2 tests no longer failing
- 1 changes in unstable tests
Python3 Pylint check: succeeded
Pylint py3k check: succeeded
Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13608/artifact/artifacts/PullRequestReport.html

vkuznet · 2022-10-05T13:57:17Z

For the references:

I implemented similar GitHub Action in https://github.com/vkuznet/wmcore-builds
I explained the whole procedure in Automate docker image creation via GitHub Action CI/CD #10921 (comment)
And, once WMCore packages appeared in PyPi I was able to build PyPi based image, e.g. for reqmgr2 service the Dockerfile is the following:

FROM debian:latest
MAINTAINER Valentin Kuznetsov [email protected]
RUN apt-get update
RUN apt-get install -y curl libcurl4 libcurl4-openssl-dev python3-pycurl pip
RUN ln -s /usr/bin/python3 /usr/bin/python
RUN pip install reqmgr2
ENV WDIR=/data
WORKDIR $WDIR
CMD ["python3"]

Please note, the image is debian based, it does not have RPMs, it uses reqmgr2 PyPi package, it is 628MB image size.

But using reqmgr2ms package (and similar Dockefile) I get the following issue:

INFO: pip is looking at multiple versions of py to determine which version is compatible with other requirements. This could take a while.
Collecting py<2.0.0,>=1.4.26
  Downloading py-1.10.0-py2.py3-none-any.whl (97 kB)
  Downloading py-1.9.0-py2.py3-none-any.whl (99 kB)
  Downloading py-1.8.2-py2.py3-none-any.whl (83 kB)
  Downloading py-1.8.1-py2.py3-none-any.whl (83 kB)
  Downloading py-1.8.0-py2.py3-none-any.whl (83 kB)
  Downloading py-1.7.0-py2.py3-none-any.whl (83 kB)
  Downloading py-1.6.0-py2.py3-none-any.whl (83 kB)
  Downloading py-1.5.4-py2.py3-none-any.whl (83 kB)
  Downloading py-1.5.3-py2.py3-none-any.whl (84 kB)
  Downloading py-1.5.2-py2.py3-none-any.whl (88 kB)
  Downloading py-1.5.1-py2.py3-none-any.whl (88 kB)
  Downloading py-1.4.34-py2.py3-none-any.whl (84 kB)
  Downloading py-1.4.33-py2.py3-none-any.whl (83 kB)
  Downloading py-1.4.32-py2.py3-none-any.whl (82 kB)
  Downloading py-1.4.31-py2.py3-none-any.whl (81 kB)
  Downloading py-1.4.30-py2.py3-none-any.whl (81 kB)
  Downloading py-1.4.29-py2.py3-none-any.whl (82 kB)
  Downloading py-1.4.28-py2.py3-none-any.whl (81 kB)
  Downloading py-1.4.27-py2.py3-none-any.whl (81 kB)
  Downloading py-1.4.26.tar.gz (190 kB)
INFO: pip is looking at multiple versions of py to determine which version is compatible with other requirements. This could take a while.
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
INFO: pip is looking at multiple versions of decorator to determine which version is compatible with other requirements. This could take a while.
Collecting decorator>=3.4.2
  Downloading decorator-5.1.0-py3-none-any.whl (9.1 kB)
  Downloading decorator-5.0.9-py3-none-any.whl (8.9 kB)
  Downloading decorator-5.0.8-py3-none-any.whl (8.9 kB)
  Downloading decorator-5.0.7-py3-none-any.whl (8.8 kB)
  Downloading decorator-5.0.6-py3-none-any.whl (8.8 kB)
  Downloading decorator-5.0.5-py3-none-any.whl (8.8 kB)
  Downloading decorator-4.4.2-py2.py3-none-any.whl (9.2 kB)
  Downloading decorator-4.4.1-py2.py3-none-any.whl (9.2 kB)
  Downloading decorator-4.4.0-py2.py3-none-any.whl (8.3 kB)
  Downloading decorator-4.3.2-py2.py3-none-any.whl (9.1 kB)
  Downloading decorator-4.3.1-py2.py3-none-any.whl (8.8 kB)
  Downloading decorator-4.3.0-py2.py3-none-any.whl (9.2 kB)
  Downloading decorator-4.2.1-py2.py3-none-any.whl (9.3 kB)
  Downloading decorator-4.1.2-py2.py3-none-any.whl (9.1 kB)
  Downloading decorator-4.1.1-py2.py3-none-any.whl (9.0 kB)
  Downloading decorator-4.1.0-py2.py3-none-any.whl (9.0 kB)
  Downloading decorator-4.0.11-py2.py3-none-any.whl (8.9 kB)
  Downloading decorator-4.0.10-py2.py3-none-any.whl (9.2 kB)
  Downloading decorator-4.0.9-py2.py3-none-any.whl (22 kB)
  Downloading decorator-4.0.6-py2.py3-none-any.whl (21 kB)
  Downloading decorator-4.0.4-py2.py3-none-any.whl (8.7 kB)
  Downloading decorator-4.0.3-py2.py3-none-any.whl (8.5 kB)
  Downloading decorator-4.0.2-py2.py3-none-any.whl (8.5 kB)
  Downloading decorator-4.0.1-py2.py3-none-any.whl (8.5 kB)
  Downloading decorator-4.0.0-py2.py3-none-any.whl (8.6 kB)
  Downloading decorator-3.4.2.tar.gz (6.1 kB)
INFO: pip is looking at multiple versions of decorator to determine which version is compatible with other requirements. This could take a while.
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
INFO: pip is looking at multiple versions of retry to determine which version is compatible with other requirements. This could take a while.

The bottom line is that we still have dependency chain issue with python packages which will require special attention since building and pushing packages to pypi is half way through, we need to install them to see if our builds provide valid packages.

khurtado · 2022-10-05T18:00:59Z

@vkuznet Wonderful! @ericvaandering is working on splitting reqmgr2ms into different pypy packages at present. Let's come back to the issue you are seeing after he manages to do #10159

cmsdmwmbot · 2022-10-05T18:09:40Z

Jenkins results:

Python3 Unit tests: failed
- 12 new failures
- 2 tests no longer failing
- 2 changes in unstable tests
Python3 Pylint check: succeeded
Pylint py3k check: succeeded
Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13610/artifact/artifacts/PullRequestReport.html

cmsdmwmbot · 2022-10-05T18:38:56Z

Jenkins results:

Python3 Unit tests: failed
- 1 new failures
- 2 tests no longer failing
- 2 changes in unstable tests
Python3 Pylint check: succeeded
Pylint py3k check: succeeded
Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13611/artifact/artifacts/PullRequestReport.html

khurtado · 2022-10-06T16:10:59Z

@germanfgv Do you have any insight on the t0 pypy package that exists already?
https://pypi.org/project/t0/

Do we need to provide support on WMCore or does T0 have that covered?

amaltaro

@khurtado Kenyi, can you please update this PR with the recent changes made by Erik in this PR:
#11324
?

We now have separated microservice builds, and reqmgr2ms no longer needs to be packaged and uploaded to PyPi. @goughes can correct me here as well, in case I am missing anything.

vkuznet · 2022-10-07T17:27:08Z

I created a separate issue about packages which are already in pypi but for some reason which cannot be installed, see #11327 So far I only was able to install reqmgr2, but reqmgr2ms and wmagent are not installable.

khurtado · 2022-10-07T18:56:27Z

@amaltaro I have updated the list of pypy packages to go according to Erik changes.

amaltaro

I guess GH actions are only available for this architecture:

    runs-on: ubuntu-latest

? Don't we have the possibility to use an RPM-based system (RH/Fedora/CentOS)?

Can you please squash these commits? And also ping me on slack, such that we can define the PyPi token. Thanks

.github/workflows/pypy_build_publish_template.yaml

cmsdmwmbot · 2022-10-07T19:11:19Z

Jenkins results:

Python3 Unit tests: failed
- 2 new failures
- 1 tests no longer failing
- 1 changes in unstable tests
Python3 Pylint check: succeeded
Pylint py3k check: succeeded
Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13627/artifact/artifacts/PullRequestReport.html

…) and publish them to pypy, using tag pushes

cmsdmwmbot · 2022-10-07T20:21:28Z

Jenkins results:

Python3 Unit tests: succeeded
- 1 tests no longer failing
- 2 changes in unstable tests
Python3 Pylint check: succeeded
Pylint py3k check: succeeded
Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13628/artifact/artifacts/PullRequestReport.html

cmsdmwmbot · 2022-10-07T20:21:47Z

Jenkins results:

Python3 Unit tests: succeeded
- 1 tests no longer failing
- 1 changes in unstable tests
Python3 Pylint check: succeeded
Pylint py3k check: succeeded
Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13629/artifact/artifacts/PullRequestReport.html

amaltaro

Thanks for the clarification, Kenyi and Valentin.
From our chat on slack, I have created the Pypi token and it should be all set now. I'm creating a new tag soon to run a real test.

amaltaro · 2022-10-07T22:06:06Z

Haaa, nice! In 1min, we managed to build and upload everything to PyPi. Here is one example:
https://pypi.org/project/wmagent/2.1.4rc2/

I know it's unrelated to this PR, but I wonder if the PyPi release number should be 2.1.4.rc2, or the one created 2.1.4rc2? @goughes any thoughts? We might as well refactor our pre-releases to actually release candidates.

khurtado · 2022-10-09T21:55:39Z

@amaltaro I think only .postN and .devN are allowed to have a . after the numerical version numbers. Anything else like rc, a, b, etc can't have a dot, see the PEP 440 naming convention below:

https://peps.python.org/pep-0440/#public-version-identifiers

ericvaandering · 2022-10-11T08:45:46Z

Not me! Sent from a mobile device. Please excuse my brevity or transcription errors. On Oct 5, 2022, at 2:01 PM, Kenyi Hurtado ***@***.***> wrote: @vkuznet<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_vkuznet&d=DwMCaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=EHaoB-POFWGrYFvPXoj1bQ&m=eJSRmgLyrKiLvQLrHU-0XTgiKqj-__yMi9qhJTFxg1ytM6TRw-dZoNNDixFnl0f9&s=tRYgCrwmetTF9GA3nNWiinWliIDw2g0qGIf3ckDrO5M&e=> Wonderful! @ericvaandering<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_ericvaandering&d=DwMCaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=EHaoB-POFWGrYFvPXoj1bQ&m=eJSRmgLyrKiLvQLrHU-0XTgiKqj-__yMi9qhJTFxg1ytM6TRw-dZoNNDixFnl0f9&s=b6D3uXR6HbnvaZoFiJvj3llUvh2Kq8BV8BNh72h-I2c&e=> is working on splitting reqmgr2ms into different pypy packages at present. Let's come back to the issue you are seeing after he manages to do #10159<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_dmwm_WMCore_issues_10159&d=DwMCaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=EHaoB-POFWGrYFvPXoj1bQ&m=eJSRmgLyrKiLvQLrHU-0XTgiKqj-__yMi9qhJTFxg1ytM6TRw-dZoNNDixFnl0f9&s=6DskT2vBNEdCQSzYMjX_Zi1BsDtfBHQhJW2siFzYHuw&e=> — Reply to this email directly, view it on GitHub<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_dmwm_WMCore_pull_11318-23issuecomment-2D1268760649&d=DwMCaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=EHaoB-POFWGrYFvPXoj1bQ&m=eJSRmgLyrKiLvQLrHU-0XTgiKqj-__yMi9qhJTFxg1ytM6TRw-dZoNNDixFnl0f9&s=rOujeohSZU0u8BJzMHO9NgktqHd3q9TcaVzBlccArIw&e=>, or unsubscribe<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AAMYJLWRM3NGIAADK6U4QCLWBW66NANCNFSM6AAAAAAQ5SF2TI&d=DwMCaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=EHaoB-POFWGrYFvPXoj1bQ&m=eJSRmgLyrKiLvQLrHU-0XTgiKqj-__yMi9qhJTFxg1ytM6TRw-dZoNNDixFnl0f9&s=g_pJqRkZu1fqpqn6xYpCtG5-gU97BY5aARqRgX9J6wY&e=>. You are receiving this because you were mentioned.Message ID: ***@***.***>

khurtado added PR: Do not merge yet PR: Work in progress labels Oct 5, 2022

khurtado force-pushed the pypy branch from dd0d919 to 134e469 Compare October 5, 2022 13:25

khurtado force-pushed the pypy branch from 134e469 to c0e98c1 Compare October 5, 2022 17:55

khurtado mentioned this pull request Oct 5, 2022

Automated packaging and upload of WMCore services to Pypi #11276

Closed

khurtado removed the PR: Work in progress label Oct 5, 2022

khurtado requested a review from amaltaro October 5, 2022 18:15

khurtado force-pushed the pypy branch from c0e98c1 to 73246dd Compare October 5, 2022 18:29

amaltaro reviewed Oct 7, 2022

View reviewed changes

amaltaro requested changes Oct 7, 2022

View reviewed changes

.github/workflows/pypy_build_publish_template.yaml Show resolved Hide resolved

khurtado force-pushed the pypy branch from e98d3af to 753538e Compare October 7, 2022 20:06

Build WMCore services (except for t0 right now, due to name conflicts…

d50c2d5

…) and publish them to pypy, using tag pushes

khurtado force-pushed the pypy branch from 753538e to d50c2d5 Compare October 7, 2022 20:08

amaltaro approved these changes Oct 7, 2022

View reviewed changes

amaltaro merged commit 28be7dc into dmwm:master Oct 7, 2022

amaltaro mentioned this pull request Oct 10, 2022

Build and upload PyPI packages with each WMCore release #10403

Closed

khurtado mentioned this pull request Oct 11, 2022

Exclude GH actions to match with tags starting with JENKINS #11337

Merged

vkuznet mentioned this pull request Oct 11, 2022

Add initial draft for building WMCore images via CI/CD #11339

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build WMCore services and publish them to pypy, usinig to tag pushes #11318

Build WMCore services and publish them to pypy, usinig to tag pushes #11318

khurtado commented Oct 5, 2022 •

edited

Loading

cmsdmwmbot commented Oct 5, 2022

cmsdmwmbot commented Oct 5, 2022

vkuznet commented Oct 5, 2022

khurtado commented Oct 5, 2022

cmsdmwmbot commented Oct 5, 2022

cmsdmwmbot commented Oct 5, 2022

khurtado commented Oct 6, 2022

amaltaro left a comment

vkuznet commented Oct 7, 2022

khurtado commented Oct 7, 2022

amaltaro left a comment

cmsdmwmbot commented Oct 7, 2022

cmsdmwmbot commented Oct 7, 2022

cmsdmwmbot commented Oct 7, 2022

amaltaro left a comment

amaltaro commented Oct 7, 2022

khurtado commented Oct 9, 2022

ericvaandering commented Oct 11, 2022 via email

Build WMCore services and publish them to pypy, usinig to tag pushes #11318

Build WMCore services and publish them to pypy, usinig to tag pushes #11318

Conversation

khurtado commented Oct 5, 2022 • edited Loading

Status

Description

Is it backward compatible (if not, which system it affects?)

External dependencies / deployment changes

cmsdmwmbot commented Oct 5, 2022

cmsdmwmbot commented Oct 5, 2022

vkuznet commented Oct 5, 2022

khurtado commented Oct 5, 2022

cmsdmwmbot commented Oct 5, 2022

cmsdmwmbot commented Oct 5, 2022

khurtado commented Oct 6, 2022

amaltaro left a comment

Choose a reason for hiding this comment

vkuznet commented Oct 7, 2022

khurtado commented Oct 7, 2022

amaltaro left a comment

Choose a reason for hiding this comment

cmsdmwmbot commented Oct 7, 2022

cmsdmwmbot commented Oct 7, 2022

cmsdmwmbot commented Oct 7, 2022

amaltaro left a comment

Choose a reason for hiding this comment

amaltaro commented Oct 7, 2022

khurtado commented Oct 9, 2022

ericvaandering commented Oct 11, 2022 via email

khurtado commented Oct 5, 2022 •

edited

Loading