Automate docker image creation via GitHub Action CI/CD

[vkuznet]

Impact of the new feature
Automate deployment procedure to cmsweb k8s cluster via GtiHub Action CI/CD pipeline

Is your feature request related to a problem? Please describe.
Reduce operational cost associated with current deployment procedure

Describe the solution you'd like
Use GitHub CI/CD pipeline to trigger automatic builds of all WMCore services upon new tag creation. The build should include the following steps:

• build new release, python tar ball
• upload it to PyPI
• (optionally) upload new image to cmsweb testbed (or dev cluster(s)) k8s cluster via image bot

Describe alternatives you've considered
Some of the functionality exists in Jenkins, but it is limited to testing new release and does not take into account docker image and integration with k8s infrastructure

Additional context
With fully automated builds we can significantly reduce cost of operations, especially if you start managing multiple services.

[vkuznet]

I want to share with you that I was able for the first time to successfully build WMCore in GitHub Action. To make this thing I setup independent repository [1] where I'm testing GitHUb Action builds.

The build is based on Debian distribution (GitHub action does not support cern/cc7) and pypi WMCore build. You can view the GitHub action workflow over here [2].

It almost build everything with only few hick-ups I found during the build:

• there is no gfal2-python package on Debian and so far I did not found clear recipe how to build it manually
• I also found that pycurl requires curl-config which comes from libcurl library which I install manually via apt-get
• I also found that our requirements.txt uses dbs3-client version 4.0.10 depends on pycurl~=7.43.0.6 and build fails since we have pycurl~=7.45
  • for this item I do not really know if we managed to build WMCore via pypi using the requirements.txt Does anybody know?
  • anyway, I replaced the version of pycurl to be 7.43.0.6 and finally got the first GitHub WMCore build using pypi.

This repo [1] and my recipe [2] can be used as initial point in further progress towards CI/CD and GitHub Actions. We need to polish details but the proof-of-concept is there and it shows that we may have a GitHub Action build. The next step would be to make Docker image based on this build, and if we manage it, then I can add action to deploy the image to k8s cluster(s).

[1] https://github.com/vkuznet/wmcore-builds
[2] https://github.com/vkuznet/wmcore-builds/blob/main/.github/workflows/python-app.yml

[amaltaro]

This is great news, Valentin.
Reviewing our recent discussions on PyPi, RPM-less packaging and deployment, I would suggest to reduce the scope of this GH issue to only "automated build of docker image and upload to CERN registry". Otherwise it's very hard to accomplish everything that you mention in the initial description. What do you think?

[vkuznet]

yes, we can easily reduce the scope to only building the image and upload to CERN registry. So far, I'm in a stage of make a normal WMCore build, not even build the image. If someone, e.g. @goughes , will tell me how to invoke python build xxx for specific systems then I can try to build these systems. Then, if it is successful we can build the image. If you have Dockerfiles laying around please also tell me which one to use for which build, otherwise I'll use my experience to use generic Dockerfile and build some image(s).

[vkuznet]

There is an issue with wmagent, and reqmgr2ms installation, see #11327 It should be addressed before we can move forward with this one.

[vkuznet]

Based on suggestion in #11327 (comment) we should adapt building docker images automatically via CI/CD pipeline. The procedure can be used from https://github.com/dmwm/WMArchive/blob/master/.github/workflows/build.yml#L65 till the end of the file which fetches image, then build it and upload to registry and even put it on testbed

[amaltaro]

The scope of this GH is too large. IMO, it should simply request the following deliverables:

• build new docker image and upload it to CERN registry

build to PyPi has been done already. Automatic deployment of images to the kubernetes infrastructure (dev/testbed/production) is likely to be further discussed, and well worth it its own GH issue. Run some basic workflows would also deserve its own GH issue.
@vkuznet can you please update the initial description here?

[vkuznet]

@amaltaro , I updated (simplified) description of this issue and will proceed with it. This is what we discussed on a zoom call, i.e. once we have images we'll automate their creation and upload to registry. The only technical detail we need to agree is tagging naming convention. For instance the tag should include release tag plus pypi prefix or suffix, e.g. 1.2.3-pypi. Do you agree?

[vkuznet]

We can easily move with this issue since now pypi images are built and I even validated reqmgr2, see #11354

@amaltaro I need your input on this and #11339 to move forward with this. Once #11339 is incorporated then we will have full CI/CD in place.