Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add initial draft for building WMCore images via CI/CD #11339

Closed
wants to merge 6 commits into from
Closed

Add initial draft for building WMCore images via CI/CD #11339

Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
9f95715
Add initial draft for building WMCore images via CI/CD
vkuznet Oct 11, 2022
0d1835e
Fix steps.get-ref parts
vkuznet Oct 13, 2022
fa51a70
Added workqueue and t0_reqmon
vkuznet Oct 14, 2022
82904e8
Added reqmgr2 service
vkuznet Oct 14, 2022
07cb8e9
Adjust tagging convention to match other build workflows
vkuznet Oct 14, 2022
18e98d5
Switch from Dockerfile to Dockerfile.dist
vkuznet Oct 28, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
129 changes: 129 additions & 0 deletions .github/workflows/images.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,129 @@
name: Build

on:
push:
tags:
- '*'
- '!JENKINS*'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given that we are currently struggling with this syntax in the PyPi workflow, I checked the documentation and it seems to be one of the ways to allow/disallow actions based on the tag name.
Another option would be:

tags-ignore:
 - 'JENKINS*'

as documented in: https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#onpushbranchestagsbranches-ignoretags-ignore

Just an FYI @khurtado


jobs:

build:
name: Build
runs-on: ubuntu-latest
steps:

- name: Get the Ref
id: get-ref
uses: ankitvgupta/ref-to-tag-action@master
with:
ref: ${{ github.ref }}
head_ref: ${{ github.head_ref }}

- name: Build wmagent image
run: |
curl -ksLO https://raw.githubusercontent.com/dmwm/CMSKubernetes/master/docker/pypi/wmagent/Dockerfile.dist
sed -i -e "s,ENV TAG=.*,ENV TAG=${{steps.get-ref.outputs.tag}},g" Dockerfile.dist
cat Dockerfile.dist
docker build . --tag registry.cern.ch/cmsweb/wmagent

- name: Build workqueue image
run: |
curl -ksLO https://raw.githubusercontent.com/dmwm/CMSKubernetes/master/docker/pypi/workqueue/Dockerfile.dist
sed -i -e "s,ENV TAG=.*,ENV TAG=${{steps.get-ref.outputs.tag}},g" Dockerfile.dist
cat Dockerfile.dist
docker build . --tag registry.cern.ch/cmsweb/workqueue

- name: Build reqmon image
run: |
curl -ksLO https://raw.githubusercontent.com/dmwm/CMSKubernetes/master/docker/pypi/reqmon/Dockerfile.dist
sed -i -e "s,ENV TAG=.*,ENV TAG=${{steps.get-ref.outputs.tag}},g" Dockerfile.dist
cat Dockerfile.dist
docker build . --tag registry.cern.ch/cmsweb/reqmon

- name: Build t0_reqmon image
run: |
curl -ksLO https://raw.githubusercontent.com/dmwm/CMSKubernetes/master/docker/pypi/t0_reqmon/Dockerfile.dist
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We still do not have this t0_reqmon package in PyPi. I need to review it, but I think the Dockerfile will have to:

RUN pip install reqmon-$TAG.tar.gz

we need to investigate if:

  • t0_reqmon is even expected to be packaged in WMCore
  • if so, if the package is the same as reqmon. It could be that it's during service start up that we decide which CouchApps to use and whatnot.

sed -i -e "s,ENV TAG=.*,ENV TAG=${{steps.get-ref.outputs.tag}},g" Dockerfile.dist
cat Dockerfile.dist
docker build . --tag registry.cern.ch/cmsweb/t0_reqmon

- name: Build reqmgr2 image
run: |
curl -ksLO https://raw.githubusercontent.com/dmwm/CMSKubernetes/master/docker/pypi/reqmgr2/Dockerfile.dist
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This Dockerfile seems to be incorrect.

sed -i -e "s,ENV TAG=.*,ENV TAG=${{steps.get-ref.outputs.tag}},g" Dockerfile.dist
cat Dockerfile.dist
docker build . --tag registry.cern.ch/cmsweb/reqmgr2

- name: Build reqmgr2ms-output image
run: |
curl -ksLO https://raw.githubusercontent.com/dmwm/CMSKubernetes/master/docker/pypi/reqmgr2ms-output/Dockerfile.dist
sed -i -e "s,ENV TAG=.*,ENV TAG=${{steps.get-ref.outputs.tag}},g" Dockerfile.dist
cat Dockerfile.dist
docker build . --tag registry.cern.ch/cmsweb/reqmgr2ms-output

- name: Build reqmgr2ms-rulecleaner image
run: |
curl -ksLO https://raw.githubusercontent.com/dmwm/CMSKubernetes/master/docker/pypi/reqmgr2ms-rulecleaner/Dockerfile.dist
sed -i -e "s,ENV TAG=.*,ENV TAG=${{steps.get-ref.outputs.tag}},g" Dockerfile.dist
cat Dockerfile.dist
docker build . --tag registry.cern.ch/cmsweb/reqmgr2ms-rulecleaner

- name: Build reqmgr2ms-transferor image
run: |
curl -ksLO https://raw.githubusercontent.com/dmwm/CMSKubernetes/master/docker/pypi/reqmgr2ms-transferor/Dockerfile.dist
sed -i -e "s,ENV TAG=.*,ENV TAG=${{steps.get-ref.outputs.tag}},g" Dockerfile.dist
cat Dockerfile.dist
docker build . --tag registry.cern.ch/cmsweb/reqmgr2ms-transferor

- name: Build reqmgr2ms-monitor image
run: |
curl -ksLO https://raw.githubusercontent.com/dmwm/CMSKubernetes/master/docker/pypi/reqmgr2ms-monitor/Dockerfile.dist
sed -i -e "s,ENV TAG=.*,ENV TAG=${{steps.get-ref.outputs.tag}},g" Dockerfile.dist
cat Dockerfile.dist
docker build . --tag registry.cern.ch/cmsweb/reqmgr2ms-monitor

- name: Build reqmgr2ms-unmerged image
run: |
curl -ksLO https://raw.githubusercontent.com/dmwm/CMSKubernetes/master/docker/pypi/reqmgr2ms-unmerged/Dockerfile.dist
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For this MSUnmerged, I see that the Dockerfile actually depends on two images:

FROM registry.cern.ch/cmsweb/gfal:latest as gfal
FROM python:3.8-bullseye

can you please point to this gfal Dockerfile?

sed -i -e "s,ENV TAG=.*,ENV TAG=${{steps.get-ref.outputs.tag}},g" Dockerfile.dist
cat Dockerfile.dist
docker build . --tag registry.cern.ch/cmsweb/reqmgr2ms-unmerged

# - name: Login to registry.cern.ch
# uses: docker/[email protected]
# with:
# registry: registry.cern.ch
# username: ${{ secrets.CERN_LOGIN }}
# password: ${{ secrets.CERN_TOKEN }}

# - name: Publish image to registry.cern.ch
# uses: docker/build-push-action@v1
# with:
# username: ${{ secrets.CERN_LOGIN }}
# password: ${{ secrets.CERN_TOKEN }}
# registry: registry.cern.ch
# repository: cmsweb/wmarchive
# tag_with_ref: true

# - name: Login to docker github registry
# uses: docker/[email protected]
# with:
# registry: docker.pkg.github.com
# username: ${{ github.actor }}
# password: ${{ secrets.GITHUB_TOKEN }}

# - name: Login to Registry
# uses: docker/[email protected]
# with:
# registry: docker.pkg.github.com
# username: ${{ github.actor }}
# password: ${{ secrets.GITHUB_TOKEN }}

# - name: Push new image to k8s
# run: |
# curl -ksLO https://raw.githubusercontent.com/vkuznet/imagebot/main/imagebot.sh
# sed -i -e "s,COMMIT,${{github.sha}},g" -e "s,REPOSITORY,${{github.repository}},g" -e "s,NAMESPACE,wma,g" -e "s,TAG,${{steps.get-ref.outputs.tag}},g" -e "s,IMAGE,registry.cern.ch/cmsweb/wmarchive,g" -e "s,SERVICE,wmarchive,g" -e "s,HOST,${{secrets.IMAGEBOT_URL}},g" imagebot.sh
# chmod +x imagebot.sh
# cat imagebot.sh
# sh ./imagebot.sh