Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(ci): add image scanning workflow #1161

Merged
merged 37 commits into from
Apr 21, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
5e88a22
Add scaffolding for image scanning workflow
p5 Apr 19, 2024
e1893fb
Add SBOM generation and scanning steps
p5 Apr 19, 2024
ab35ae5
Remove unused install-grype step
p5 Apr 19, 2024
39fa2d3
Generate unique artifact name
p5 Apr 19, 2024
66b1501
Enable some builds to debug build container outputs
p5 Apr 19, 2024
0416d3a
Skip long-running build steps
p5 Apr 19, 2024
a50ed5a
Enable builds and create workflow output
p5 Apr 19, 2024
f0c8e57
Unsure why I enabled the builds...
p5 Apr 19, 2024
dfaf874
Tie it all together
p5 Apr 19, 2024
9e0f9e2
Fix outputs and enable single nvidia build
p5 Apr 19, 2024
e174a29
Try reworking the jq command to output array rather than space-separa…
p5 Apr 19, 2024
82ed6b7
Minify JSON
p5 Apr 19, 2024
e8a824e
Try dummy array
p5 Apr 19, 2024
c84ab17
Fix output ref|
p5 Apr 19, 2024
20f78c4
Revert to real outputs
p5 Apr 19, 2024
057b156
Share images through job artifacts
p5 Apr 19, 2024
78131fe
Fix file names to be included in artifacts
p5 Apr 19, 2024
ceed21e
Add a merge before downloadingi artifacts
p5 Apr 20, 2024
a3e5c10
Use unique file names
p5 Apr 20, 2024
808ce62
Fix upload-artifact target file
p5 Apr 20, 2024
be1e73a
Another attempt at targeting correct file
p5 Apr 20, 2024
dd71e89
Create job output in correct format
p5 Apr 20, 2024
6f88990
Use @ to separate image name and digest
p5 Apr 20, 2024
0404a7d
Disable fail-fast and fix 38 digest
p5 Apr 20, 2024
fe438fa
Clear storage space before generating SBOM
p5 Apr 20, 2024
0edc9ca
Cleanup artifacts, enable verbose syft and exclude ostree files from …
p5 Apr 20, 2024
50ab63f
Enable syft debug logging
p5 Apr 20, 2024
ca96412
Remove incorrect verbose arg
p5 Apr 20, 2024
8457a85
Enable scans for only F38
p5 Apr 20, 2024
a9a2307
Enable other builds again
p5 Apr 20, 2024
b2ffb3a
Disable syft verbose logging and remove unexpected changes
p5 Apr 20, 2024
2af97b2
Remove unexpected changes
p5 Apr 20, 2024
cc469f8
Update .github/workflows/reusable-image-scan.yml
p5 Apr 20, 2024
30dc0da
Do not run scan on PRs
p5 Apr 20, 2024
bf323c3
Move check step to end (to ensure outputs are set)
p5 Apr 20, 2024
8f1511c
Enable for all Bluefin builds
p5 Apr 20, 2024
a9238b7
Merge branch 'main' into sbom-generation-image-scanning
p5 Apr 21, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .github/syft.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
# Exclude some files Syft will never get anything useful from
exclude:
- '/sysroot/ostree/repo/objects/**'
- '/usr/share/icons/**'
- '/usr/share/doc/**'
14 changes: 12 additions & 2 deletions .github/workflows/build-38-bluefin.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ on:
- '**.md'
- 'system_files/kinoite/**'
schedule:
- cron: '42 16 * * *' # 16:42 UTC everyday
- cron: '42 16 * * *' # 16:42 UTC everyday
workflow_dispatch:

jobs:
Expand All @@ -19,4 +19,14 @@ jobs:
secrets: inherit
with:
brand_name: bluefin
fedora_version: 38
fedora_version: 38

scan:
# Scan can still be ran when some builds fail since only successfully built
# images will be stored in the output
if: github.event_name != 'pull_request' && always()
uses: ./.github/workflows/reusable-image-scan.yml
needs: build
secrets: inherit
with:
images: ${{ needs.build.outputs.images }}
12 changes: 11 additions & 1 deletion .github/workflows/build-39-bluefin.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,4 +19,14 @@ jobs:
secrets: inherit
with:
brand_name: bluefin
fedora_version: 39
fedora_version: 39

scan:
# Scan can still be ran when some builds fail since only successfully built
# images will be stored in the output
if: github.event_name != 'pull_request' && always()
uses: ./.github/workflows/reusable-image-scan.yml
needs: build
secrets: inherit
with:
images: ${{ needs.build.outputs.images }}
12 changes: 11 additions & 1 deletion .github/workflows/build-40-bluefin.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,4 +19,14 @@ jobs:
secrets: inherit
with:
brand_name: bluefin
fedora_version: 40
fedora_version: 40

scan:
# Scan can still be ran when some builds fail since only successfully built
# images will be stored in the output
if: github.event_name != 'pull_request' && always()
uses: ./.github/workflows/reusable-image-scan.yml
needs: build
secrets: inherit
with:
images: ${{ needs.build.outputs.images }}
81 changes: 70 additions & 11 deletions .github/workflows/reusable-build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,10 @@ on:
description: "'aurora' or 'bluefin'"
required: true
type: string
outputs:
images:
description: 'An array of images built and pushed to the registry'
value: ${{ jobs.check.outputs.images }}
env:
IMAGE_REGISTRY: ghcr.io/${{ github.repository_owner }}

Expand All @@ -22,6 +26,8 @@ jobs:
name: image
runs-on: ubuntu-latest
continue-on-error: false
outputs:
image_full: ${{ steps.generate-outputs.outputs.image }}
strategy:
fail-fast: false
matrix:
Expand Down Expand Up @@ -262,24 +268,77 @@ jobs:
COSIGN_EXPERIMENTAL: false
COSIGN_PRIVATE_KEY: ${{ secrets.SIGNING_SECRET }}

- name: Echo outputs
- name: Generate file containing outputs
if: github.event_name != 'pull_request'
run: |
echo "${{ toJSON(steps.push.outputs) }}"
env:
DIGEST: ${{ steps.push.outputs.digest }}
IMAGE_REGISTRY: ${{ env.IMAGE_REGISTRY }}/${{ env.IMAGE_NAME }}
IMAGE_NAME: ${{ env.IMAGE_NAME }}
IMAGE_FLAVOR: ${{ matrix.image_flavor }}
FEDORA_VERSION: ${{ matrix.fedora_version }}
run:
echo "${IMAGE_REGISTRY}@${DIGEST}" > "${IMAGE_NAME}-${IMAGE_FLAVOR}-${FEDORA_VERSION}.txt"

- name: Upload artifact
if: github.event_name != 'pull_request'
uses: actions/upload-artifact@v4
with:
name: image-${{ env.IMAGE_NAME }}-${{ matrix.image_flavor }}-${{ matrix.fedora_version }}
retention-days: 1
if-no-files-found: error
path: |
${{ env.IMAGE_NAME }}-${{ matrix.image_flavor }}-${{ matrix.fedora_version }}.txt
check:
name: Check all ${{ inputs.brand_name }} ${{ inputs.fedora_version }} builds successful
if: ${{ !cancelled() }}
if: always()
runs-on: ubuntu-latest
needs: [build_container]
outputs:
images: ${{ steps.generate-outputs.outputs.images }}
steps:
- name: Exit on failure
if: ${{ needs.build_container.result == 'failure' }}
shell: bash
run: exit 1
- name: Exit
shell: bash
run: exit 0
- name: Download artifacts
if: github.event_name != 'pull_request'
id: download-artifacts
uses: actions/download-artifact@v4
with:
pattern: image-*
merge-multiple: true

- name: Create output
if: github.event_name != 'pull_request'
id: generate-outputs
env:
JOBS: ${{ toJson(needs) }}
ARTIFACT_PATH: ${{ steps.download-artifacts.outputs.download-path }}
run: |
# Initialize the array
images=()
# Populate the array with each line from each file in the artifacts directory
for file in $ARTIFACT_PATH/*; do
while IFS= read -r line; do
images+=("$line")
done < "$file"
done
# Create the GITHUB_OUTPUT in the format '["image1", "image2", ...]'
echo "images=$(printf '%s\n' "${images[@]}" | jq -R -s -c 'split("\n") | .[:-1]')" >> $GITHUB_OUTPUT
- name: Check Jobs
env:
JOBS: ${{ toJson(needs) }}
run: |
echo "Job status:"
echo $JOBS | jq -r 'to_entries[] | " - \(.key): \(.value.result)"'
for i in $(echo $JOBS | jq -r 'to_entries[] | .value.result'); do
if [ "$i" != "success" ] && [ "$i" != "skipped" ]; then
echo ""
echo "Status check not okay!"
exit 1
fi
done
build_iso:
name: iso
Expand Down
73 changes: 73 additions & 0 deletions .github/workflows/reusable-image-scan.yml
Copy link
Member Author

@p5 p5 Apr 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO we should centralise this configuration in ublue-os/main or somewhere before implementing scanning in other repos. There's nothing bluefin-specific in here, so no need to copy & paste it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can start here as a proof of concept.

We could then move it as it's own repo and have it be a callable action if we genericize it enough.

Copy link
Member Author

@p5 p5 Apr 21, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I plan to add things like uploading the SERIF files to GitHub, upload SBOMs to R2 etc. But that all comes later. Right now, I just want to build the files and push them to job artifacts in only this repo.

None of this will be specific to ublue and/or bluefin

Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
name: Scan Image
on:
workflow_call:
inputs:
images:
description: "A comma-separated list of images to scan. E.G. '[\"docker.io/library/alpine:3.14.0\", \"docker.io/library/alpine:3.13.6\"]'"
required: true
type: string

jobs:
generate-matrix:
runs-on: ubuntu-latest
outputs:
matrix: ${{ steps.set-matrix.outputs.matrix }}
steps:
- name: Set matrix
id: set-matrix
env:
IMAGES: ${{ inputs.images }}
run: |
echo "matrix=$IMAGES" >> $GITHUB_OUTPUT
scan-image:
needs: generate-matrix
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
image: ${{fromJson(needs.generate-matrix.outputs.matrix)}}
steps:
- name: Checkout repository
uses: actions/checkout@v4

- name: Maximize build space
uses: ublue-os/remove-unwanted-software@v6

- name: Install Syft
shell: bash
run: |
curl -sSfL https://raw.githubusercontent.com/anchore/syft/main/install.sh | sh -s -- -b /usr/local/bin
syft version
- name: Generate SBOM
env:
IMAGE: ${{ matrix.image }}
run: |
syft ${IMAGE} \
--output cyclonedx-json=sbom.json \
--config ./.github/syft.yml
- name: Scan SBOM
id: scan
uses: anchore/scan-action@3343887d815d7b07465f6fdcd395bd66508d486a # v3
with:
sbom: sbom.json
output-format: json
fail-build: false

- name: Generate artifact name
id: artifact-name
env:
IMAGE: ${{ matrix.image }}
run: |
echo "name=$(echo ${IMAGE} | awk -F'/' '{print $NF}' | sed 's/:/-/g')" >> $GITHUB_OUTPUT
- name: Upload scan results
uses: actions/upload-artifact@v4
with:
name: security-${{ steps.artifact-name.outputs.name }}
if-no-files-found: error
path: |
sbom.json
${{ steps.scan.outputs.json }}
Loading