Feat/build tooling (#78)

* Blah blah * Made atils not fail if there is no kubernetes configuration available * Did a bunch * Added the dockerfile at its new, lower user * Some stuff * This should be fine * Adding our first CI workflow * You'll see * Didn't get it all * Got the delete * Adding docker login to actions * Let's see this build * Getting our CI right * Fixing our workflow * Bumping version to test our CI * Think this is right? * Grabbed the wrong thing * Buncha stuff * Alright, think this will run * That was silly * Here we go * We're almost there * Wrong order * Who knows * Can't use absolute paths in our CI env * Need to set up Docker to build multi-arch images * Updating atils description, mostly to trigger a build * Hopefully this works * Fixing this * Updating the atils-ci image, and ensuring it can build * Made some changes * That's annoying * Think we got it * What * Test * This needs to work * Switching back to buildx? * Need the setup actions for cross-arch builds * Bumping minor version to force rebuild * Pretty dumb * Finalizing on the manifest build, rather than all-in-one * Had the wrong actions for the ci-build-publish action * Forgot the context for docker buildx * That was dumb * Fixing more stupidity * Setting up build for toolshed. Also updated the backstage version * Think we got it
AidanHilt · Mar 15, 2024 · 4c14f87 · 4c14f87
1 parent 1c3f0d9
commit 4c14f87
Show file tree

Hide file tree

Showing 11 changed files with 1,969 additions and 1,156 deletions.
diff --git a/atils/docs/roadmap.md b/atils/docs/roadmap.md
@@ -0,0 +1,16 @@
+# Roadmap
+
+## Tests
+We want to create tests for all of our testable functions. By testable functions, we mean either functions that have a return value that we can check, or tests where we can easily mock the calls it makes. We'll get into specifics more as we take up that project
+
+## Better PVC Management
+We have a PVC manager job, but its hideous. It creates a job that just runs for a while, then creates an exec command that you have to paste in. Let's fix it with a different hideous solution, that does the job if the PVC is unattached, and otherwise creates an ephemeral container that mounts the volume, then lets you mess with stuff.
+
+### Create a working exec function
+### Patch the pod
+
+## Parallel Build
+Let certain build steps run in parallel. No idea how that'll work.
+
+## Docker Compose Stacks
+We have some fun little Docker compose stacks, for dependencies we might have like a postgres DB or the like. Let's figure out a way to manage the startup and teardown from atils
diff --git a/atils/mkdocs.yaml b/atils/mkdocs.yaml
@@ -10,3 +10,4 @@ nav:
   - jobs: jobs.md
   - kubernetes: kubernetes.md
   - vault: vault.md
+  - Roadmap: roadmap.md
diff --git a/custom-images/atils-ci/.atils_buildconfig.json b/custom-images/atils-ci/.atils_buildconfig.json
@@ -45,7 +45,7 @@
       "name": "create-manifest-latest",
       "command": ". ./config.env; docker manifest create $IMAGE_NAME:latest -a $IMAGE_NAME:$IMAGE_TAG-arm64 -a $IMAGE_NAME:$IMAGE_TAG-amd64",
       "order": 15,
-      "description": "Create the manifest for the multi-arch tagged release"
+      "description": "Create the manifest for the multi-arch latest release"
     },
     {
       "name": "push-manifest-versioned",
@@ -58,7 +58,7 @@
       "name": "push-manifest-latest",
       "command": ". ./config.env; docker manifest push $IMAGE_NAME:latest",
       "order": 17,
-      "description": "Push the manifest for the multi-arch tagged release",
+      "description": "Push the manifest for the multi-arch latest release",
       "ci_only": true
     }
   ],

diff --git a/custom-images/atils-ci/Dockerfile b/custom-images/atils-ci/Dockerfile
@@ -4,20 +4,27 @@ USER root
 
 SHELL ["/bin/bash", "-c"]
 
+# Install all the apt packages we'll need
 RUN apt-get update && \
-    apt-get install -y python3 pipx apt-transport-https ca-certificates curl gnupg-agent software-properties-common && \
-    apt-get install ca-certificates curl && \
-    install -m 0755 -d /etc/apt/keyrings && \
-    curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc && \
-    chmod a+r /etc/apt/keyrings/docker.asc
+    apt-get install -y python3 pipx apt-transport-https ca-certificates curl gnupg-agent software-properties-common ca-certificates curl jq
+
 
-RUN echo \
+# Get us ready for, then install Docker and buildx. This image runs on GH action runners, so we don't need to set up or enable buildx here
+RUN install -m 0755 -d /etc/apt/keyrings && \
+    curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc && \
+    chmod a+r /etc/apt/keyrings/docker.asc && \
+    echo \
     "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian \
     $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
     tee /etc/apt/sources.list.d/docker.list > /dev/null && \
     apt-get update && \
     apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
 
-RUN pip3 install poetry
+# Install npm, yarn, and all that. We use this to build our Toolshed image
+RUN curl -fsSL https://deb.nodesource.com/setup_20.x | bash - && \
+    apt-get install -y nodejs && \
+    corepack enable
+
+RUN pip3 install poetry check-jsonschema
 
 ENV ATILS_CI_ENV=true
diff --git a/mac-setup/homebrew/Brewfile b/mac-setup/homebrew/Brewfile
@@ -56,4 +56,6 @@ brew 'openjdk'
 brew 'nvm'
 brew 'yarn'
 brew 'libpq'
-brew 'check-jsonschema'
+brew 'check-jsonschema'
+brew 'jq'
+brew 'yq'
diff --git a/toolshed/.atils_buildconfig.json b/toolshed/.atils_buildconfig.json
@@ -6,25 +6,94 @@
       "order": 0
     },
     {
-      "name": "install",
+      "name": "install-ci",
       "command": "yarn install --frozen-lockfile",
       "order": 10
     },
+    {
+      "name": "install-dev",
+      "command": "yarn install",
+      "order": 11
+    },
     {
       "name": "type-definitions",
       "command": "yarn tsc",
-      "order": 11
+      "order": 12
     },
     {
       "name": "build-backend",
       "command": "yarn build:backend --config ../../app-config.yaml",
-      "order": 12
+      "order": 13
     },
     {
-      "name": "docker-build",
-      "command": "docker build .",
-      "order": 13
+      "name": "docker-build-local",
+      "command": "docker build . --file packages/backend/Dockerfile -t aidanhilt/toolshed:latest --load",
+      "order": 14
+    },
+    {
+      "name": "docker-build-and-push-arm64",
+      "command": "VERSION=$(cat backstage.json | jq -r '.version'); docker buildx build . --file packages/backend/Dockerfile -t aidanhilt/toolshed:$VERSION-arm64 --push",
+      "order": 20,
+      "ci_only": true
+    },
+    {
+      "name": "docker-build-and-push-amd64",
+      "command": "VERSION=$(cat backstage.json | jq -r '.version'); docker buildx build . --file packages/backend/Dockerfile -t aidanhilt/toolshed:$VERSION-amd64 --push",
+      "order": 21,
+      "ci_only": true
+    },
+    {
+      "name": "create-manifest-versioned",
+      "command": "VERSION=$(cat backstage.json | jq -r '.version'); docker manifest create aidanhilt/toolshed:$VERSION -a aidanhilt/toolshed:$VERSION-arm64 -a aidanhilt/toolshed:$VERSION-amd64",
+      "order": 22,
+      "description": "Create the manifest for the multi-arch tagged release"
+    },
+    {
+      "name": "create-manifest-latest",
+      "command": "VERSION=$(cat backstage.json | jq -r '.version'); docker manifest create aidanhilt/toolshed:latest -a aidanhilt/toolshed:$VERSION-arm64 -a aidanhilt/toolshed:$VERSION-amd64",
+      "order": 23,
+      "description": "Create the manifest for the multi-arch latest release"
+    },
+    {
+      "name": "push-manifest-versioned",
+      "command": "VERSION=$(cat backstage.json | jq -r '.version'); docker manifest push aidanhilt/toolshed:$VERSION",
+      "order": 24,
+      "description": "Push the manifest for the multi-arch tagged release",
+      "ci_only": true
+    },
+    {
+      "name": "push-manifest-latest",
+      "command": "VERSION=$(cat backstage.json | jq -r '.version'); docker manifest push aidanhilt/toolshed:latest",
+      "order": 25,
+      "description": "Push the manifest for the multi-arch latest release",
+      "ci_only": true
     }
   ],
-  "action_sets": []
+  "action_sets": [
+    {
+      "name": "local-build",
+      "actions": [
+        "validate-buildconfig",
+        "install-dev",
+        "type-definitions",
+        "build-backend",
+        "docker-build-local"
+      ]
+    },
+    {
+      "name": "ci-build-publish",
+      "actions": [
+        "validate-buildconfig",
+        "install-ci",
+        "type-definitions",
+        "build-backend",
+        "docker-build-and-push-arm64",
+        "docker-build-and-push-amd64",
+        "create-manifest-versioned",
+        "create-manifest-latest",
+        "push-manifest-versioned",
+        "push-manifest-latest"
+      ]
+    }
+  ]
 }
diff --git a/toolshed/backstage.json b/toolshed/backstage.json
@@ -1,3 +1,3 @@
 {
-  "version": "1.22.1"
+  "version": "1.23.4"
 }
diff --git a/toolshed/package.json b/toolshed/package.json
@@ -31,8 +31,8 @@
     ]
   },
   "devDependencies": {
-    "@backstage/cli": "^0.25.1",
-    "@backstage/e2e-test-utils": "^0.1.0",
+    "@backstage/cli": "^0.25.2",
+    "@backstage/e2e-test-utils": "^0.1.1",
     "@playwright/test": "^1.32.3",
     "@spotify/prettier-config": "^12.0.0",
     "concurrently": "^8.0.0",

diff --git a/toolshed/packages/app/package.json b/toolshed/packages/app/package.json
@@ -14,31 +14,31 @@
     "lint": "backstage-cli package lint"
   },
   "dependencies": {
-    "@backstage/app-defaults": "^1.4.7",
-    "@backstage/catalog-model": "^1.4.3",
-    "@backstage/cli": "^0.25.1",
-    "@backstage/core-app-api": "^1.11.3",
-    "@backstage/core-components": "^0.13.10",
-    "@backstage/core-plugin-api": "^1.8.2",
-    "@backstage/integration-react": "^1.1.23",
-    "@backstage/plugin-api-docs": "^0.10.3",
-    "@backstage/plugin-catalog": "^1.16.1",
-    "@backstage/plugin-catalog-common": "^1.0.20",
-    "@backstage/plugin-catalog-graph": "^0.3.3",
-    "@backstage/plugin-catalog-import": "^0.10.5",
-    "@backstage/plugin-catalog-react": "^1.9.3",
-    "@backstage/plugin-github-actions": "^0.6.10",
-    "@backstage/plugin-org": "^0.6.19",
-    "@backstage/plugin-permission-react": "^0.4.19",
-    "@backstage/plugin-scaffolder": "^1.17.1",
-    "@backstage/plugin-search": "^1.4.5",
-    "@backstage/plugin-search-react": "^1.7.5",
-    "@backstage/plugin-tech-radar": "^0.6.12",
-    "@backstage/plugin-techdocs": "^1.9.3",
-    "@backstage/plugin-techdocs-module-addons-contrib": "^1.1.4",
-    "@backstage/plugin-techdocs-react": "^1.1.15",
-    "@backstage/plugin-user-settings": "^0.8.0",
-    "@backstage/theme": "^0.5.0",
+    "@backstage/app-defaults": "^1.5.0",
+    "@backstage/catalog-model": "^1.4.4",
+    "@backstage/cli": "^0.25.2",
+    "@backstage/core-app-api": "^1.12.0",
+    "@backstage/core-components": "^0.14.0",
+    "@backstage/core-plugin-api": "^1.9.0",
+    "@backstage/integration-react": "^1.1.24",
+    "@backstage/plugin-api-docs": "^0.11.0",
+    "@backstage/plugin-catalog": "^1.17.0",
+    "@backstage/plugin-catalog-common": "^1.0.21",
+    "@backstage/plugin-catalog-graph": "^0.4.0",
+    "@backstage/plugin-catalog-import": "^0.10.6",
+    "@backstage/plugin-catalog-react": "^1.10.0",
+    "@backstage/plugin-github-actions": "^0.6.11",
+    "@backstage/plugin-org": "^0.6.20",
+    "@backstage/plugin-permission-react": "^0.4.20",
+    "@backstage/plugin-scaffolder": "^1.18.0",
+    "@backstage/plugin-search": "^1.4.6",
+    "@backstage/plugin-search-react": "^1.7.6",
+    "@backstage/plugin-tech-radar": "^0.6.13",
+    "@backstage/plugin-techdocs": "^1.10.0",
+    "@backstage/plugin-techdocs-module-addons-contrib": "^1.1.5",
+    "@backstage/plugin-techdocs-react": "^1.1.16",
+    "@backstage/plugin-user-settings": "^0.8.1",
+    "@backstage/theme": "^0.5.1",
     "@material-ui/core": "^4.12.2",
     "@material-ui/icons": "^4.9.1",
     "history": "^5.0.0",
@@ -49,7 +49,7 @@
     "react-use": "^17.2.4"
   },
   "devDependencies": {
-    "@backstage/test-utils": "^1.4.7",
+    "@backstage/test-utils": "^1.5.0",
     "@playwright/test": "^1.32.3",
     "@testing-library/jest-dom": "^6.0.0",
     "@testing-library/react": "^14.0.0",

diff --git a/toolshed/packages/backend/package.json b/toolshed/packages/backend/package.json
@@ -16,26 +16,26 @@
     "build-image": "docker build ../.. -f Dockerfile --tag backstage"
   },
   "dependencies": {
-    "@backstage/backend-common": "^0.20.1",
-    "@backstage/backend-tasks": "^0.5.14",
-    "@backstage/catalog-client": "^1.5.2",
-    "@backstage/catalog-model": "^1.4.3",
+    "@backstage/backend-common": "^0.21.3",
+    "@backstage/backend-tasks": "^0.5.18",
+    "@backstage/catalog-client": "^1.6.0",
+    "@backstage/catalog-model": "^1.4.4",
     "@backstage/config": "^1.1.1",
-    "@backstage/plugin-app-backend": "^0.3.57",
-    "@backstage/plugin-auth-backend": "^0.20.3",
-    "@backstage/plugin-auth-node": "^0.4.3",
-    "@backstage/plugin-catalog-backend": "^1.16.1",
-    "@backstage/plugin-catalog-backend-module-scaffolder-entity-model": "^0.1.6",
+    "@backstage/plugin-app-backend": "^0.3.61",
+    "@backstage/plugin-auth-backend": "^0.21.3",
+    "@backstage/plugin-auth-node": "^0.4.8",
+    "@backstage/plugin-catalog-backend": "^1.17.3",
+    "@backstage/plugin-catalog-backend-module-scaffolder-entity-model": "^0.1.10",
     "@backstage/plugin-permission-common": "^0.7.12",
-    "@backstage/plugin-permission-node": "^0.7.20",
-    "@backstage/plugin-proxy-backend": "^0.4.7",
-    "@backstage/plugin-scaffolder-backend": "^1.20.0",
-    "@backstage/plugin-search-backend": "^1.4.9",
-    "@backstage/plugin-search-backend-module-catalog": "^0.1.13",
-    "@backstage/plugin-search-backend-module-pg": "^0.5.18",
-    "@backstage/plugin-search-backend-module-techdocs": "^0.1.13",
-    "@backstage/plugin-search-backend-node": "^1.2.13",
-    "@backstage/plugin-techdocs-backend": "^1.9.2",
+    "@backstage/plugin-permission-node": "^0.7.24",
+    "@backstage/plugin-proxy-backend": "^0.4.11",
+    "@backstage/plugin-scaffolder-backend": "^1.21.3",
+    "@backstage/plugin-search-backend": "^1.5.3",
+    "@backstage/plugin-search-backend-module-catalog": "^0.1.17",
+    "@backstage/plugin-search-backend-module-pg": "^0.5.22",
+    "@backstage/plugin-search-backend-module-techdocs": "^0.1.17",
+    "@backstage/plugin-search-backend-node": "^1.2.17",
+    "@backstage/plugin-techdocs-backend": "^1.9.6",
     "app": "link:../app",
     "better-sqlite3": "^9.0.0",
     "dockerode": "^3.3.1",
@@ -46,7 +46,7 @@
     "winston": "^3.2.1"
   },
   "devDependencies": {
-    "@backstage/cli": "^0.25.1",
+    "@backstage/cli": "^0.25.2",
     "@types/dockerode": "^3.3.0",
     "@types/express": "^4.17.6",
     "@types/express-serve-static-core": "^4.17.5",