github: Add release workflow #4

minor-fixes · 2024-07-11T15:57:19Z

This change adds a workflow that implements the release process. The majority of the release process is done by a bash script, which uses git, bazel, and the Github CLI to:

add a tag with the release version
build release artifacts from said tag
create a release with auto-generated release notes

Tested: able to manually test some of the validation steps using SCRIPT_DEBUG:

semver check
gh cli download + verification
branch check (negative case only)
build
release creation (gh CLI runs and finds files, but release creation is unsuccessful; likely due to permissions)

Bug: linear/CUS-332

jayconrod

I'm mostly going to hold off reviewing until #3 is landed, since a lot of the same comments apply. Some high level comments though:

Avoid actions that haven't gone through a security review. These run in a very privileged position, affecting the release of software that has access to sensitive data. We should be extremely cautious here. Let's script this ourselves.
Let's try and do this with a single workflow if we can. I'd be fine with only release.yml.

minor-fixes · 2024-07-12T15:49:13Z

Compacted everything down to a single workflow that triggers a custom script - PTAL

.bazelrc

.github/workflows/release.yml

cmd/engflow_auth/BUILD

infra/release.sh

jayconrod · 2024-07-12T21:14:46Z

infra/release.sh

+
+set -euo pipefail
+
+readonly RELEASE_VERSION="$1"


Check that exactly 1 argument was passed in, report an error if not.

Check that the version follows semver.

Verify that HEAD is either on main or release/vX.Y where vX.Y. is a prefix of $RELEASE_VERSION. (It must not be possible to release from an unprotected branch).

infra/release.sh

jayconrod · 2024-07-12T21:25:30Z

infra/release.sh

+    //:release_artifacts
+
+# Create release
+gh release create \


We don't have the gh command installed in the CI environment. Can do you do this step instead with the actions/create-release or with actions/github-script?

Added some code to download + verify the GH CLI on-the-fly; added a TODO to see if we can get this action in a Docker container as a follow-on

jayconrod · 2024-07-16T20:39:33Z

infra/release.sh

+fi
+
+readonly RELEASE_VERSION="$1"
+readonly GH_CLI_URL='https://github.com/cli/cli/releases/download/v2.52.0/gh_2.52.0_linux_amd64.tar.gz'


Copy this file to the engflow-tools-public GCS bucket. We vendor binaries and installable packages there so that we still have a copy if it changes or becomes unavailable.

You can copy there with gsutil cp gh_2.52.0_linux_amd64.tar.gz gs://engflow-tools-public/github.com/cli/cli/releases/download/v2.52.0/gh_2.52.0_linux_amd64.tar.gz. Use the URL as the blob key so we know where it came from.

You can download it here by prepending storage.googleapis.com/engflow-tools-public/ to the URL.

I don't have permissions to write to engflow-tools-public :/

Are permissions controlled by terraform? Grepping through prod and engflow repos doesn't reveal any controlling terraform; I also can't read IAM permissions on the bucket to see who has access currently

I've uploaded this for now to speed things along. Ulf or Corbin can grant you access though.

jayconrod · 2024-07-16T20:45:41Z

infra/release.sh

+readonly GH_CLI_EXPECTED_SHA256='3ea6ed8b2585f406a064cecd7e1501e58f56c8e7ca764ae1f3483d1b8ed68826'
+
+# Check that supplied version string follows semver
+grep '[0-9]\+\.[0-9]\+\.[0-9]\+' <<<${RELEASE_VERSION} || {


I think the user should have to specify a version number that begins with v. Go version numbers always must start with v, so it's surprising that they're not allowed to include that here, but we add it automatically elsewhere.

Also make sure to accept prerelease versions. semver.org has a suggested regex, but you can omit the build metadata group (Go doesn't allow that in tags). You'll likely need egrep to parse that.

Seems reasonable - we're no longer editing strings in files, so we can tolerate a version string with a prepended v everywhere.

Stole the regex from semver.org - but v isn't part of semver I guess, so that's prepended to the regex.

jayconrod · 2024-07-16T20:56:26Z

infra/release.sh

+readonly EXPECTED_RELEASE_BRANCH="$(sed 's|\([0-9]\+\.[0-9]\+\)\.[0-9]\+|release/v\1|' <<<${RELEASE_VERSION})"
+git branch \
+  --contains "$(git rev-parse HEAD)" \
+  | grep "main|${EXPECTED_RELEASE_BRANCH}" \


Need to either use egrep or escape the | character in the regexp.

I'd recommend using egrep as well as sed -E so you don't have to escape the | ( ) + operators.

jayconrod · 2024-07-16T20:58:26Z

infra/release.sh

+readonly GH_CLI_EXPECTED_SHA256='3ea6ed8b2585f406a064cecd7e1501e58f56c8e7ca764ae1f3483d1b8ed68826'
+
+# Check that supplied version string follows semver
+grep '[0-9]\+\.[0-9]\+\.[0-9]\+' <<<${RELEASE_VERSION} || {


Sorry I promised not to nitpick so much, but I think the || here is difficult to read. Using an if statement would be clearer.

I'd also suggest using --quiet so the grep output isn't printed to stdout.

changed both

jayconrod · 2024-07-16T21:00:18Z

infra/release.sh

+    exit 1
+  }
+
+# Add a tag


I'm pretty sure creating a release will also create a tag, so this may prevent that last call from working. In any case, don't create the tag until everything else has succeeded, certainly not until after the build has succeeded. Go versions cannot be modified once they're published, so this has to be the last step.

Yeah, looks like the docs say that either the tag is existing when you create the release, or there's an option to create the tag with the release. GH CLI docs are a bit more nebulous - there's a --verify-tag flag that asserts the tag already exists, suggesting that tag creation is the default behavior.

Removed the tag create and push steps

jayconrod · 2024-07-17T17:32:17Z

infra/release.sh

+fi
+
+readonly RELEASE_VERSION="$1"
+readonly GH_CLI_URL='https://github.com/cli/cli/releases/download/v2.52.0/gh_2.52.0_linux_amd64.tar.gz'


I've uploaded this for now to speed things along. Ulf or Corbin can grant you access though.

jayconrod · 2024-07-17T17:35:01Z

.github/workflows/release.yml

+
+jobs:
+  release:
+    runs-on: ubuntu-latest


Let's run this on our own runners now that they're set up for the other workflows. You should be able to use the same runs-on configuration for linux / x64.

This change implements two workflows that together implement the release process. The first workflow is manually triggered, takes a human-specified semver version string, and creates an automated commit that updates the version string in the required places, and then tags the new commit. The second workflow is triggered on release semver tags, and performs the build and creates the release. Tested: no - these workflows are hard to test manually, and will likely require trial+error on Github itself. Bug: linear/CUS-332

minor-fixes requested review from jayconrod and WillEngFlow July 11, 2024 15:57

minor-fixes force-pushed the scott/release_workflow branch from 1b8fc09 to fd087e7 Compare July 11, 2024 15:57

minor-fixes force-pushed the scott/presubmit_workflow branch from ebb5c8a to 6d82f4f Compare July 11, 2024 16:01

minor-fixes force-pushed the scott/release_workflow branch from fd087e7 to dd575d8 Compare July 11, 2024 16:02

jayconrod reviewed Jul 11, 2024

View reviewed changes

minor-fixes force-pushed the scott/presubmit_workflow branch 2 times, most recently from 75b1a87 to 83613d6 Compare July 12, 2024 15:16

minor-fixes force-pushed the scott/release_workflow branch from dd575d8 to f614276 Compare July 12, 2024 15:46

minor-fixes changed the title ~~github: Add release workflows~~ github: Add release workflow Jul 12, 2024

minor-fixes requested a review from jayconrod July 12, 2024 15:48

minor-fixes force-pushed the scott/presubmit_workflow branch from 83613d6 to a25b7c0 Compare July 12, 2024 18:28

Base automatically changed from scott/presubmit_workflow to main July 12, 2024 20:44

minor-fixes force-pushed the scott/release_workflow branch from f614276 to 859e117 Compare July 12, 2024 20:45

jayconrod reviewed Jul 12, 2024

View reviewed changes

minor-fixes force-pushed the scott/release_workflow branch from 859e117 to 236f790 Compare July 16, 2024 14:36

minor-fixes requested a review from jayconrod July 16, 2024 14:37

minor-fixes force-pushed the scott/release_workflow branch from 236f790 to bc03e17 Compare July 16, 2024 14:41

jayconrod reviewed Jul 16, 2024

View reviewed changes

minor-fixes force-pushed the scott/release_workflow branch 3 times, most recently from 54ee019 to c3929aa Compare July 17, 2024 15:57

jayconrod approved these changes Jul 17, 2024

View reviewed changes

minor-fixes force-pushed the scott/release_workflow branch from c3929aa to f9e2cca Compare July 17, 2024 20:10

minor-fixes requested a review from jayconrod July 17, 2024 20:11

jayconrod approved these changes Jul 17, 2024

View reviewed changes

minor-fixes merged commit 3513080 into main Jul 17, 2024
4 checks passed

minor-fixes deleted the scott/release_workflow branch July 17, 2024 20:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

github: Add release workflow #4

github: Add release workflow #4

minor-fixes commented Jul 11, 2024 •

edited

Loading

jayconrod left a comment

minor-fixes commented Jul 12, 2024

jayconrod Jul 12, 2024

minor-fixes Jul 16, 2024

jayconrod Jul 12, 2024

minor-fixes Jul 16, 2024

jayconrod Jul 16, 2024

minor-fixes Jul 17, 2024

jayconrod Jul 17, 2024

jayconrod Jul 16, 2024

minor-fixes Jul 17, 2024

jayconrod Jul 16, 2024

minor-fixes Jul 17, 2024

jayconrod Jul 16, 2024

minor-fixes Jul 17, 2024

jayconrod Jul 16, 2024

minor-fixes Jul 17, 2024

jayconrod Jul 17, 2024

jayconrod Jul 17, 2024

minor-fixes Jul 17, 2024

minor-fixes Jul 17, 2024

github: Add release workflow #4

github: Add release workflow #4

Conversation

minor-fixes commented Jul 11, 2024 • edited Loading

jayconrod left a comment

Choose a reason for hiding this comment

minor-fixes commented Jul 12, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

minor-fixes commented Jul 11, 2024 •

edited

Loading