move to atomfs with fixed umount, update golang to 1.22 (#654)

* feat: move to atomfs v1.1.3 and golang 1.22 atomfs 1.1.3 has a new API for mounting that changes it to store metadata about the mounted molecules in a known location. It also cleans up the unmount code path and added a lot of tests to atomfs. This moves to the new atomfs version and removes an obsolete mount option. atomfs also has a min golang version of 1.22, which we adopt for stacker here too because 1.21 is EOL at this time. This requires a go mod bump and tweaks to a few github CI yamls. note the build.yaml stackerfile now exports GOTOOLCHAIN=auto, which is the default set in $GOROOT/go.env if you install go via the tgz at go.dev/dl, but apparently the alpine package doesn't do that. We want auto, in order to have go respect the `toolchain` directive in go.mod. This does download a new go toolchain during the build, though, so one could argue that the right thing here would be to just update the alpine package, but we can do that at a later time. It also fixes a typo in atomfs.bats - this test case is also now run in the atomfs repo itself and the typos were fixed there, but we leave it here for extra coverage. Signed-off-by: Michael McCracken <[email protected]> * test: add shellcheck to bats tests, fix lint A previous test bug would have been caught by linting the bats test code, so let's do that now. That error was using an undefined variable in a test. ($last_layer_hash in atomfs.bats) It had been defined in a different test in the same file, but was undefined in the test in question. Unfortunately because of the way bats does bash, shellcheck only identifies that as an INFO level possible error SC2031. So this adds a new `lintbats` target that first checks for JUST SC2031, then checks separately for only error level issues. This commit also fixes a few error level issues in other files: - asterisk and whiteout : use glob instead of looping over ls. /shrug - bom: [ -n "${ZOT_HOST}:${ZOT_PORT}" ] will always be true because of the colon in there, so this commit eliminates the colon. - config.bats had a complaint about the indentation of the EOF because it's in yaml in another here doc? OK then. Also add the shellcheck package to the build deps. Signed-off-by: Michael McCracken <[email protected]> * test: remove dep on zothub.io in squashfs test this is testing import of squash images then building tar on top, we can test that without also testing zothub.io's nine fives. Signed-off-by: Michael McCracken <[email protected]> * chore: build umoci directly for tests Instead of using `go get` which now does nothing after moving to 1.22, or `go install` which fails in github CI, let's just clone and build umoci ourselves. Signed-off-by: Michael McCracken <[email protected]> * test: avoid dockerhub dependency two tests had bare image names, causing problems with dockerhub rate limiting. move instead to AWS also clarifies a skip message which was a little confusing because most of the test is actually completed. Signed-off-by: Michael McCracken <[email protected]> * chore: bump x/net to v0.33.0 for CVE-2024-45337 also bumps the golang toolchain to latest current toolchain as of right... now Signed-off-by: Michael McCracken <[email protected]> --------- Signed-off-by: Michael McCracken <[email protected]>
project-stacker · Jan 16, 2025 · f6d1a69 · f6d1a69
1 parent ad9a694
commit f6d1a69
Show file tree

Hide file tree

Showing 18 changed files with 123 additions and 93 deletions.
diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml
@@ -9,7 +9,7 @@ on:
         type: string
         description: 'Stringified JSON object listing go versions'
         default: >-
-          ["1.21.x"]
+          ["1.22.x", "1.23.x"]
       privilege-level:
         required: false
         type: string

diff --git a/.github/workflows/cloc.yml b/.github/workflows/cloc.yml
@@ -20,7 +20,7 @@ jobs:
       - name: Install go
         uses: actions/setup-go@v3
         with:
-          go-version: 1.21.x
+          go-version: 1.22.x
       - name: Check out source code
         uses: actions/checkout@v3
       - name: Install dependencies

diff --git a/.github/workflows/codeql-analysis.yml b/.github/workflows/codeql-analysis.yml
@@ -48,7 +48,7 @@ jobs:
     - name: Install go
       uses: actions/setup-go@v3
       with:
-        go-version: 1.21.x
+        go-version: 1.22.x
 
     # Initializes the CodeQL tools for scanning.
     - name: Initialize CodeQL

diff --git a/.github/workflows/coverage.yaml b/.github/workflows/coverage.yaml
@@ -9,7 +9,7 @@ on:
         type: string
         description: 'Stringified JSON object listing go versions'
         default: >-
-          ["1.21.x"]
+          ["1.22.x", "1.23.x"]
       privilege-level:
         required: false
         type: string

diff --git a/.github/workflows/release.yaml b/.github/workflows/release.yaml
@@ -23,7 +23,7 @@ jobs:
       # note >-, args needs to be strings to be used as inputs
       # for the reusable build.yaml workflow
       go-version: >-
-        ["1.21.x"]
+        ["1.22.x"]
       privilege-level: >-
         ["priv"]
       build-id: "${{needs.build-id.outputs.build-id}}"

diff --git a/Makefile b/Makefile
@@ -48,6 +48,8 @@ BATS_VERSION := v1.10.0
 # OCI registry
 ZOT := $(TOOLS_D)/bin/zot
 ZOT_VERSION := v2.1.0
+UMOCI := $(TOOLS_D)/bin/umoci
+UMOCI_VERSION := main
 
 export PATH := $(TOOLS_D)/bin:$(PATH)
 
@@ -122,7 +124,7 @@ go-test:
 	go tool cover -html coverage.txt  -o $(HACK_D)/coverage.html
 
 .PHONY: download-tools
-download-tools: $(GOLANGCI_LINT) $(REGCLIENT) $(ZOT) $(BATS)
+download-tools: $(GOLANGCI_LINT) $(REGCLIENT) $(ZOT) $(BATS) $(UMOCI)
 
 $(GOLANGCI_LINT):
 	@mkdir -p $(dir $@)
@@ -160,6 +162,12 @@ $(BATS):
 	cd bats-core; ./install.sh $(TOOLS_D); cd ..; \
 	rm -rf bats-core
 
+$(UMOCI):
+	mkdir -p ${GOPATH}/src/github.com/opencontainers/
+	git clone https://github.com/opencontainers/umoci.git ${GOPATH}/src/github.com/opencontainers/umoci
+	cd ${GOPATH}/src/github.com/opencontainers/umoci ; git reset --hard ${UMOCI_VERSION} ; make umoci ; mv umoci $(UMOCI)
+	$(UMOCI) --version
+
 TEST?=$(patsubst test/%.bats,%,$(wildcard test/*.bats))
 PRIVILEGE_LEVEL?=
 
@@ -169,7 +177,7 @@ PRIVILEGE_LEVEL?=
 check: lint test go-test
 
 .PHONY: test
-test: stacker download-tools
+test: stacker download-tools lintbats
 	sudo -E PATH="$(PATH)" \
 		STACKER_BUILD_ALPINE_IMAGE=$(STACKER_BUILD_ALPINE_IMAGE) \
 		STACKER_BUILD_BUSYBOX_IMAGE=$(STACKER_BUILD_BUSYBOX_IMAGE) \
@@ -179,6 +187,13 @@ test: stacker download-tools
 		$(shell [ -z $(PRIVILEGE_LEVEL) ] || echo --privilege-level=$(PRIVILEGE_LEVEL)) \
 		$(patsubst %,test/%.bats,$(TEST))
 
+.PHONY: lintbats
+lintbats:
+	# check only SC2031 which finds undefined variables in bats tests but is only an INFO
+	shellcheck -i SC2031 $(patsubst %,test/%.bats,$(TEST))
+	# check all error level issues
+	shellcheck -S error  $(patsubst %,test/%.bats,$(TEST))
+
 .PHONY: check-cov
 check-cov: lint test-cov
 

diff --git a/build.yaml b/build.yaml
@@ -90,6 +90,11 @@ build:
     export LXC_VERSION=$(git -C /lxc rev-parse HEAD)
     export VERSION_FULL=${{VERSION_FULL}}
 
+    # apk go pkg doesn't seem to have the standard go.env which would set GOTOOLCHAIN=auto.
+    # therefore it defaults to 'local', which ignores the `toolchain` line in go.mod
+    # let's re-set it here so we get the right toolchain version as specified in go.mod
+    export GOTOOLCHAIN=auto
+
     cd /stacker-tree
     make BUILD_D=/build show-info
     make BUILD_D=/build -C cmd/stacker/lxc-wrapper clean

diff --git a/cmd/stacker/internal_go.go b/cmd/stacker/internal_go.go
@@ -3,7 +3,6 @@ package main
 import (
 	"fmt"
 	"os"
-	"path"
 	"runtime"
 	"strings"
 
@@ -177,14 +176,8 @@ func doAtomfsMount(ctx *cli.Context) error {
 	tag := ctx.Args().Get(0)
 	mountpoint := ctx.Args().Get(1)
 
-	wd, err := os.Getwd()
-	if err != nil {
-		return errors.WithStack(err)
-	}
-
 	opts := atomfs.MountOCIOpts{
 		OCIDir:                 config.OCIDir,
-		MetadataPath:           path.Join(wd, "atomfs-metadata"),
 		Tag:                    tag,
 		Target:                 mountpoint,
 		AllowMissingVerityData: true,

diff --git a/go.mod b/go.mod
@@ -1,8 +1,8 @@
 module stackerbuild.io/stacker
 
-go 1.21.0
+go 1.22
 
-toolchain go1.21.6
+toolchain go1.22.10
 
 require (
 	github.com/anmitsu/go-shlex v0.0.0-20200514113438-38f4b401e2be
@@ -25,13 +25,13 @@ require (
 	github.com/pkg/xattr v0.4.9
 	github.com/sirupsen/logrus v1.9.3
 	github.com/smartystreets/goconvey v1.7.2
-	github.com/stretchr/testify v1.8.4
+	github.com/stretchr/testify v1.9.0
 	github.com/twmb/algoimpl v0.0.0-20170717182524-076353e90b94
 	github.com/udhos/equalfile v0.3.0
 	github.com/urfave/cli/v2 v2.25.0
 	github.com/vbatts/go-mtree v0.5.3
-	golang.org/x/sys v0.16.0
-	golang.org/x/term v0.16.0
+	golang.org/x/sys v0.28.0
+	golang.org/x/term v0.27.0
 	gopkg.in/yaml.v2 v2.4.0
 	sigs.k8s.io/bom v0.6.0
 	sigs.k8s.io/yaml v1.3.0
@@ -42,13 +42,14 @@ require (
 	github.com/Masterminds/semver/v3 v3.2.1 // indirect
 	github.com/freddierice/go-losetup v0.0.0-20220711213114-2a14873012db // indirect
 	github.com/martinjungblut/go-cryptsetup v0.0.0-20220520180014-fd0874fd07a6 // indirect
+	github.com/moby/sys/user v0.3.0 // indirect
 )
 
 require (
 	dario.cat/mergo v1.0.0 // indirect
 	github.com/AdaLogics/go-fuzz-headers v0.0.0-20230106234847-43070de90fa1 // indirect
 	github.com/AdamKorcz/go-118-fuzz-build v0.0.0-20221215162035-5330a85ea652 // indirect
-	github.com/BurntSushi/toml v1.2.1 // indirect
+	github.com/BurntSushi/toml v1.3.2 // indirect
 	github.com/CycloneDX/cyclonedx-go v0.7.2 // indirect
 	github.com/DataDog/zstd v1.4.8 // indirect
 	github.com/MakeNowJust/heredoc/v2 v2.0.1 // indirect
@@ -92,7 +93,7 @@ require (
 	github.com/containers/storage v1.45.3 // indirect
 	github.com/cpuguy83/go-md2man/v2 v2.0.3 // indirect
 	github.com/cyberphone/json-canonicalization v0.0.0-20220623050100-57a0ce2678a7 // indirect
-	github.com/cyphar/filepath-securejoin v0.2.4 // indirect
+	github.com/cyphar/filepath-securejoin v0.3.5 // indirect
 	github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
 	github.com/deitch/magic v0.0.0-20230404182410-1ff89d7342da // indirect
 	github.com/distribution/reference v0.5.0 // indirect
@@ -182,7 +183,7 @@ require (
 	github.com/mitchellh/mapstructure v1.5.0 // indirect
 	github.com/mitchellh/reflectwalk v1.0.2 // indirect
 	github.com/moby/locker v1.0.1 // indirect
-	github.com/moby/sys/mountinfo v0.6.2 // indirect
+	github.com/moby/sys/mountinfo v0.7.1 // indirect
 	github.com/moby/sys/sequential v0.5.0 // indirect
 	github.com/moby/sys/signal v0.7.0 // indirect
 	github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
@@ -191,8 +192,8 @@ require (
 	github.com/nwaples/rardecode v1.1.3 // indirect
 	github.com/oklog/ulid v1.3.1 // indirect
 	github.com/olekukonko/tablewriter v0.0.5 // indirect
-	github.com/opencontainers/runc v1.1.8 // indirect
-	github.com/opencontainers/runtime-spec v1.1.0-rc.1 // indirect
+	github.com/opencontainers/runc v1.2.3 // indirect
+	github.com/opencontainers/runtime-spec v1.2.0 // indirect
 	github.com/opencontainers/selinux v1.11.0 // indirect
 	github.com/package-url/packageurl-go v0.1.2 // indirect
 	github.com/pborman/indent v1.2.1 // indirect
@@ -241,7 +242,7 @@ require (
 	github.com/theupdateframework/go-tuf v0.5.2 // indirect
 	github.com/titanous/rocacheck v0.0.0-20171023193734-afe73141d399 // indirect
 	github.com/ulikunitz/xz v0.5.11 // indirect
-	github.com/urfave/cli v1.22.12 // indirect
+	github.com/urfave/cli v1.22.14 // indirect
 	github.com/vbatts/tar-split v0.11.3 // indirect
 	github.com/vbauerster/mpb/v7 v7.5.3 // indirect
 	github.com/vifraa/gopom v1.0.0 // indirect
@@ -260,24 +261,24 @@ require (
 	go.opentelemetry.io/otel/metric v1.21.0 // indirect
 	go.opentelemetry.io/otel/trace v1.21.0 // indirect
 	go.uber.org/multierr v1.11.0 // indirect
-	golang.org/x/crypto v0.18.0 // indirect
+	golang.org/x/crypto v0.31.0 // indirect
 	golang.org/x/exp v0.0.0-20231206192017-f3f8817b8deb // indirect
-	golang.org/x/mod v0.14.0 // indirect
-	golang.org/x/net v0.20.0 // indirect
-	golang.org/x/sync v0.6.0 // indirect
-	golang.org/x/text v0.14.0 // indirect
-	golang.org/x/tools v0.17.0 // indirect
+	golang.org/x/mod v0.17.0 // indirect
+	golang.org/x/net v0.33.0 // indirect
+	golang.org/x/sync v0.10.0 // indirect
+	golang.org/x/text v0.21.0 // indirect
+	golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d // indirect
 	golang.org/x/tools/go/vcs v0.1.0-deprecated // indirect
 	golang.org/x/xerrors v0.0.0-20231012003039-104605ab7028 // indirect
 	google.golang.org/genproto v0.0.0-20231211222908-989df2bf70f3 // indirect
 	google.golang.org/genproto/googleapis/rpc v0.0.0-20231212172506-995d672761c0 // indirect
 	google.golang.org/grpc v1.60.0 // indirect
-	google.golang.org/protobuf v1.32.0 // indirect
+	google.golang.org/protobuf v1.36.1 // indirect
 	gopkg.in/ini.v1 v1.67.0 // indirect
 	gopkg.in/square/go-jose.v2 v2.6.0 // indirect
 	gopkg.in/warnings.v0 v0.1.2 // indirect
 	gopkg.in/yaml.v3 v3.0.1 // indirect
-	machinerun.io/atomfs v1.1.1
+	machinerun.io/atomfs v1.1.3
 	modernc.org/libc v1.37.6 // indirect
 	modernc.org/mathutil v1.6.0 // indirect
 	modernc.org/memory v1.7.2 // indirect