feat: allow setting platform for scratch image

[dsseng]

BuildKit refers to platform set in the current LLB node for determining whether or not executing commands should use QEMU. This makes some build flows working fine without BuildKit to fail with it. This is of particular importance for cross-compilation cases.

Make scratch image a no exception and allow platform to be set, therefore determining the architecture of binaries planned to be ran (e.g. if they are COPYed from contexts)

Example use case: siderolabs/stagex@eec19a8 - here we build for arm64 and amd64 targets, but use linux/386 compiler binary, as these are cross-compilation stages. So if scratch is used, then buildkit would use target platform and assume QEMU must be called to run /bin/sh, which crashed the build, since BuildKit called qemu-aarch64, which errored on i386 ELF file supplied. But with this patch I'm able to declare platform build happens in, even though I only add images later on (and for example stage3 pulls in multiplatform stage2, so in this case FROM stage2 directly won't work either)

Example build (listing logs with linux/386->arm64) https://github.com/siderolabs/stagex/actions/runs/12834542603/job/35792080087

[tonistiigi]

I'm not sure I still understand what case this is. If you want to build 386 image then you should pass --platform linux/386 as a build flag. The behavior where you request arm but the Dockerfile still provides you with 386 image instead is not desirable (such cases may be possible for intermediate cross-compilation stages but not something that we should promote for final stages). If you want to make sure that your Dockerfile can only build for --platform 386 or does something else when building other platforms then that can be done with TARGETARCH.

FROM scratch AS build-only-386
...


FROM build-only-${TARGETARCH}

We have some warnings already when we detect that the build result doesn't match what user asked in the build request. Atm we detect misbehaving frontends, but in the future, we could check binaries etc.

[dsseng]

I want to use it for a case when I want to cross-compile, so my target is arm64, but compiler has to run on i386. And the compiler itself is to be copied from context, therefore that won't set LLB to cross-compilation.

PTAL at stage3 from my linked commit as an example. It pulls in stage2, a cross-compiler which has i386 binaries and aarch64 output

[matejvasek]

Possibly related #5691

[matejvasek]

@dsseng I think I have the same problem as you, see the discussion #5691
Am I right?

[dsseng]

@dsseng I think I have the same problem as you, see the discussion #5691 Am I right?

Yes, that's the same. You could try also building from this PR's branch and specifying a syntax using a comment. Should help accomplish the task

[tonistiigi]

Looking at #5691 that's more of an issue with the emulators fallback. If emulators would be installed in kernel via setup-qemu-action for example then I think it would work. It would still run without emulator, but emulator missing in kernel confuses it and forces to look for stage platform instead of reading platform from binary. FROM --platform scratch might be a workaround for this case though.

[matejvasek]

I observe to issue only on our GH Action. Locally it works for me with both moby and podman. I think on my machine kernel level binfmt is in use so respective qemu-*-static is used. Any idea how to fix it up in GH Action?

[dsseng]

Yeah, but FROM platform also allows buildkit to properly detect that it's a cross build. This should be more future-proof

[matejvasek]

Any idea how to fix it up in GH Action?

now see that setup-qemu-action is for GH Actions

[dsseng]

I observe to issue only on our GH Action. Locally it works for me with both moby and podman. I think on my machine kernel level binfmt is in use so respective qemu-*-static is used. Any idea how to fix it up in GH Action?

For some scenarios you cannot just add binfmt to your environment. And honestly it seems more like a workaround rather than letting buildkit know what you're going to run so that it handles the build properly

[panekj]

I'm not sure I still understand what case this is.

From my understanding, this fixes the case where BUILDPLATFORM is not understood because it's not inherited
While this isn't probably the best way to cross-compile, it allows to drop other stuff from the image, like environment variables.

FROM scratch AS build-386

COPY --from=context-386 . /

ARG TARGETARCH # arm64

[matejvasek]

@tonistiigi it looks like adding setup-qemu-action to my GH workflow fixes the issue for me.
But I agree with @dsseng that it preferably should work even without it.

[dsseng]

I'm not sure I still understand what case this is.

From my understanding, this fixes the case where BUILDHOST is not understood because it's not inherited
While this isn't probably the best way to cross-compile, it allows to drop other stuff from the image, like environment variables.

FROM scratch AS build-386

COPY --from=context-386 . /

ARG TARGETARCH # arm64

Build contexts are selected based on the current step arch, so this won't work unless the current step is primed for needed arch either by FROM with a platform, or my proposed solution

[tonistiigi]

I think we can bring this in, but we need an integration test.

[dsseng]

Okay, will work on this

I think we can bring this in, but we need an integration test.

[dsseng]

I'm not so sure about correctness of this one and didn't have time to run this through. I ran unit tests though and they feel to be sufficient for the change

[dsseng]

An issue faced during practical usage: ERROR: failed to solve: no match for platform in manifest: not found. Even though my scratch layer is linux/386, the image being requested is linux/arm64, which is not present in the registry. Reference how I copy:
--build-context stagex/bootstrap-stage1=docker-image://ghcr.io/siderolabs/stagex/bootstrap-stage1, COPY --from=stagex/bootstrap-stage1 . /

So it appears this is not sufficient to solve the problem without using FROM directly and not using COPY from image-based contexts at all during cross stages.

Note: once bootstrap-stage1 image was not built with --build-arg BUILDKIT_MULTI_PLATFORM=1, this problem wasn't present, since the image was platform-neutral and was been able to be pulled via context. Ref: #5671