Proposal: buildx history command

[tonistiigi]

Description

Goals

• Provide access to data stored by BuildKit via history record (provenance, logs, traces etc.) to docker buildx users. History APIs were added to BuildKit in version v0.11
• Allow buildx users to better understand the internals of their builds
• Allow replaying the logs without running the whole build again with --progress=plain (in default TTY mode logs disappear once a build step completes successfully). The same can be used to print logs without the multiplexing switches that happen when build steps run in parallel.

Build summary (could be separate PR)

Currently, when running buildx build or buildx bake in a machine where Docker Desktop is installed, “View build details: docker-desktop://dashboard/build/…” will be printed after build is complete allowing access to Docker Desktop Build Views. This should be replaced with more generic “summary” feature.

Summary should contain:

• Duration of the build (especially important for --progress=plain that does not log this).
• Link to open build in Docker Desktop (if it is installed)
• Exported outputs. Eg. image names that were exported to docker or pushed to registry. Paths for local exports. Optionally in debug mode could even show example commands for running the image or inspecting with with buildx imagetools if it was pushed. If possible image can also be opened in Docker Desktop.
• Example command to replay the logs. docker buildx history logs
• Probably some summary of an error (eg. build step name that failed), but no need to duplicate whole error information again that is already printed.

Build completed: 21.0s (53 steps, 34 cached)
View build details: docker-desktop://dashboard/build/..
Outputs:
	Image    foobar                   (Exported to Docker)
	Image    docker.io/foo/bar:v2     (Pushed)
Replay logs: docker buildx history logs abc123def567

docker buildx history ls

Lists all the builds BuildKit history API knows about. Most recent builds are shown on top.

BUILD ID          NAME             STATUS          CREATED      DURATION           
abcd              binaries         Completed       5 min ago    1m03s        Open
abcef             binaries         Error           2 hours ago  3s           Open

Open link can be used to open build in Docker Desktop if supported.

Note that this command shows only the build for the active builder in Buildx, not like the build UI in Docker Desktop, that shows builds across all your builders.

docker buildx history logs [<id>]

Replay the logs for a specified build.

The logs would appear with the same formatting as --progress=plain , but there is no need to ever do multiplexing of build steps. Every step will only appear once in the output and print its whole log.

If <id> argument is not set, then the last build for the current builder will be shown. If there are no builds, then an error is shown.

docker buildx history inspect [<id>]

Print all details for a specified build. This includes data about duration, status, build request, outputs, build materials etc.
With some --verbose flag, we could also print the whole graph of steps, but this can be left for a future iteration.
Inspect output could end with examples of all other open/trace/logs etc commands for the same build.

Optionally, this command can be used to retrieve raw blobs from the build history, eg. the raw in-toto provenance, SBOM, image config, or opentelemetry trace. This can be done either with some use of --format flag or smth. like --type <mediatype>.

docker buildx history open [<id>]

Open specified build in Docker Desktop. This command requires Docker Desktop to be installed.

docker buildx history trace [<id>]

Open opentelemetry trace for the specified build.

In the initial implementation, this can run JaegerUI container (if one does not already exist), load the trace into it via POST request, and then open the link to the browser to view it. Optionally user could set their own URL to their JaegerUI instance with a flag. JaegerUI remains running, and users would need to call docker stop if they want to stop it.

docker buildx history rm [<id>...]

Delete a history record from the active BuildKit instance.

docker buildx history export [<id>…]

Export a specified build to a .dockerbuild archive that can be then imported to Docker Desktop (in another machine). This is the same archive currently generated by Docker’s Github Actions. The current implementation in Actions can start to just call buildx directly for this command.

docker buildx history import < file

Import a build into Docker Desktop from a .dockerbuild file created by history export command or downloaded from Docker’s Github Actions. This command requires Docker Desktop to be installed as the store for imported builds is managed by Desktop(that is across builders). When completed, Buildx will show a link for opening the imported build in Desktop.

[crazy-max]

Should we have a buildx history prune command? I was looking at current buildx prune which clear caches. Was then wondering if buildx cache du/prune would make more sense.

[tonistiigi]

Should we have a buildx history prune command? I was looking at current buildx prune which clear caches. Was then wondering if buildx cache du/prune would make more sense.

buildx history rm could maybe be expanded with filters for multiple deletions. May be confusing to have multiple commands for deletion. Or maybe it makes sense to replace rm with prune.

[tonistiigi]

Opened WIP PR in #2891

Some UX questions have arisen:

The list command lists all builds for the current builder instance. This is a good default, but I think we could also provide:

• Current directory. Filter builds that match the current working directory as build context. In the first milestone.
• List builds that have been imported to the Docker desktop in the local Docker desktop store instead of the builder instance. The second milestone with import/export commands.
• If the user has GITHUB_TOKEN defined in the environment, we could also provide quick access to the CI builds by looking up the .dockerbuild files for the Git origin URL. Optional sometimes in the future.

What would make sense for the UX of these options/filters? Only the first one would probably be in the initial version, but we should think about future-proofing for the other ones.

For the builds in Desktop's storage, how could these builds interact with other commands like inpsect/open/logs etc. ? An easy way would be to use something like.

docker --builder=_imported buildx history ls
docker --builder=_imported buildx history inspect foo
BUILDX_BUILDER=_imported docker buildx history open foo

This is the special key we currently use in Desktop Build UI, but is it too hacky to expose to CLI users?

---

This proposal doesn't currently clarify syntax for viewing raw files like attestations, traces, image configs etc. If the build is multi-platform, then the attestations are per-platform.

Is something like docker buildx history inspect --type=https://slsa.dev/provenance/v0.2 --platform=linux/arm64 foo too hacky or ok?

---

Docker containers/images support accessing objects by a prefix of the ID. BuildKit History API does not support this, so full ID needs to be passed by the user all the time(it is 26 character string). Is this OK, or do we somehow need to support prefixes as well? We could just do ls and filter IDs on the client side, but I think this is too inefficient when there are lots of builds. So if we need something like this, we would need to add some new API to BuildKit itself.

@crazy-max @colinhemmings @dvdksn

[crazy-max]

Current directory. Filter builds that match the current working directory as build context. In the first milestone.

I guess this will be based on VCS info and fallback to localstate? Maybe a final fackback on buildName if both return nothing?

• If the user has GITHUB_TOKEN defined in the environment, we could also provide quick access to the CI builds by looking up the .dockerbuild files for the Git origin URL. Optional sometimes in the future.

Why would we need to rely on GITHUB_TOKEN? Is it to check if it runs on GHA? If that's the case I think we should use GITHUB_REPOSITORY instead and check builds matching with that in VCS info?

Maybe also list both local builds and imported when filtering builds of working directory from 1).

What would make sense for the UX of these options/filters? Only the first one would probably be in the initial version, but we should think about future-proofing for the other ones.

With my previous comment I think we could merge 1) and 3) so it displays both local and imported builds matching VCS info.

For the builds in Desktop's storage, how could these builds interact with other commands like inpsect/open/logs etc. ? An easy way would be to use something like.

docker --builder=_imported buildx history ls
docker --builder=_imported buildx history inspect foo
BUILDX_BUILDER=_imported docker buildx history open foo

This is the special key we currently use in Desktop Build UI, but is it too hacky to expose to CLI users?

I guess you mean the global builder flag on buildx command, not docker one: docker buildx --builder=_imported history ls

Maybe a shorthand --desktop that would be --builder=_imported as a global flag?

This proposal doesn't currently clarify syntax for viewing raw files like attestations, traces, image configs etc. If the build is multi-platform, then the attestations are per-platform.

Is something like docker buildx history inspect --type=https://slsa.dev/provenance/v0.2 --platform=linux/arm64 foo too hacky or ok?

I was thinking maybe inspect could print the list of blobs attached to the record with digest: content-type and then inspect blob --digest sha256:... to get it?

Command will be quite huge though so maybe a new docker buildx history blob to interact with blobs for a record?

[tonistiigi]

Why would we need to rely on GITHUB_TOKEN? Is it to check if it runs on GHA? If that's the case I think we should use GITHUB_REPOSITORY instead and check builds matching with that in VCS info?

The token is needed to pull the .dockerbuild files from Github via the REST API. This is not run in Github itself so GITHUB_REPOSITORY is not defined. But we can get the repo address by looking for the origin URL in .git.

Maybe also list both local builds and imported when filtering builds of working directory from 1).
With my previous comment I think we could merge 1) and 3) so it displays both local and imported builds matching VCS info.

I don't think it is a good idea to mix builds from specific builder with the imported builds in desktop storage. They look very different concepts and also there is some overhead of loading both.

I do get that while "instance builds", "desktop builds", "ci builds" are all "build sources", the "current directory" case is more like a filter, that could maybe be applied to all sources, and maybe should be one case of more generic filtering mechanism (eg. another filter could be by a Dockerfile name). Still question about the flags for these.

Maybe a shorthand --desktop that would be --builder=_imported as a global flag?

Possibly indeed. And --ci ? Or would --local/--imported make more sense for --desktop?

I was thinking maybe inspect could print the list of blobs attached to the record with digest: content-type and then inspect blob --digest sha256:... to get it?

This would mean you always need to run a command to copy digest if you wish to, for example, download a provenance/trace/sbom/error for a build. Not ideal I think.

Command will be quite huge though so maybe a new docker buildx history blob to interact with blobs for a record?

I'm ok with the subcommand but UX question is still how to set what type and platform of blob is needed. I don't want to hardcode the names as well if we should add more; that's why I thought maybe the media type is the ok solution.

[dvdksn]

Maybe a shorthand --desktop that would be --builder=_imported as a global flag?

Possibly indeed. And --ci ? Or would --local/--imported make more sense for --desktop?

--desktop seems like a good abstraction. For the other properties, maybe a generic --filter option?

• --desktop and --builder to define the store and builder
• --filter to within that scope

E.g.

docker buildx history --builder my-builder ls
docker buildx history --desktop ls
docker buildx history --desktop --filter src=imported ls

And, not for the first iteration, but, I guess we could imagine more ways to filter the ls view. E.g.,

docker buildx history \
    --builder cloud-docker-default \
    --filter context=https://github.com/foo/bar.git \
    --filter duration=>1m \
    --filter created=2025-01-01:2025-01-09 \
    --filter status=failed \
    --filter platform=linux/amd64 \
    --filter tag=foo/bar:v1 \
    ls

This proposal doesn't currently clarify syntax for viewing raw files like attestations, traces, image configs etc. If the build is multi-platform, then the attestations are per-platform.

Is something like docker buildx history inspect --type=https://slsa.dev/provenance/v0.2 --platform=linux/arm64 foo too hacky or ok?

How about reusing the --provenance and --sbom shorthands? Maybe not as flexible as the above but easier to use.

[crazy-max]

The token is needed to pull the .dockerbuild files from Github via the REST API. This is not run in Github itself so GITHUB_REPOSITORY is not defined.

Ah ok this is to download the artifact through GitHub API, looks good then!

But we can get the repo address by looking for the origin URL in .git.

Yes good point

Possibly indeed. And --ci ? Or would --local/--imported make more sense for --desktop?

--desktop seems like a good abstraction. For the other properties, maybe a generic --filter option?

* `--desktop` and `--builder` to define the store and builder

The local store was strictly linked to the Docker Desktop case but with the history command now being able to import builds and inspect them I agree that --desktop is not that generic enough. I think --imported is good then. --local could be confused with local docker engine.

This would mean you always need to run a command to copy digest if you wish to, for example, download a provenance/trace/sbom/error for a build. Not ideal I think.

You're right, it feels better to use the content type as input but still think we should have a way to print attached blobs and display <content-type>: <digest> or just <content-type> through inspect.

How about reusing the --provenance and --sbom shorthands? Maybe not as flexible as the above but easier to use.

Yeah they could be shorthand for --type=.