Runbook for building the Baseprinter container images

[castedo]

@khinsen, in the last paragraph of this toot you hit on three points.

For the first and third points, installation complexity and huge resource consumption, I have recorded into GitLab issues https://gitlab.com/perm.pub/baseprinter/-/issues/4 and https://gitlab.com/perm.pub/baseprinter/-/issues/5.

There is of course still an underlying trade-off between container image space vs local install problems. Not much can be done about that. Pick your poison. At this early stage of development I recommend the container path of the two. I think most users value their time more than their disk space. But other users are free to risk the local install challenges to save disk space.

This GitHub discussion is regarding the second point of "no auditability". I'm not totally sure what you meant so please feel free to chime in here. I can guess that this is partly due to something I don't really like either: it's not clear that the OCI container image in the documentation, namely docker.io/castedo/baseprinter is built by running this Bash script from a Linux distro with podman/buildah installed.

One idea is to add a "Build the Baseprinter Container Image" to the How-To Guides for Baseprinter.

But this seems off. How-To Guides are really geared towards what one expects users to want to do. Whereas knowing how to build the Baseprinter Container Image is more for auditability. It seems to me that most users DO NOT want to do this and would much rather have me make it for them. But they want to know HOW I do it for them.

So rather than a "How-To Guide" I'm thinking of calling it a runbook for building the image that goes into a separate documentation area of an "Open Infrastructure Book" or "Administration" or some other name. Or maybe just another area of the Baseprinter doc pages. Baseprinter is just an authoring tool. It's not the only way to generate a Baseprint document snapshot.

Any clarification and/or thoughts appreciated.

[khinsen]

My reference to auditability was meant exactly as you understood it: with a container image, users have no means to know what's inside, nor how the image was constructed. With attacks on package managers becoming more common, that's a risk factor.

Providing build instructions for the container is a big step forward. Few people will actually build and check, but it becomes possible to do so. Perhaps one day we will see service provider certifying container image builds.

[castedo]

Regarding certifying container image builds, RedHat does certification of some container images, but there aren't many of them. Not exactly the same, but more relevant to this situation is container signing:

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/building_running_and_managing_containers/assembly_signing-container-images_building-running-and-managing-containers

So in theory, a small group of trusted individuals could keep GPG signing keys and use them to sign container images built in a CI/CD workflow, and publish the public keys on a trusted website.

[castedo]

@khinsen the following two statements you made are largely incorrect:

users have no means to know what's inside,

Distro packages and container images expose their data in different ways. But users can definitely see what's inside a container image, altough not as easily. In some ways container images give users MORE insight because distro packages can run scripts and it will be extremely difficult for a user to figure out what those install scripts actually did. In contrast, all the files in image layers are there to be audited. An OCI image has multiple layers with separate details on what files were added and deleted.

podman inspect docker.io/castedo/baseprinter | jq .[].RootFS.Layers

How to inspect all the added/deleted files in those layers I'll skip.

nor how the image was constructed

I believe all the major container image build tools include a history of what installation commands were run to build the image. For instance, the following will print the commands recorded from the building of the Baseprinter container image:

podman inspect docker.io/castedo/baseprinter | jq .[].History

However, this depends on how the container image is built. There isn't a fundamental reason this history must be present and accurate in the image. So it is true that a malicious container image builder could include a history that is a fake out.

[khinsen]

When I wrote that "users have no means to know what's inside", I was referring to generic containers. Additional conventions can make containers inspectable, and I hope we will see more of that in the future. For now, in scientific computing, most containers I see are created using Docker or Singularity by building up a system inside the container and saving the resulting image.

The typical question a user might have (again, in scientific computing) is something like "there was an important bugfix in libfoo 3.3.2, does this container contain an older version?" If libfoo is a shared library, it is possible to find the answer but only with some serious effort and knowledge of containerization tools. If libfoo is statically linked to a binary, or a header-only C++ library, there is no way to find the answer. With distro packages, it's a lot easier, though still too diffiicult for my taste.

[castedo]

Clarification on which inspectable data is convention and which is not:

If "what's inside" means "what files are inside" then the .Layers part is not convention based. My understanding of OCI container images is all file adds, changes, and deletions in the layers are inspectable regardless of convention. So "what's inside", as in "what files are inside" is something that users can know even in the absense of a convention.

For the sense of "what installation is inside" where "installation" means the process of installing the files, then the .History part of inspect is fundamentally convention based.

it is possible to find the answer but only with some serious effort and knowledge of containerization tools.

I agree that it requires knowledge of containerization tools. I disagree that determining whether a bugfix is in libfoo requires serious effort if libfoo is a shared library in a container image. Partly depends on how the container image is built. If it's built in a typical way then it's as easy as making this determination on a local non-container system.

[khinsen]

Overall I agree, it's just that in the use cases that I see, file inspection is not sufficient. I already mentioned header-only C++ libraries (quite popular in efficiency-oriented code), but there's also Python and R code (where the packaging technique defines if the version information is available in the file system). Plus the worst case (rare but happens) of a compiler bug, where you need to know the compiler version that was used to compile the binaries in the container. Even most distro packages cannot provide that information (though Nix and Guix do).

[castedo]

My currently medium-term plan is to do automated building of the Baseprinter container image over on gitlab.

I'm currently setting up automated OCI container image building on gitlab now for a separate but related project.

[castedo]

GitLab issue tracking this at https://gitlab.com/perm.pub/baseprinter/-/issues/7