[Bug]: Certificates vanish in legacy docker engine builder

[DanielHabenicht]

Steps to Reproduce

FROM smallstep/step-ca AS certs
ARG RESOURCE_REAPER_SESSION_ID="00000000-0000-0000-0000-000000000000"
LABEL "org.testcontainers.resource-reaper-session"=$RESOURCE_REAPER_SESSION_ID

RUN pwd && step certificate create "Smallstep Root CA" "/home/step/cacert.pem" "/home/step/cakey.pem" \
    --no-password --insecure \
    --profile root-ca \
    --not-before "2021-01-01T00:00:00+00:00" \
    --not-after "2031-01-01T00:00:00+00:00" \
    --san "example.com" \
    --san "mail.example.com" \
    --kty RSA --size 2048


FROM docker.io/mailserver/docker-mailserver

ARG RESOURCE_REAPER_SESSION_ID="00000000-0000-0000-0000-000000000000"
LABEL "org.testcontainers.resource-reaper-session"=$RESOURCE_REAPER_SESSION_ID

ENV OVERRIDE_HOSTNAME=mail.example.com

ENV SSL_TYPE=manual
ENV SSL_CERT_PATH=/tmp/dms/custom-certs/cacert.pem
ENV SSL_KEY_PATH=/tmp/dms/custom-certs/cakey.pem
COPY --from=certs /home/step/* /tmp/dms/custom-certs/

$env:DOCKER_BUILDKIT = 0
docker build . --progress plain --no-cache
Sending build context to Docker daemon  27.14kB
Step 1/12 : FROM smallstep/step-ca AS certs
 ---> 624886655681
Step 2/12 : ARG RESOURCE_REAPER_SESSION_ID="00000000-0000-0000-0000-000000000000"
 ---> Using cache
 ---> 5171ce28d72f
Step 3/12 : LABEL "org.testcontainers.resource-reaper-session"=$RESOURCE_REAPER_SESSION_ID
 ---> Using cache
 ---> 5ec7b70062f1
Step 4/12 : RUN pwd && step certificate create "Smallstep Root CA" "/home/step/cacert.pem" "/home/step/cakey.pem"     --no-password --insecure     --profile root-ca     --not-before "2021-01-01T00:00:00+00:00"     --not-after "2031-01-01T00:00:00+00:00"     --san "example.com"     --san "mail.example.com"     --kty RSA --size 2048
 ---> Running in 0d8b2380a61d
/home/step
Your certificate has been saved in /home/step/cacert.pem.
Your private key has been saved in /home/step/cakey.pem.
Removing intermediate container 0d8b2380a61d
 ---> 9c5acd20555e
Step 5/12 : FROM docker.io/mailserver/docker-mailserver
 ---> 73d5c52ca1c1
Step 6/12 : ARG RESOURCE_REAPER_SESSION_ID="00000000-0000-0000-0000-000000000000"
 ---> Using cache
 ---> 5882cc031902
Step 7/12 : LABEL "org.testcontainers.resource-reaper-session"=$RESOURCE_REAPER_SESSION_ID
 ---> Using cache
 ---> 790dc8489200
Step 8/12 : ENV OVERRIDE_HOSTNAME=mail.example.com
 ---> Using cache
 ---> 1f2f0613094c
Step 9/12 : ENV SSL_TYPE=manual
 ---> Using cache
 ---> 30da37287448
Step 10/12 : ENV SSL_CERT_PATH=/tmp/dms/custom-certs/cacert.pem
 ---> Using cache
 ---> a72b524fb224
Step 11/12 : ENV SSL_KEY_PATH=/tmp/dms/custom-certs/cakey.pem
 ---> Using cache
 ---> b70ec7ff1412
Step 12/12 : COPY --from=certs /home/step/* /tmp/dms/custom-certs/
COPY failed: no source files were specified

Your Environment

• OS - Windows
• step-ca Version - latest ---> 624886655681

Expected Behavior

Files should still be there

Actual Behavior

Files can't be copied, as they are not there.

Additional Context

I am initially coming from here: testcontainers/testcontainers-dotnet#914

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[DanielHabenicht]

Workaround, for now, is to write the files to the /tmp directory.
This comment mentioned some volumes mounted while building the base image, which might be the reason:
https://forums.docker.com/t/resolved-files-missing-after-dockerfile-run-downloads-them/4827

[jdoss]

Hey @DanielHabenicht How are you running the container? When running step-ca inside a container, you would normally want to mount /home/step/ inside the container to something outside on the host or in a Docker Volume so the files generated persist.

[DanielHabenicht]

I am only running docker build . --progress plain --no-cache as I am building the image for use in our test environment.
The step container is used as a stage for the build of the image.

[jdoss]

If you want to persist certificates you need to mount a volume to do so. Otherwise the stage in the build will not persist any data at the end of the build.

[DanielHabenicht]

But I just want them to be included in my test image, as stated in my initial message.

I can understand if you took measures to remove certificates from the home directory for improved security. If so than it should documented.

Also this would raise the question why it is working with the newer docker build engine (not BUILDKIT)?

[tashian]

@DanielHabenicht you shouldn't need the step-ca image for this, since you're not running an online CA server here.
Try replacing smallstep/step-ca with smallstep/step-cli, and see if that helps.

[DanielHabenicht]

Yes that helps:

FROM smallstep/step-cli AS certs
ARG RESOURCE_REAPER_SESSION_ID="00000000-0000-0000-0000-000000000000"
LABEL "org.testcontainers.resource-reaper-session"=$RESOURCE_REAPER_SESSION_ID

RUN pwd && step certificate create "Smallstep Root CA" "/home/step/cacert.pem" "/home/step/cakey.pem" \
    --no-password --insecure \
    --profile root-ca \
    --not-before "2021-01-01T00:00:00+00:00" \
    --not-after "2031-01-01T00:00:00+00:00" \
    --san "example.com" \
    --san "mail.example.com" \
    --kty RSA --size 2048


FROM docker.io/mailserver/docker-mailserver

ARG RESOURCE_REAPER_SESSION_ID="00000000-0000-0000-0000-000000000000"
LABEL "org.testcontainers.resource-reaper-session"=$RESOURCE_REAPER_SESSION_ID

ENV OVERRIDE_HOSTNAME=mail.example.com

ENV SSL_TYPE=manual
ENV SSL_CERT_PATH=/tmp/dms/custom-certs/cacert.pem
ENV SSL_KEY_PATH=/tmp/dms/custom-certs/cakey.pem
COPY --from=certs /home/step/* /tmp/dms/custom-certs/

[jdoss]

Hey @DanielHabenicht it looks like @tashian was able to help you sort out your issues here. I am going to close this ticket out.