open /var/lib/containers/storage/overlay/.has-mount-program: no such file or directory

[dragsu]

I am currently working on a GitLab pipeline to build container images. My GitLab runner is running in a docker image in privilege mode and it is using buildha to build the below Docker content.

FROM registry.access.redhat.com/ubi8/ubi:8.6-855

ENV PATH /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
CMD ["/bin/bash"]

Extract of config.toml

 [runners.docker]
    tls_verify = false
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/var/lib/containers:/var/lib/containers","/cache"]
    shm_size = 0

In my .gitlab-ci.yml I tried using the below command,

buildah --storage-opt mount_program=/usr/bin/fuse-overlayfs --storage-driver overlay2 bud  -t localhost:test:latest .

and I'm getting the below output with an error

STEP 1/5: FROM registry.access.redhat.com/ubi8/ubi:8.6-855
Trying to pull registry.access.redhat.com/ubi8/ubi:8.6-855...
Getting image source signatures
Checking if image destination supports signatures
Copying blob sha256:028bdc977650c08fcf7a2bb4a7abefaead71ff8a84a55ed5940b6dbc7e466045
Copying blob sha256:0c673eb68f88b60abc0cba5ef8ddb9c256eaf627bfd49eb7e09a2369bb2e5db0
Copying blob sha256:0c673eb68f88b60abc0cba5ef8ddb9c256eaf627bfd49eb7e09a2369bb2e5db0
Copying blob sha256:028bdc977650c08fcf7a2bb4a7abefaead71ff8a84a55ed5940b6dbc7e466045
Copying config sha256:2fd9e1478809c0c53820517b998e1b2a044207515e5d61eccb6295276bbbaffc
Writing manifest to image destination
Storing signatures
error mounting new container: error mounting build container "373bbfbe192d28bac1ff1cada693eec7c7e9de86db3db07b16e31a79a778ef69": open /var/lib/containers/storage/overlay/.has-mount-program: no such file or directory

When I list /var/lib/containers I only see,

total 0
drwxr-xr-x. 3 root root 22 Jul 11 04:23 .
drwxr-xr-x. 1 root root 58 Jul 11 04:23 ..
drwxr-xr-x. 2 root root  6 Apr 19 10:19 sigsto

Then I added the below command before running buildha bud in .gitlab-ci.yml,

mkdir -p "/var/lib/containers/storage/overlay/" && touch "/var/lib/containers/storage/overlay/.has-mount-program"

and managed to move a bit further and got the below error,

error committing container for step {Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin container=oci] Command:cmd Args:[/bin/bash] Flags:[] Attrs:map[json:true] Message:CMD /bin/bash Original:CMD ["/bin/bash"]}: error copying layers and metadata for container "0c77d5e9339ea2b6dbc82761b7c8b396b2d41c90c36d579b1cdba7fc5d954bba": initializing source containers-storage:ubi-working-container: error extracting layer "8fb7c2252a6873d080ce8a29dc1d4bb1b431ae986a2c19fb0dca0476925f6b8c": stat /var/lib/containers/storage/overlay/8fb7c2252a6873d080ce8a29dc1d4bb1b431ae986a2c19fb0dca0476925f6b8c: no such file or directory

If I run with --storage-driver vfs I can successfully build the image. @giuseppe can you please shed some light?

[giuseppe]

what version of buildah are you using?

[dragsu]

• buildah version 1.26.2(image-spec 1.0.2-dev, runtie-spec 1.0.2-dev)
• RHEL Version 8.6