buildah --mount cache tests not working under podman-remote

[Luap99]

Seen in the latest buildah vendor (#25130) which added a new cache feature for mount content (containers/buildah#5691).

Two new test caes are failing with podman-remote:

[+0417s] not ok 20 bud --layers with --mount type bind should burst cache if content is changed
         # (from function `assert' in file tests/[helpers.bash, line 584](https://github.com/containers/podman/blob/141bd613b80603350685837a4d2feed510ffa5d0/test/system/helpers.bash#L584),
         #  in test file tests/[bud.bats, line 418](https://github.com/containers/podman/blob/141bd613b80603350685837a4d2feed510ffa5d0/test/system/bud.bats#L418))
         #   `assert "$output" !~ "samplefile"' failed
         # /var/tmp/go/src/github.com/containers/podman/test-buildah-v1.38.1-0.20250125114111-92015b7f4301/tests /var/tmp/go/src/github.com/containers/podman/test-buildah-v1.38.1-0.20250125114111-92015b7f4301
         # # [checking for: docker.io/library/alpine]
         # # [restoring from cache: /var/tmp/bats-run-Y6YQPM/suite/buildah-image-cache / docker.io/library/alpine]
         # Getting image source signatures
         # Copying blob sha256:9d16cba9fb961d1aafec9542f2bf7cb64acfc55245f9e4eb5abecd4cdc38d749
         # Copying config sha256:961769676411f082461f9ef46626dd7a2d1e2b2a38e6a44364bcbecf51e66dd4
         # Writing manifest to image destination
         # # podman-remote build --force-rm=false --signature-policy /var/tmp/go/src/github.com/containers/podman/test-buildah-v1.38.1-0.20250125114111-92015b7f4301/tests/policy.json --layers -t source -f /var/tmp/buildah_tests.i5jimr/bud/platform/Containerfile /var/tmp/buildah_tests.i5jimr/bud/platform
         # STEP 1/2: FROM alpine
         # STEP 2/2: RUN --mount=type=bind,target=/test,Z ls /test
         # Containerfile
         # samplefile
         # COMMIT source
         # --> 5a194ca19ce0
         # Successfully tagged localhost/source:latest
         # 5a194ca19ce0776e370df42296e5db26d3cc4e68d8747b18f5ddba64885b381b
         # # podman-remote build --force-rm=false --signature-policy /var/tmp/go/src/github.com/containers/podman/test-buildah-v1.38.1-0.20250125114111-92015b7f4301/tests/policy.json --layers -t source -f /var/tmp/buildah_tests.i5jimr/bud/platform/Containerfile /var/tmp/buildah_tests.i5jimr/bud/platform
         # STEP 1/2: FROM alpine
         # STEP 2/2: RUN --mount=type=bind,target=/test,Z ls /test
         # Containerfile
         # samplefile
         # COMMIT source
         # --> a90cbb577250
         # Successfully tagged localhost/source:latest
         # a90cbb5772508e03adbadfc2fa74860fbf3e62473d4adf3316c70e9aa9c532ec
         # #/vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
         # #|     FAIL: podman-remote build --force-rm=false --signature-policy /var/tmp/go/src/github.com/containers/podman/test-buildah-v1.38.1-0.20250125114111-92015b7f4301/tests/policy.json --layers -t source -f /var/tmp/buildah_tests.i5jimr/bud/platform/Containerfile /var/tmp/buildah_tests.i5jimr/bud/platform
         # #| expected: !~ 'samplefile'
         # #|   actual:    'STEP 1/2: FROM alpine'
         # #|         >    'STEP 2/2: RUN --mount=type=bind,target=/test,Z ls /test'
         # #|         >    'Containerfile'
         # #|         >    'samplefile'
         # #|         >    'COMMIT source'
         # #|         >    '--> a90cbb577250'
         # #|         >    'Successfully tagged localhost/source:latest'
         # #|         >    'a90cbb5772508e03adbadfc2fa74860fbf3e62473d4adf3316c70e9aa9c532ec'
         # #\^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
         # teardown: stopping podman server 18973
         # /var/tmp/go/src/github.com/containers/podman/test-buildah-v1.38.1-0.20250125114111-92015b7f4301
[+0422s] not ok 21 bud --layers with --mount type bind should burst and multiple mounts cache if content is changed
         # (from function `assert' in file tests/[helpers.bash, line 584](https://github.com/containers/podman/blob/141bd613b80603350685837a4d2feed510ffa5d0/test/system/helpers.bash#L584),
         #  in test file tests/[bud.bats, line 456](https://github.com/containers/podman/blob/141bd613b80603350685837a4d2feed510ffa5d0/test/system/bud.bats#L456))
         #   `assert "$output" !~ "samplefile"' failed
         # /var/tmp/go/src/github.com/containers/podman/test-buildah-v1.38.1-0.20250125114111-92015b7f4301/tests /var/tmp/go/src/github.com/containers/podman/test-buildah-v1.38.1-0.20250125114111-92015b7f4301
         # # [checking for: docker.io/library/alpine]
         # # [restoring from cache: /var/tmp/bats-run-Y6YQPM/suite/buildah-image-cache / docker.io/library/alpine]
         # Getting image source signatures
         # Copying blob sha256:9d16cba9fb961d1aafec9542f2bf7cb64acfc55245f9e4eb5abecd4cdc38d749
         # Copying config sha256:961769676411f082461f9ef46626dd7a2d1e2b2a38e6a44364bcbecf51e66dd4
         # Writing manifest to image destination
         # # podman-remote build --force-rm=false --signature-policy /var/tmp/go/src/github.com/containers/podman/test-buildah-v1.38.1-0.20250125114111-92015b7f4301/tests/policy.json --layers -t source -f /var/tmp/buildah_tests.jviamx/bud/platform/Containerfile /var/tmp/buildah_tests.jviamx/bud/platform
         # STEP 1/2: FROM alpine
         # STEP 2/2: RUN --mount=type=bind,target=/test,Z --mount=type=bind,source=testfile,target=testfile,Z ls /test && cat testfile
         # Containerfile
         # samplefile
         # testfile
         # Helloworld
         # COMMIT source
         # --> 83f6b40008da
         # Successfully tagged localhost/source:latest
         # 83f6b40008dacda7e0fd4402ba0c7445454e2d8d83b41cd62ad9a4dcee227f00
         # # podman-remote build --force-rm=false --signature-policy /var/tmp/go/src/github.com/containers/podman/test-buildah-v1.38.1-0.20250125114111-92015b7f4301/tests/policy.json --layers -t source -f /var/tmp/buildah_tests.jviamx/bud/platform/Containerfile /var/tmp/buildah_tests.jviamx/bud/platform
         # STEP 1/2: FROM alpine
         # STEP 2/2: RUN --mount=type=bind,target=/test,Z --mount=type=bind,source=testfile,target=testfile,Z ls /test && cat testfile
         # Containerfile
         # samplefile
         # testfile
         # Helloworld
         # COMMIT source
         # --> 93fec02f708b
         # Successfully tagged localhost/source:latest
         # 93fec02f708bacaee05466983a1af6dab58715cd38d33630b34ca1fa53869fd7
         # #/vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
         # #|     FAIL: podman-remote build --force-rm=false --signature-policy /var/tmp/go/src/github.com/containers/podman/test-buildah-v1.38.1-0.20250125114111-92015b7f4301/tests/policy.json --layers -t source -f /var/tmp/buildah_tests.jviamx/bud/platform/Containerfile /var/tmp/buildah_tests.jviamx/bud/platform
         # #| expected: !~ 'samplefile'
         # #|   actual:    'STEP 1/2: FROM alpine'
         # #|         >    'STEP 2/2: RUN --mount=type=bind,target=/test,Z --mount=type=bind,source=testfile,target=testfile,Z ls /test && cat testfile'
         # #|         >    'Containerfile'
         # #|         >    'samplefile'
         # #|         >    'testfile'
         # #|         >    'Helloworld'
         # #|         >    'COMMIT source'
         # #|         >    '--> 93fec02f708b'
         # #|         >    'Successfully tagged localhost/source:latest'
         # #|         >    '93fec02f708bacaee05466983a1af6dab58715cd38d33630b34ca1fa53869fd7'
         # #\^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
         # teardown: stopping podman server 19320
         # /var/tmp/go/src/github.com/containers/podman/test-buildah-v1.38.1-0.20250125114111-92015b7f4301

Failing log: https://api.cirrus-ci.com/v1/artifact/task/5781189207785472/html/bud-remote-fedora-41-root-host-sqlite.log.html

I am going to skip them on remote for now but I think this should be investigated.

[flouthoc]

I am not sure if these tests should be triggerd in remote or not, because the modification signature of file are changed everytime when files are sent from client to backend causing the cache burst, in case of remote setup and we cant do these calculations on client side since processing of Containerfile only happens on backend.

[flouthoc]

I will comment here if I can think of any method.

[Luap99]

Well if caching does not work with remote here that limits a lot of users (given windows/macos...)

Isn't the proper solution to teach the remote client when sending the tarball to preserve the file modified time and other file attributes that are used to check for cache invalidation?