Podman fails to pull nested OCI index

[mprasil]

Issue Description

Last few gitlab-runner images fail to pull:

podman pull gitlab/gitlab-runner:alpine3.18-0bb36a02
Trying to pull docker.io/gitlab/gitlab-runner:alpine3.18-0bb36a02...
Error: copying system image from manifest list: Unexpectedly received a manifest list instead of a manifest for a single image

Note that there is something specific about these that makes podman fail, older versions of gitlab-runner pull fine:

# this works as expected
podman pull gitlab/gitlab-runner:alpine-v17.7.1

All of the above images pull with docker just fine.

Steps to reproduce the issue

Steps to reproduce the issue

1. podman pull gitlab/gitlab-runner:alpine3.18-0bb36a02

Describe the results you received

Got the error:

Error: copying system image from manifest list: Unexpectedly received a manifest list instead of a manifest for a single image

Describe the results you expected

Image pulls as usual.

podman info output

I've been able to reproduce this with many versions of podman including this one:


Client:       Podman Engine
Version:      5.3.1
API Version:  5.3.1
Go Version:   go1.23.3
Built:        Thu Nov 21 00:00:00 2024
OS/Arch:      linux/amd64

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

Yes

Additional environment details

Note that this is not specific to Docker hub, pulling the same image from gitlab registry also fails:

podman pull registry.gitlab.com/gitlab-org/gitlab-runner:alpine-0bb36a02
Trying to pull registry.gitlab.com/gitlab-org/gitlab-runner:alpine-0bb36a02...
Error: copying system image from manifest list: Unexpectedly received a manifest list instead of a manifest for a single image

Additional information

Podman 5.2.3 also has this issue.

[formwandler]

Same issue for me pulling the gitlab-runner image since last GitLab update. I tested also on a different system without any container running. Same Podman version as in bug report.

[vrothberg]

The error points out the underlying issue:

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.index.v1+json",
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.index.v1+json",
      "digest": "sha256:cb1f1872857f8ba219bd08d81260503a168057960704df66e52217f2c85ba1fc",
      "size": 3117,
      "annotations": {
        "org.opencontainers.image.created": "2025-01-18T04:56:52Z"
      }
    }
  ]
}

registry.gitlab.com/gitlab-org/gitlab-runner:alpine-0bb36a02 is a nested OCI index and Podman does not support it.

@mtrmac PTAL

[vrothberg]

Docker is able to walk the list and pull the image.

[mprasil]

Note, that gitlab worked around this issue so latest gitlab-runner images should work now. The linked comment also contains some extra info that is likely relevant, I assume this is not the only project that might be using buildx bake in this way.

[vrothberg]

Thanks for sharing!

[mtrmac]

registry.gitlab.com/gitlab-org/gitlab-runner:alpine-0bb36a02 is a nested OCI index and Podman does not support it.

That’s a bit at odds with the c/image/copy conception of an image. The code could be built to recurse everywhere copying a nested structure, or to recurse looking for a single platform-specific image, but … what does that structure mean?

(Technically, if we recursed, we would need to, at least, arrange that in c/storage, the image can be looked up using digests of all manifests on the way, not just the per-platform one and the top-level one.)