[Bug]: Cannot sync from an Harbor registry

[pedroosorio]

zot version

v2.1.0

Describe the bug

I'm configuring Zot to mirror from an Harbor registry - https://harbor.skao.int. The pull-through (on demand) is working fine, but i cannot get the scheduled scan to work.

To reproduce

My configuration is:

{
  "distSpecVersion": "1.1.0",
  "storage": {
    "rootDirectory": "/data/zot",
    "dedupe": true,
    "gc": true
  },
  "http": {
    "address": "0.0.0.0",
    "port": "9090",
    "realm": "zot",
    "auth": {
      "htpasswd": {
        "path": "/etc/zot/htpasswd"
      },
      "failDelay": 5
    },
    "accessControl": {
      "repositories": {
        "**": {
          "anonymousPolicy": ["read"]
        }
      },
      "adminPolicy": {
        "users": ["admin"],
        "actions": ["read", "create", "update", "delete"]
      }
    }
  },
  "log": {
    "level": "debug"
  },
  "extensions": {
    "search": {
      "enable": true
    },
    "ui": {
      "enable": true
    },
    "sync": {
      "credentialsFile": "/etc/zot/credentials.json",
      "registries": [
        {
          "urls": ["https://harbor.skao.int"],
          "pollInterval": "1m",
          "onDemand": true,
          "tlsVerify": true,
          "maxRetries": 3,
          "retryDelay": "5m",
          "content": [
            {
              "prefix": "/test-promotion/ska-tango-tangogql"
            }
          ]
        }
      ]
    }
  }
}

Expected behavior

I would expect the sync to actually work. I suspect this might come from the fact that we have custom Nginx code in front of Harbor to allow it to "answer" to other domains on a specific project. Although, the registry works just fine with podman/docker CLI login and pull/push wise. It also works with ORAS cli, so if there was something really wrong under the hood, some of these would fail i guess.

Screenshots

Jan 14 16:15:54 stfc-techops-staging-oci-test-m1 zot[98908]: {"level":"info","goroutine":40,"caller":"zotregistry.dev/zot/pkg/extensions/sync/service.go:508","time":"2025-01-14T16:15:54.690285291Z","message":"getting available client"}
Jan 14 16:15:54 stfc-techops-staging-oci-test-m1 zot[98908]: {"level":"error","error":"Get \"?account=<redacted>&scope=&service=\": unsupported protocol scheme \"\"","url":"?account=<redacted>&scope=&service=","component":"sync","errorType":"*url.Error","goroutine":40,"caller":"zotregistry.dev/zot/pkg/extensions/sync/httpclient/client.go:275","time":"2025-01-14T16:15:54.741970344Z","message":"failed to make request"}
Jan 14 16:15:54 stfc-techops-staging-oci-test-m1 zot[98908]: {"level":"error","error":"Get \"?account=<redacted>&scope=&service=\": unsupported protocol scheme \"\"","url":"https://harbor.skao.int/v2/_catalog","component":"sync","errorType":"*url.Error","goroutine":40,"caller":"zotregistry.dev/zot/pkg/extensions/sync/httpclient/client.go:244","time":"2025-01-14T16:15:54.742057614Z","message":"failed to get token from authorization realm"}
Jan 14 16:15:54 stfc-techops-staging-oci-test-m1 zot[98908]: {"level":"error","error":"Get \"?account=<redacted>&scope=&service=\": unsupported protocol scheme \"\"","url":"https://harbor.skao.int/v2/_catalog","component":"sync","errorType":"*url.Error","goroutine":40,"caller":"zotregistry.dev/zot/pkg/extensions/sync/httpclient/client.go:206","time":"2025-01-14T16:15:54.742122775Z","message":"failed to make request"}
Jan 14 16:15:54 stfc-techops-staging-oci-test-m1 zot[98908]: {"level":"error","errorType":"*url.Error","remote registry":"https://harbor.skao.int","error":"Get \"?account=<redacted>&scope=&service=\": unsupported protocol scheme \"\"","goroutine":40,"caller":"zotregistry.dev/zot/pkg/extensions/sync/service.go:228","time":"2025-01-14T16:15:54.742173066Z","message":"failed to get repository list from remote registry"}
Jan 14 16:15:54 stfc-techops-staging-oci-test-m1 zot[98908]: {"level":"error","component":"scheduler","error":"Get \"?account=<redacted>&scope=&service=\": unsupported protocol scheme \"\"","generator":"SyncGenerator","goroutine":40,"caller":"zotregistry.dev/zot/pkg/scheduler/scheduler.go:468","time":"2025-01-14T16:15:54.742215717Z","message":"failed to execute generator"}

Additional context

No response

[rchincha]

@pedroosorio can you also report logs from harbor side?
There should really be no difference unless there is some uri mismatch (somehow)

[eusebiu-constantin-petu-dbk]

Hello @pedroosorio

I need to know how the www-authenticate header looks if you make an unathenticated GET on the catalog endpoint.
Can you please show that? I think I can fix it if I know this information.

like curl -vvv https://harbor.skao.int/v2/_catalog

Thank you!

[andaaron]

Hello @pedroosorio

I need to know how the www-authenticate header looks if you make an unathenticated GET on the catalog endpoint. Can you please show that? I think I can fix it if I know this information.

like curl -vvv https://harbor.skao.int/v2/_catalog

Thank you!

* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
< HTTP/2 401
< date: Thu, 16 Jan 2025 09:58:17 GMT
< content-type: application/json; charset=utf-8
< content-length: 108
< server: nginx
< docker-distribution-api-version: registry/2.0
< set-cookie: sid=04d85f456e26e9df59abf82e1450572c; Path=/; HttpOnly
< x-request-id: 0de31032-2c01-4b63-a64f-878d47f87108
< www-authenticate: Basic realm="harbor"
<
{"errors":[{"code":"UNAUTHORIZED","message":"unauthorized to list catalog: unauthorized to list catalog"}]}

[vooon]

Got the same problem, and the source is here:

https://github.com/goharbor/harbor/blob/9e8e647b713b9e33d9b8c32adbad066f1d7b5bf6/src/server/middleware/v2auth/auth.go#L95-L98

	if len(auth) > 0 || lib.V2CatalogURLRe.MatchString(req.URL.Path) {
		// Return basic auth challenge by default, incl. request to '/v2/_catalog'
		return `Basic realm="harbor"`
	}

[vooon]

At the same time it's unclear to me, why Zot uses bearer protocol for basic auth. Podman, Docker, Scopeo works well with the Harbor.

[eusebiu-constantin-petu-dbk]

Added a patch that is fixing this issue.

At the same time it's unclear to me, why Zot uses bearer protocol for basic auth. Podman, Docker, Scopeo works well with the Harbor.

Because zot is taking the auth scheme from /v2/ route, maybe we should change that.

[andaaron]

Because zot is taking the auth scheme from /v2/ route, maybe we should change that.

Why? That is what the OCI community decided to do.

[andaaron]

And here's the relevant harbor issue: goharbor/harbor#13573

[eusebiu-constantin-petu-dbk]

@vooon Can you try the patch. I already tested it with harbor's demo instance and it.s working.

[vooon]

@eusebiu-constantin-petu-dbk i applied your patch on top of #2865 (as i need that functionality).

Works fine! Both, onDemand: true and false!

[pedroosorio]

Hi all !

Wasn't expecting this much traction, but i really appreciate it. As i understand your comments, it is a bug in Harbor as much as it is a bug in Zot ?

Got a little bit confused 💃