Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Terraform provider / module download from custom private registry doesn't pass bearer token #36011

Open
yolocs opened this issue Nov 15, 2024 · 3 comments
Open

Terraform provider / module download from custom private registry doesn't pass bearer token #36011

yolocs opened this issue Nov 15, 2024 · 3 comments
Labels
documentation new new issue not yet triaged

Comments

@yolocs
Copy link

yolocs commented Nov 15, 2024

Terraform Version

terraform version
Terraform v1.9.8
on darwin_arm64

Terraform Configuration Files

terraform {
  required_providers {
    random = {
      source  = "ar-terraform-registry.us-central1.run.app/tf-registry-backend/random"
      version = "3.6.3"
    }
  }
}

resource "random_id" "rand" {
  byte_length = 8
}

The private registry at host ar-terraform-registry.us-central1.run.app requires authentication. I have manually updated ~/.terraform.d/credentials.tfrc.json to include the auth token.

{
  "credentials": {
    "ar-terraform-registry.us-central1.run.app": {
      "token": "eyJh..."
    }
  }
}

Debug Output

TF_LOG=TRACE terraform init
2024-11-14T21:50:00.684-0800 [INFO]  Terraform version: 1.9.8
2024-11-14T21:50:00.684-0800 [DEBUG] using github.com/hashicorp/go-tfe v1.58.0
2024-11-14T21:50:00.684-0800 [DEBUG] using github.com/hashicorp/hcl/v2 v2.20.0
2024-11-14T21:50:00.684-0800 [DEBUG] using github.com/hashicorp/terraform-svchost v0.1.1
2024-11-14T21:50:00.684-0800 [DEBUG] using github.com/zclconf/go-cty v1.14.4
2024-11-14T21:50:00.684-0800 [INFO]  Go runtime version: go1.22.7
2024-11-14T21:50:00.684-0800 [INFO]  CLI args: []string{"terraform", "init"}
2024-11-14T21:50:00.684-0800 [TRACE] Stdout is a terminal of width 198
2024-11-14T21:50:00.684-0800 [TRACE] Stderr is a terminal of width 198
2024-11-14T21:50:00.684-0800 [TRACE] Stdin is a terminal
2024-11-14T21:50:00.684-0800 [DEBUG] Attempting to open CLI config file: /Users/cshou/.terraformrc
2024-11-14T21:50:00.684-0800 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2024-11-14T21:50:00.684-0800 [INFO]  Loading CLI configuration from /Users/cshou/.terraform.d/credentials.tfrc.json
2024-11-14T21:50:00.685-0800 [DEBUG] ignoring non-existing provider search directory terraform.d/plugins
2024-11-14T21:50:00.685-0800 [DEBUG] ignoring non-existing provider search directory /Users/cshou/.terraform.d/plugins
2024-11-14T21:50:00.685-0800 [DEBUG] ignoring non-existing provider search directory /Users/cshou/Library/Application Support/io.terraform/plugins
2024-11-14T21:50:00.685-0800 [DEBUG] ignoring non-existing provider search directory /Library/Application Support/io.terraform/plugins
2024-11-14T21:50:00.686-0800 [INFO]  CLI command args: []string{"init"}
Initializing the backend...
2024-11-14T21:50:00.688-0800 [TRACE] Meta.Backend: no config given or present on disk, so returning nil config
2024-11-14T21:50:00.688-0800 [TRACE] Meta.Backend: backend has not previously been initialized in this working directory
2024-11-14T21:50:00.688-0800 [TRACE] Meta.Backend: using default local state only (no backend configuration, and no existing initialized backend)
2024-11-14T21:50:00.688-0800 [TRACE] Meta.Backend: instantiated backend of type <nil>
2024-11-14T21:50:00.689-0800 [TRACE] providercache.fillMetaCache: scanning directory .terraform/providers
2024-11-14T21:50:00.689-0800 [TRACE] getproviders.SearchLocalDirectory: failed to resolve symlinks for .terraform/providers: lstat .terraform: no such file or directory
2024-11-14T21:50:00.689-0800 [TRACE] providercache.fillMetaCache: error while scanning directory .terraform/providers: cannot search .terraform/providers: lstat .terraform/providers: no such file or directory
2024-11-14T21:50:00.689-0800 [TRACE] providercache.fillMetaCache: scanning directory .terraform/providers
2024-11-14T21:50:00.689-0800 [TRACE] getproviders.SearchLocalDirectory: failed to resolve symlinks for .terraform/providers: lstat .terraform: no such file or directory
2024-11-14T21:50:00.689-0800 [TRACE] providercache.fillMetaCache: error while scanning directory .terraform/providers: cannot search .terraform/providers: lstat .terraform/providers: no such file or directory
2024-11-14T21:50:00.689-0800 [TRACE] providercache.fillMetaCache: scanning directory .terraform/providers
2024-11-14T21:50:00.689-0800 [TRACE] getproviders.SearchLocalDirectory: failed to resolve symlinks for .terraform/providers: lstat .terraform: no such file or directory
2024-11-14T21:50:00.689-0800 [TRACE] providercache.fillMetaCache: error while scanning directory .terraform/providers: cannot search .terraform/providers: lstat .terraform/providers: no such file or directory
2024-11-14T21:50:00.689-0800 [DEBUG] checking for provisioner in "."
2024-11-14T21:50:00.693-0800 [DEBUG] checking for provisioner in "/opt/homebrew/bin"
2024-11-14T21:50:00.693-0800 [TRACE] Meta.Backend: backend <nil> does not support operations, so wrapping it in a local backend
2024-11-14T21:50:00.693-0800 [TRACE] backend/local: state manager for workspace "default" will:
 - read initial snapshot from terraform.tfstate
 - write new snapshots to terraform.tfstate
 - create any backup at terraform.tfstate.backup
2024-11-14T21:50:00.693-0800 [TRACE] statemgr.Filesystem: reading initial snapshot from terraform.tfstate
2024-11-14T21:50:00.693-0800 [TRACE] statemgr.Filesystem: snapshot file has nil snapshot, but that's okay
2024-11-14T21:50:00.693-0800 [TRACE] statemgr.Filesystem: read nil snapshot
Initializing provider plugins...
- Reusing previous version of ar-terraform-registry.us-central1.run.app/tf-registry-backend/random from the dependency lock file
2024-11-14T21:50:00.694-0800 [DEBUG] Service discovery for ar-terraform-registry.us-central1.run.app at https://ar-terraform-registry.us-central1.run.app/.well-known/terraform.json
2024-11-14T21:50:00.694-0800 [TRACE] HTTP client GET request to https://ar-terraform-registry.us-central1.run.app/.well-known/terraform.json
2024-11-14T21:50:00.813-0800 [DEBUG] GET https://ar-terraform-registry.us-central1.run.app/v1/providers/tf-registry-backend/random/versions
2024-11-14T21:50:00.813-0800 [TRACE] HTTP client GET request to https://ar-terraform-registry.us-central1.run.app/v1/providers/tf-registry-backend/random/versions
2024-11-14T21:50:01.089-0800 [TRACE] providercache.fillMetaCache: scanning directory .terraform/providers
2024-11-14T21:50:01.089-0800 [TRACE] getproviders.SearchLocalDirectory: failed to resolve symlinks for .terraform/providers: lstat .terraform: no such file or directory
2024-11-14T21:50:01.089-0800 [TRACE] providercache.fillMetaCache: error while scanning directory .terraform/providers: cannot search .terraform/providers: lstat .terraform/providers: no such file or directory
2024-11-14T21:50:01.089-0800 [DEBUG] GET https://ar-terraform-registry.us-central1.run.app/v1/providers/tf-registry-backend/random/3.6.3/download/darwin/arm64
2024-11-14T21:50:01.089-0800 [TRACE] HTTP client GET request to https://ar-terraform-registry.us-central1.run.app/v1/providers/tf-registry-backend/random/3.6.3/download/darwin/arm64
2024-11-14T21:50:02.087-0800 [DEBUG] GET https://ar-terraform-registry.us-central1.run.app/download/provider/tf-registry-backend/asset/random:3.6.3-darwin-arm64:terraform-provider-random_3.6.3_SHA256SUMS
2024-11-14T21:50:02.087-0800 [TRACE] HTTP client GET request to https://ar-terraform-registry.us-central1.run.app/download/provider/tf-registry-backend/asset/random:3.6.3-darwin-arm64:terraform-provider-random_3.6.3_SHA256SUMS
╷
│ Error: Failed to install provider
│
│ Error while installing ar-terraform-registry.us-central1.run.app/tf-registry-backend/random v3.6.3: could not query provider registry for
│ ar-terraform-registry.us-central1.run.app/tf-registry-backend/random: failed to retrieve authentication checksums for provider: 403 Forbidden returned from
│ ar-terraform-registry.us-central1.run.app
╵

Expected Behavior

The download URL https://ar-terraform-registry.us-central1.run.app/download/provider/tf-registry-backend/asset/random:3.6.3-darwin-arm64:terraform-provider-random_3.6.3_SHA256SUMS is also behind the same host. Since I've configured ~/.terraform.d/credentials.tfrc.json, I expected Terraform to use the credential I added there for the downloads.

Actual Behavior

As shown in the debug output, Terraform failed to download the file because it's not using the credential I have specified. I also confirmed that the server side didn't receive any bearer token. It seems Terraform assumes the download URLs are publicly accessible.

Also notice, the previous calls for service discovery and getting version (e.g. GET https://ar-terraform-registry.us-central1.run.app/v1/providers/tf-registry-backend/random/3.6.3/download/darwin/arm64) were all successful because Terraform used the auth token in those requests. It's only the download call that's impacted.

Steps to Reproduce

I can't make the service accessible to you. But I'd expect any other custom private registry will have the same problem if the download URLs require authentication.

Additional Context

I also tried to set TF_TOKEN_ar__terraform__registry_us__central1_run_app to my auth token per https://developer.hashicorp.com/terraform/cli/config/config-file#environment-variable-credentials. The result is the same. The download requests didn't honor this config.

The other reference is https://developer.hashicorp.com/terraform/internals/provider-registry-protocol#response-properties-1. It doesn't have any mentioning that the download URLs must be public accessible links.

To prove my custom private registry actually works, terraform init worked out fine if I disable authentication. See below:

TF_LOG=TRACE terraform init
2024-11-14T22:06:44.882-0800 [INFO]  Terraform version: 1.9.8
2024-11-14T22:06:44.883-0800 [DEBUG] using github.com/hashicorp/go-tfe v1.58.0
2024-11-14T22:06:44.883-0800 [DEBUG] using github.com/hashicorp/hcl/v2 v2.20.0
2024-11-14T22:06:44.883-0800 [DEBUG] using github.com/hashicorp/terraform-svchost v0.1.1
2024-11-14T22:06:44.883-0800 [DEBUG] using github.com/zclconf/go-cty v1.14.4
2024-11-14T22:06:44.883-0800 [INFO]  Go runtime version: go1.22.7
2024-11-14T22:06:44.883-0800 [INFO]  CLI args: []string{"terraform", "init"}
2024-11-14T22:06:44.883-0800 [TRACE] Stdout is a terminal of width 198
2024-11-14T22:06:44.883-0800 [TRACE] Stderr is a terminal of width 198
2024-11-14T22:06:44.883-0800 [TRACE] Stdin is a terminal
2024-11-14T22:06:44.883-0800 [DEBUG] Attempting to open CLI config file: /Users/cshou/.terraformrc
2024-11-14T22:06:44.883-0800 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2024-11-14T22:06:44.883-0800 [DEBUG] ignoring non-existing provider search directory terraform.d/plugins
2024-11-14T22:06:44.883-0800 [DEBUG] ignoring non-existing provider search directory /Users/cshou/.terraform.d/plugins
2024-11-14T22:06:44.883-0800 [DEBUG] ignoring non-existing provider search directory /Users/cshou/Library/Application Support/io.terraform/plugins
2024-11-14T22:06:44.883-0800 [DEBUG] ignoring non-existing provider search directory /Library/Application Support/io.terraform/plugins
2024-11-14T22:06:44.883-0800 [INFO]  CLI command args: []string{"init"}
Initializing the backend...
2024-11-14T22:06:44.884-0800 [TRACE] Meta.Backend: no config given or present on disk, so returning nil config
2024-11-14T22:06:44.884-0800 [TRACE] Meta.Backend: backend has not previously been initialized in this working directory
2024-11-14T22:06:44.884-0800 [TRACE] Meta.Backend: using default local state only (no backend configuration, and no existing initialized backend)
2024-11-14T22:06:44.884-0800 [TRACE] Meta.Backend: instantiated backend of type <nil>
2024-11-14T22:06:44.884-0800 [TRACE] providercache.fillMetaCache: scanning directory .terraform/providers
2024-11-14T22:06:44.884-0800 [TRACE] getproviders.SearchLocalDirectory: failed to resolve symlinks for .terraform/providers: lstat .terraform: no such file or directory
2024-11-14T22:06:44.884-0800 [TRACE] providercache.fillMetaCache: error while scanning directory .terraform/providers: cannot search .terraform/providers: lstat .terraform/providers: no such file or directory
2024-11-14T22:06:44.884-0800 [DEBUG] checking for provisioner in "."
2024-11-14T22:06:44.884-0800 [DEBUG] checking for provisioner in "/opt/homebrew/bin"
2024-11-14T22:06:44.885-0800 [TRACE] Meta.Backend: backend <nil> does not support operations, so wrapping it in a local backend
2024-11-14T22:06:44.885-0800 [TRACE] backend/local: state manager for workspace "default" will:
 - read initial snapshot from terraform.tfstate
 - write new snapshots to terraform.tfstate
 - create any backup at terraform.tfstate.backup
2024-11-14T22:06:44.885-0800 [TRACE] statemgr.Filesystem: reading initial snapshot from terraform.tfstate
2024-11-14T22:06:44.885-0800 [TRACE] statemgr.Filesystem: snapshot file has nil snapshot, but that's okay
2024-11-14T22:06:44.885-0800 [TRACE] statemgr.Filesystem: read nil snapshot
Initializing provider plugins...
- Reusing previous version of ar-terraform-registry.us-central1.run.app/tf-registry-backend/random from the dependency lock file
2024-11-14T22:06:44.885-0800 [DEBUG] Service discovery for ar-terraform-registry.us-central1.run.app at https://ar-terraform-registry.us-central1.run.app/.well-known/terraform.json
2024-11-14T22:06:44.885-0800 [TRACE] HTTP client GET request to https://ar-terraform-registry.us-central1.run.app/.well-known/terraform.json
2024-11-14T22:06:44.984-0800 [DEBUG] GET https://ar-terraform-registry.us-central1.run.app/v1/providers/tf-registry-backend/random/versions
2024-11-14T22:06:44.984-0800 [TRACE] HTTP client GET request to https://ar-terraform-registry.us-central1.run.app/v1/providers/tf-registry-backend/random/versions
2024-11-14T22:06:45.160-0800 [TRACE] providercache.fillMetaCache: scanning directory .terraform/providers
2024-11-14T22:06:45.161-0800 [TRACE] getproviders.SearchLocalDirectory: failed to resolve symlinks for .terraform/providers: lstat .terraform: no such file or directory
2024-11-14T22:06:45.161-0800 [TRACE] providercache.fillMetaCache: error while scanning directory .terraform/providers: cannot search .terraform/providers: lstat .terraform/providers: no such file or directory
2024-11-14T22:06:45.161-0800 [DEBUG] GET https://ar-terraform-registry.us-central1.run.app/v1/providers/tf-registry-backend/random/3.6.3/download/darwin/arm64
2024-11-14T22:06:45.161-0800 [TRACE] HTTP client GET request to https://ar-terraform-registry.us-central1.run.app/v1/providers/tf-registry-backend/random/3.6.3/download/darwin/arm64
2024-11-14T22:06:45.645-0800 [DEBUG] GET https://ar-terraform-registry.us-central1.run.app/download/provider/tf-registry-backend/asset/random:3.6.3-darwin-arm64:terraform-provider-random_3.6.3_SHA256SUMS
2024-11-14T22:06:45.646-0800 [TRACE] HTTP client GET request to https://ar-terraform-registry.us-central1.run.app/download/provider/tf-registry-backend/asset/random:3.6.3-darwin-arm64:terraform-provider-random_3.6.3_SHA256SUMS
2024-11-14T22:06:45.824-0800 [DEBUG] GET https://ar-terraform-registry.us-central1.run.app/download/provider/tf-registry-backend/asset/random:3.6.3-darwin-arm64:terraform-provider-random_3.6.3_SHA256SUMS.sig
2024-11-14T22:06:45.824-0800 [TRACE] HTTP client GET request to https://ar-terraform-registry.us-central1.run.app/download/provider/tf-registry-backend/asset/random:3.6.3-darwin-arm64:terraform-provider-random_3.6.3_SHA256SUMS.sig
- Installing ar-terraform-registry.us-central1.run.app/tf-registry-backend/random v3.6.3...
2024-11-14T22:06:46.026-0800 [TRACE] providercache.Dir.InstallPackage: installing ar-terraform-registry.us-central1.run.app/tf-registry-backend/random v3.6.3 from https://ar-terraform-registry.us-central1.run.app/download/provider/tf-registry-backend/asset/random:3.6.3-darwin-arm64:terraform-provider-random_3.6.3_darwin_arm64.zip
2024-11-14T22:06:46.027-0800 [TRACE] HTTP client GET request to https://ar-terraform-registry.us-central1.run.app/download/provider/tf-registry-backend/asset/random:3.6.3-darwin-arm64:terraform-provider-random_3.6.3_darwin_arm64.zip
2024-11-14T22:06:46.653-0800 [DEBUG] Provider signed by 34365D9472D7468F HashiCorp Security (hashicorp.com/security) <[email protected]>
2024-11-14T22:06:46.823-0800 [TRACE] providercache.fillMetaCache: scanning directory .terraform/providers
2024-11-14T22:06:46.823-0800 [TRACE] getproviders.SearchLocalDirectory: found ar-terraform-registry.us-central1.run.app/tf-registry-backend/random v3.6.3 for darwin_arm64 at .terraform/providers/ar-terraform-registry.us-central1.run.app/tf-registry-backend/random/3.6.3/darwin_arm64
2024-11-14T22:06:46.823-0800 [TRACE] providercache.fillMetaCache: including .terraform/providers/ar-terraform-registry.us-central1.run.app/tf-registry-backend/random/3.6.3/darwin_arm64 as a candidate package for ar-terraform-registry.us-central1.run.app/tf-registry-backend/random 3.6.3
- Installed ar-terraform-registry.us-central1.run.app/tf-registry-backend/random v3.6.3 (signed by HashiCorp)

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

References

No response
@yolocs yolocs added bug new new issue not yet triaged labels Nov 15, 2024
@liamcervante
Copy link
Member

Hi @yolocs, thanks for filing this.

I can confirm that Terraform isn't attaching authentication to the follow up requests when retrieving the downloadUrl and the shasumUrl. This is because both the public and private registries return publicly available URLs for these attributes. This means that it is difficult for us to change this without breaking our primary points of integration.

I do agree that this behaviour isn't documented, I've tagged this issue as a documentation improvement and raised with the internal docs team.

Could you attach temporary credentials to the returned URLs? Or copy the behaviour of the private registry which returns temporary unauthenticated endpoints?

Thanks!

@apparentlymart apparentlymart mentioned this issue Nov 15, 2024
Open
@yolocs
Copy link
Author

yolocs commented Nov 15, 2024

Thanks for the prompt response!

A somewhat low risk change could be expanding the CLI config to indicate that downloads should also attach auth token. E.g.

{
  "credentials": {
    "ar-terraform-registry.us-central1.run.app": {
      "token": "eyJh...",
      "attachToDownloads": true
    }
  }
}

Existing user journeys should not be interrupted since it's a new config field.

Could you attach temporary credentials to the returned URLs? Or copy the behaviour of the private registry which returns temporary unauthenticated endpoints?

While it's possible, it's definitely not trivial. Not every storage system comes with a signed URL feature, in which case registry implementors would have to build their own signed URLs and signing keys, etc. Plus, having publicly accessible URLs, though temporary, increases the risk of data exfiltration. And that partly forfeits the purpose of having a private registry.

@yolocs
Copy link
Author

yolocs commented Nov 15, 2024 •
edited
Loading

Thanks @apparentlymart for linking the duplicate! And there seems to be a proposed fix in #29642 and #35843

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
documentation new new issue not yet triaged
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@yolocs @liamcervante