Permadiff in pulumiLabels and effectiveLabels with empty labels

[VenelinMartinov]

Describe what happened

A user reported seeing a diff in effectiveLabels/pulumiLabels under 7.38

~ gcp:storage:Bucket example-gcp-ais-adv-bucket updating (0s) [diff: ~effectiveLabels,pulumiLabels

Sample program

GRPC logs:

{
    "method": "/pulumirpc.ResourceProvider/Diff",
    "request": {
        "id": "ID",
        "urn": "URN",
        "olds": {
            "__meta": "{\"e2bfb730-ecaa-11e6-8f88-34363bc7c4c0\":{\"create\":600000000000,\"read\":240000000000,\"update\":240000000000},\"schema_version\":\"1\"}",
            "autoclass": null,
            "cors": [],
            "customPlacementConfig": null,
            "defaultEventBasedHold": false,
            "effectiveLabels": {
                "tag": "2020-04-29",
                "prov": "cloud"
            },
            "enableObjectRetention": false,
            "encryption": {
                "defaultKmsKeyName": "KEY_NAME"
            },
            "forceDestroy": false,
            "id": "ID",
            "labels": {
                "app_id": "",
                "app_name": "",
                "env": "",
                "owner": "",
                "div": "",
                "sub_env": "",
                "tag": "2020-04-29",
                "prov": "cloud"
            },
            "lifecycleRules": [
            ],
            "location": "US",
            "logging": null,
            "name": "name",
            "project": "project",
            "projectNumber": 0,
            "publicAccessPrevention": "enforced",
            "pulumiLabels": {
                "tag": "2020-04-29",
                "prov": "cloud"
            },
            "requesterPays": false,
            "retentionPolicy": null,
            "rpo": "DEFAULT",
            "selfLink": "link",
            "softDeletePolicy": {
                "effectiveTime": "2024-09-08T19:23:04.697Z",
                "retentionDurationSeconds": 604800
            },
            "storageClass": "STANDARD",
            "uniformBucketLevelAccess": true,
            "url": "url",
            "versioning": {
                "enabled": true
            },
            "website": null
        },
        "news": {
            "__defaults": [
                "name"
            ],
            "encryption": {
                "__defaults": [],
                "defaultKmsKeyName": "name"
            },
            "forceDestroy": false,
            "labels": {
                "app_id": "",
                "app_name": "",
                "env": "",
                "owner": "",
                "div": "",
                "sub_env": "",
                "tag": "2020-04-29",
                "prov": "cloud"
            },
            "lifecycleRules": [
            ],
            "location": "us",
            "name": "name",
            "project": "project",
            "publicAccessPrevention": "enforced",
            "storageClass": "STANDARD",
            "uniformBucketLevelAccess": true,
            "versioning": {
                "__defaults": [],
                "enabled": true
            }
        },
        "oldInputs": {
            "__defaults": [
                "name"
            ],
            "encryption": {
                "__defaults": [],
                "defaultKmsKeyName": "name"
            },
            "forceDestroy": false,
            "labels": {
                "app_id": "",
                "app_name": "",
                "env": "",
                "owner": "",
                "div": "",
                "sub_env": "",
                "tag": "2020-04-29",
                "prov": "cloud"
            },
            "lifecycleRules": [
            ],
            "location": "us",
            "name": "name",
            "project": "project",
            "publicAccessPrevention": "enforced",
            "storageClass": "STANDARD",
            "uniformBucketLevelAccess": true,
            "versioning": {
                "__defaults": [],
                "enabled": true
            }
        }
    },
    "response": {
        "stables": [
            "enableObjectRetention",
            "location",
            "name",
            "project"
        ],
        "changes": "DIFF_SOME",
        "diffs": [
            "effectiveLabels",
            "pulumiLabels"
        ],
        "detailedDiff": {
            "effectiveLabels": {
                "kind": "UPDATE"
            },
            "pulumiLabels": {
                "kind": "UPDATE"
            }
        },
        "hasDetailedDiff": true
    },
    "metadata": {
        "kind": "resource",
        "mode": "client",
        "name": "gcp"
    }
}

Log output

No response

Affected Resource(s)

No response

Output of pulumi about

.

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[VenelinMartinov]

Attempted to repro

__main__.py:

import pulumi
import pulumi_gcp as gcp
from pulumi_gcp import storage


prov = gcp.Provider(
    "my-provider",
    default_labels={
        "cloud": "my-cloud",
        "empty_default_label": "",
    },
)

# Create a GCP resource (Storage Bucket)
bucket = storage.Bucket(
    "my-bucket",
    location="US",
    labels={
        "environment": "dev",
        "team": "data",
        "empty": "",
    },
    opts=pulumi.ResourceOptions(provider=prov),
)

pulumi.export("bucket_name", bucket.url)

repro.sh:

#! /bin/bash
set -e

rm -f grpc.json 
pulumi down --yes

sed -i '' 's/7.36.0/7.35.0/g' requirements.txt
pulumi install
PULUMI_DEBUG_GRPC="grpc.json" pulumi up --yes

eval "$(pulumi stack output --shell)"
gcloud storage buckets update $bucket_name --update-labels=unmanaged=value,unmanaged_empty=

sed -i '' 's/7.35.0/7.36.0/g' requirements.txt
pulumi install
PULUMI_DEBUG_GRPC="grpc.json" pulumi up --yes

pulumi preview --diff

Note here that I have tried:

• Labels on the bucket
• Empty labels on the bucket
• Default labels
• Empty default labels
• Unmanaged labels
• Empty unmanaged labels

This should cover all the ways to set a label on a GCP bucket but I have been unable to reproduce the issue so far.

[VenelinMartinov]

A few new notes on the issue:

• The issue happens when running 7.38 -> 7.38 as well - so it is NOT an upgrade issue.
• The user is using the GCP bucket from an MLC
• When going from 7.38 -> 7.38 the effectiveLabels and pulumiLabels properties are actually listed as deleted.
• The diff happens every time, even after accepting it.

[VenelinMartinov]

I believe I have a repro:

import pulumi
from pulumi_gcp import storage


bucketOne = storage.Bucket(
    "my-bucket-one",
    location="US",
    labels={
        "environment": "dev",
        "team": "data",
        "empty": "",
    },
)

# Create a GCP resource (Storage Bucket)
bucket = storage.Bucket(
    "my-bucket",
    location="US",
    labels={
        "environment": "dev",
        "team": "data",
        "empty": "",
        "other_bucket": bucketOne.name,
    },
)

pulumi.export("bucket_name", bucket.url)

Seems like the permadiff happens if the labels contain an output. Confirmed this does not happen in 7.35

[VenelinMartinov]

Simplified the repro with some help - the second resource is not necessary:

import pulumi
from pulumi_gcp import storage


bucket = storage.Bucket(
    "my-bucket",
    location="US",
    labels={
        "tag": pulumi.Output.from_input("val"),
        "empty": "",
    },
)

[VenelinMartinov]

Simplified it a bit more:

from pulumi_gcp import storage


bucket = storage.Bucket(
    "my-bucket",
    location="US",
    labels={
        "static": "static",
        "empty": "",
    },
)

Looks like the output type is not necessary either - we need to have an empty label and at least one more label for the permadiff to occur.

What happens is that the state has:

"terraform_labels":cty.MapVal(map[string]cty.Value{"static":cty.StringVal("static")})

while the plan is:

"terraform_labels":cty.MapVal(map[string]cty.Value{"empty":cty.UnknownVal(cty.String), "static":cty.StringVal("static")})

Unfortunately this reproduces in TF too:

provider "google" {
  region       = "us-central1"
}

resource "google_storage_bucket" "bucket" {
  name     = "example-bucket12312322314"
  location = "US"

  labels = {
    "static" = "static"
    "empty" = ""
  }
}

run terraform apply -auto-approve then terraform plan and observe the diff:

Terraform will perform the following actions:

  # google_storage_bucket.bucket will be updated in-place
  ~ resource "google_storage_bucket" "bucket" {
      ~ effective_labels            = {
          + "empty"                      = (known after apply)
            # (2 unchanged elements hidden)
        }
        id                          = "example-bucket12312322314"
        name                        = "example-bucket12312322314"
      ~ terraform_labels            = {
          + "empty"                      = (known after apply)
            # (2 unchanged elements hidden)
        }
        # (14 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Note that this happens under -refresh=true as well.

[VenelinMartinov]

Upstream issue: hashicorp/terraform-provider-google#16750

[VenelinMartinov]

Built a synthetic repro for the issue here: pulumi/pulumi-terraform-bridge#2412

Ran it with and without PRC to compare the values we produce and see why non-PRC passes while PRC produces the perma-diff. Very little difference, only:

RawState.v:

nil(in PRC) vs non-PRC:

v:  map[string]interface {}{
	"id":               nil,
	"labels":           nil,
	"terraform_labels": nil,
},

in both the state and plan.

[pulumi-bot]

This issue has been addressed in PR #2386 and shipped in release v8.2.0.