(aws_cdk.pipelines): ConfirmPermissionsBroadening only diffs on template differences

[bpauwels]

Describe the bug

When using the ConfirmPermissionsBroadening check on a cross account stage, the CodeBuild project lacks permissions to assume neccesary roles in the target account, cannot create a cfn changeset and performs a diff only on template differences.

Regression Issue

• Select this option if this issue appears to be a regression.

Last Known Working CDK Version

No response

Expected Behavior

A cloudformation changeset is being created in the target account and used as source to evaluate security related changes.

Current Behavior

Changeset cannot be created due to missing permissions on the Codebuild projects

fail: Could not assume role in target account using current credentials (which are for account <source account>) User: arn:aws:sts::<source account>:assumed-role/CdkCrossAccountConfirmPer-PipelinePipelinesSecurity-<redacted>/AWSCodeBuild-<redacted> is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::<target account>:role/cdk-hnb659fds-file-publishing-role-<redacted>-eu-central-1 . Please make sure that this role exists in the account. If it doesn't exist, (re)-bootstrap the environment with the right '--trust', using the latest version of the CDK CLI.
--
Could not create a change set, will base the diff on template differences (run again with -v to see the reason)

The target account is properly bootstrapped, the trust policy of the file publishing role allows sts assume role from the source account.
The codebuild service role has a condition on the relevant policy that prevents assuming the file publishing role:

        {
            "Condition": {
                "ForAnyValue:StringEquals": {
                    "iam:ResourceTag/aws-cdk:bootstrap-role": [
                        "deploy"
                    ]
                }
            },
            "Action": "sts:AssumeRole",
            "Resource": "*",
            "Effect": "Allow"
        },

Reproduction Steps

import * as cdk from "aws-cdk-lib";
import { Construct } from "constructs";
import { Repository } from "aws-cdk-lib/aws-codecommit";
import {
  CodePipeline,
  CodePipelineSource,
  ConfirmPermissionsBroadening,
  ShellStep,
} from "aws-cdk-lib/pipelines";
import * as sqs from "aws-cdk-lib/aws-sqs";

export class ExampleStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const queue = new sqs.Queue(this, "ExampleQueue");
  }
}

class MyApplication extends cdk.Stage {
  constructor(scope: Construct, id: string, props?: cdk.StageProps) {
    super(scope, id, props);
    new ExampleStack(this, "ExampleStack");
  }
}

export class CdkCrossAccountConfirmPermissionBroadeningStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const repo = new Repository(this, "CodeCommitRepo", {
      repositoryName: "TestRepo",
    });

    const pipeline = new CodePipeline(this, "Pipeline", {
      pipelineName: "CodeArtifactNPMPipeline",
      crossAccountKeys: true,
      synth: new ShellStep("Synth", {
        input: CodePipelineSource.codeCommit(repo, "main"),
        commands: ["npm ci", "npm run build", "npx cdk synth"],
      }),
    });

    const stage = new MyApplication(this, "Prod", {
      env: { account: "<target account>", region: "eu-central-1" },
    });
    pipeline.addStage(stage, {
      pre: [new ConfirmPermissionsBroadening("Check", { stage })],
    });
  }
}

Possible Solution

Fix the codebuild role

Additional Information/Context

No response

CDK CLI Version

2.176.0

Framework Version

No response

Node.js Version

v23.4.0

OS

MacOS

Language

TypeScript

Language Version

No response

Other information

No response

[ashishdhingra]

@bpauwels Looks like this issue is similar to #32482. Could you please check and confirm? If similar, please close this issue since the other one was already reviewed with the team.

Thanks,
Ashish

[bpauwels]

@ashishdhingra that looks like a different issue to me. The automatic approval was just working fine in my tests.

Thanks,
Benedikt

[ashishdhingra]

@ashishdhingra that looks like a different issue to me. The automatic approval was just working fine in my tests.

Thanks, Benedikt

@bpauwels Could you test your scenario with GitHub repo (refer #32482 (comment) for reproduction steps) and share if you are getting the same failed log? CodeCommit is not available for new customers anymore, so it might not be feasible to reproduce the issue.

Thanks,
Ashish

[bpauwels]

@ashishdhingra that looks like a different issue to me. The automatic approval was just working fine in my tests.
Thanks, Benedikt

@bpauwels Could you test your scenario with GitHub repo (refer #32482 (comment) for reproduction steps) and share if you are getting the same failed log? CodeCommit is not available for new customers anymore, so it might not be feasible to reproduce the issue.

Thanks, Ashish

Sure, here you go:

import * as cdk from "aws-cdk-lib";
import { Construct } from "constructs";
import { Repository } from "aws-cdk-lib/aws-codecommit";
import {
  CodePipeline,
  CodePipelineSource,
  ConfirmPermissionsBroadening,
  ShellStep,
} from "aws-cdk-lib/pipelines";
import { ExampleStack } from "./example-stack";

class MyApplication extends cdk.Stage {
  constructor(scope: Construct, id: string, props?: cdk.StageProps) {
    super(scope, id, props);
    new ExampleStack(this, "ExampleStack");
  }
}

export class CdkCrossAccountConfirmPermissionBroadeningStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const pipeline = new CodePipeline(this, "Pipeline", {
      pipelineName: "CodeArtifactNPMPipeline",
      crossAccountKeys: true,
      synth: new ShellStep("Synth", {
        input: CodePipelineSource.connection('<<user/repo>>', 'main', {
          connectionArn:
            'arn:aws:codestar-connections:<<REGION>>:<<ACCOUNT-ID>>:connection/<<GUID>>',
        }),
        commands: ["npm ci", "npm run build", "npx cdk synth"],
      })
    });

    const stage = new MyApplication(this, "Prod", {
      env: { account: "<<TARGET ACCOUNT ID>>>", region: "eu-central-1" },
    });
    pipeline.addStage(stage, {
      pre: [new ConfirmPermissionsBroadening("Check", { stage })],
    });
  }
}

The behaviour of the ConfirmPermissionsBroadening step did not change.

[pamann2]

+1 can confirm this bug. From my understanding the condition on the codebuild service role mentioned by @bpauwels needs to be updated. The tag on the role it tries to assume seems to be "lookup" not "deploy". When i tested a similar setup early Sept. last year everything worked fine. But i can not say exactly when this behavior changed.