Skip to content

Commit

Permalink
fix: Work around limit in lambda policy size
Browse files Browse the repository at this point in the history
Currently failing with
```
│ Error: adding Lambda Permission (arn:aws:lambda:us-east-2:591234544403:function:model-inference-development-datadog-forwarder/_aws_sagemaker_Endpoints_recs-ranker-rn-scribd-has-read-after-AllowExecutionFromCloudWatchLogs): PolicyLengthExceededException: The final policy size (20782) is bigger than the limit (20480).
│ {
│   RespMetadata: {
│     StatusCode: 400,
│     RequestID: "e220c7fb-c4c9-4b93-9f15-beba592a3afc"
│   },
│   Message_: "The final policy size (20782) is bigger than the limit (20480).",
│   Type: "User"
│ }
│
│   with module.datadog[0].aws_lambda_permission.allow_cloudwatch_logs_to_call_dd_lambda_handler["/aws/sagemaker/Endpoints/recs-ranker-rn-scribd-has-read-after"],
│   on .terraform/modules/datadog/logs_monitoring_cloudwatch_log.tf line 10, in resource "aws_lambda_permission" "allow_cloudwatch_logs_to_call_dd_lambda_handler":
│   10: resource "aws_lambda_permission" "allow_cloudwatch_logs_to_call_dd_lambda_handler" {
}
```

Current size of lambda policy:
```
$ aws lambda get-policy --function-name model-inference-development-datadog-forwarder | jq -r '.Policy' | wc
       1       1   20090
```

Now we can pass array like `log_group_prefixes = ["/aws/sagemaker/Endpoints/", "/aws/sagemaker/TransformJobs"]`
to force small size of the policy if necessary

Changed to the module are backward-compatible
  • Loading branch information
zbstof committed Sep 30, 2024
1 parent 3678529 commit 201d6c3
Show file tree
Hide file tree
Showing 3 changed files with 10 additions and 2 deletions.
1 change: 1 addition & 0 deletions locals.tf
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,5 @@ locals {
namespace = var.namespace
terraform = "true"
}
log_groups_to_use = length(var.log_group_prefixes) > 0 ? var.log_group_prefixes : var.cloudwatch_log_groups
}
6 changes: 4 additions & 2 deletions logs_monitoring_cloudwatch_log.tf
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,12 @@ resource "aws_cloudwatch_log_subscription_filter" "test_lambdafunction_logfilter
}

resource "aws_lambda_permission" "allow_cloudwatch_logs_to_call_dd_lambda_handler" {
for_each = { for lg in var.cloudwatch_log_groups : lg => lg }
for_each = { for lg in local.log_groups_to_use : lg => lg }
statement_id = "${substr(replace(each.value, "/", "_"), 0, 67)}-AllowExecutionFromCloudWatchLogs"
action = "lambda:InvokeFunction"
function_name = aws_cloudformation_stack.datadog-forwarder.outputs.DatadogForwarderArn
principal = "logs.${var.aws_region}.amazonaws.com"
source_arn = "arn:aws:logs:${var.aws_region}:${var.aws_account_id}:log-group:${each.value}:*"
source_arn = (length(var.log_group_prefixes) > 0 ?
"arn:aws:logs:${var.aws_region}:${var.aws_account_id}:log-group:${each.value}*" :
"arn:aws:logs:${var.aws_region}:${var.aws_account_id}:log-group:${each.value}:*")
}
5 changes: 5 additions & 0 deletions vars.tf
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,11 @@ variable "cloudwatch_log_groups" {
type = list(string)
default = []
}
variable "log_group_prefixes" {
description = "List of CloudWatch Log Group prefixes to create lambda permissions"
type = list(string)
default = []
}
variable "enable_datadog_aws_integration" {
description = "Use datadog provider to give datadog aws account access to our resources"
type = bool
Expand Down

0 comments on commit 201d6c3

Please sign in to comment.