[ Bug ?] Cloudwatch put_dashboard() can't update existing dashboard with new data

[fhmlsml]

Hello,

I am experiencing an issue with a Boto script that I am using to update metrics in a dashboard. Specifically, the script is not updating the metrics (widget) automatically when a new instance appears.

The script is supposed to automatically detect when a new instance appears and add its metrics to the existing dashboard. However, this is not happening, and I have to manually add the new instance's metrics to the dashboard (*edit the dashboard source json editor and add the output json text from my code manually to it.

Dashboard source

{
    "widgets": []
}

Here's my code, i followed this tutorial and adjust the script for my needs
https://aws.amazon.com/blogs/mt/update-your-amazon-cloudwatch-dashboards-automatically-using-amazon-eventbridge-and-aws-lambda/

# in this scenario, currently i have 1 instance in `My-Auto-Scaling-Group` and change the desired auto scale to 3 to trigger this function and below are the results

import copy
import datetime
import json
import os
import boto3


STATIC_WIDGET_TEMPLATE = {
    "widgets": [
        {
            "height": 7,
            "width": 8,
            "y": 0,
            "x": 0,
            "type": "metric",
            "properties": {
                "view": "timeSeries",
                "stacked": False,
                "metrics": [], # <- this will be updated with list of instances for cpu metric
                "region": "ap-southeast-3",
                "title": "CPU-USAGE"
            },
        },
        {
            "type": "metric",
            "x": 8,
            "y": 0,
            "width": 9,
            "height": 7,
            "properties": {
                "view": "timeSeries",
                "stacked": False,
                "metrics": [
                    [] # <- this will be updated with list of instances for memory metric
                ],
                "region": "ap-southeast-3",
                "title": "MEMORY-USAGE"
            },
        },
        {
            "type": "metric",
            "x": 17,
            "y": 0,
            "width": 7,
            "height": 7,
            "properties": {
                "view": "timeSeries",
                "stacked": False,
                "metrics": [], # <- this will be updated with list of instances for disk usage metric
                "region": "ap-southeast-3",
                "title": "DISK-USAGE"
            },
        },
    ]
}


class CloudWatch:
    """Base class for updated dashboard widgets"""
    def __init__(self, dashboard_name):
        """Name of dashboard to (re)create,  list of account numbers to interrogate"""
        self.cw_client = boto3.client('cloudwatch')
        self.ec2_client = boto3.client('ec2')
        self.dashboard_name = dashboard_name
        self.dashboard = None

    def get_ec2_list_in_asg(self):
        """Returns the EC2 CPU metrics for a single account and region"""
        hasil_data = {
            "instances_id": [],
            "image_id": [],
            "instance_type": []
        }
        # CloudWatch metrics can only display max 500 metrics per widget
        results = self.ec2_client.describe_instances(
            Filters=[
                {
                    'Name': 'tag-key',
                    'Values': [
                        'Environment',
                    ]
                },
            ],
            MaxResults=500
        )
        for result in results['Reservations']: 
            for instance in result['Instances']:
                for tag in instance['Tags']:
                    if instance['State']['Code'] != 48 and (tag['Key'] == "Environment" and tag['Value'] == "Production"):
                        print(f"adding instance ID, image ID, and instance type : {instance['InstanceId']} {instance['ImageId']} and {instance['InstanceType']}")
                        hasil_data["instances_id"].append(instance['InstanceId'])
                        hasil_data["image_id"].append(instance['ImageId'])
                        hasil_data["instance_type"].append(instance['InstanceType'])
                    else:
                        print('skipping terminated instance ID: {}'.format(instance['InstanceId']))
        return hasil_data
        # in this scenario, currently i have 1 instance and change the desired auto scale to 3

    def put_dashboard(self, dashboard_body):
        """Puts the updated dashboard into CloudWatch"""
        results = self.cw_client.put_dashboard(
            DashboardName=self.dashboard_name,
            DashboardBody=json.dumps(dashboard_body))
        print(results)
        # success but not updated
        # {'DashboardValidationMessages': [], 'ResponseMetadata': {'RequestId': 'e9c4aa36-87cf-40a5-8e6d-4a6bfc810277', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': 'e9c4aa36-87cf-40a5-8e6d-4a6bfc810277', 'content-type': 'text/xml', 'content-length': '292', 'date': 'Tue, 07 Mar 2023 03:43:27 GMT'}, 'RetryAttempts': 0}}



def format_widget_cpu(hasil_data):
    """
    Returns a JSON object with the widget definition for a single account and
    its EC2 instances
    """
    results = []
    for instance in hasil_data['instances_id']:
        results.append(['AWS/EC2', 'CPUUtilization', 'InstanceId', instance, { "label": instance }])
    return results

def format_widget_cwagent_memory(hasil_data):
    results = []
    instance_id = [ instance_id for instance_id in hasil_data['instances_id'] ]
    image_id = [ image_id for image_id in hasil_data['image_id'] ]
    instance_type = [ instance_type for instance_type in hasil_data['instance_type'] ]

    for instance_id, image_id, instance_type in zip(instance_id, image_id, instance_type):
        results.append([ "CWAgent", "mem_used_percent", "InstanceId", instance_id, "AutoScalingGroupName", "My-Auto-Scaling-Group", "ImageId", image_id, "InstanceType", instance_type, { "label": instance_id } ])
    return results

def format_widget_cwagent_disk(hasil_data):
    results = []
    instance_id = [ instance_id for instance_id in hasil_data['instances_id'] ]
    image_id = [ image_id for image_id in hasil_data['image_id'] ]
    instance_type = [ instance_type for instance_type in hasil_data['instance_type'] ]

    for instance_id, image_id, instance_type in zip(instance_id, image_id, instance_type):
        results.append([ "CWAgent", "disk_used_percent", "path", "/", "InstanceId", instance_id, "AutoScalingGroupName", "My-Auto-Scaling-Group", "ImageId", image_id, "InstanceType", instance_type, "device", "nvme0n1p1", "fstype", "ext4", { "label": instance_id } ])
        
    return results

def lambda_handler(event, context):
    """Entrypoint for the Lambda function"""
    cw = CloudWatch('my-existing-dashboard')
    all_instances = cw.get_ec2_list_in_asg() 
    cpu_metric = format_widget_cpu(all_instances) 
    memory_metric = format_widget_cwagent_memory(all_instances) 
    disk_metric =  format_widget_cwagent_disk(all_instances
    
    print('CPU METRIC ->>', cpu_metric)
    # 3 data in the list, because of 3 instances
    # CPU METRIC ->> [['AWS/EC2', 'CPUUtilization', 'InstanceId', 'i-0bbbac412b570fe97', {'label': 'i-0bbbac412b570fe97'}], ['AWS/EC2', 'CPUUtilization', 'InstanceId', 'i-086cf0da9f8cc8adf', {'label': 'i-086cf0da9f8cc8adf'}], ['AWS/EC2', 'CPUUtilization', 'InstanceId', 'i-0b656d2ffd94573e0', {'label': 'i-0b656d2ffd94573e0'}]]

    print('MEMORY METRIC ->>', memory_metric)
    # 3 data in the list, because of 3 instances
    # MEMORY METRIC ->> [['CWAgent', 'mem_used_percent', 'InstanceId', 'i-0bbbac412b570fe97', 'AutoScalingGroupName', 'My-Auto-Scaling-Group', 'ImageId', 'ami-05d916989a5f3a6ec', 'InstanceType', 't3.micro', {'label': 'i-0bbbac412b570fe97'}], ['CWAgent', 'mem_used_percent', 'InstanceId', 'i-086cf0da9f8cc8adf', 'AutoScalingGroupName', 'My-Auto-Scaling-Group', 'ImageId', 'ami-05d916989a5f3a6ec', 'InstanceType', 't3.micro', {'label': 'i-086cf0da9f8cc8adf'}], ['CWAgent', 'mem_used_percent', 'InstanceId', 'i-0b656d2ffd94573e0', 'AutoScalingGroupName', 'My-Auto-Scaling-Group', 'ImageId', 'ami-05d916989a5f3a6ec', 'InstanceType', 't3.micro', {'label': 'i-0b656d2ffd94573e0'}]]

    print('DISK METRIC ->>', disk_metric)
    # 3 data in the list, because of 3 instances
    # DISK METRIC ->> [['CWAgent', 'disk_used_percent', 'path', '/', 'InstanceId', 'i-0bbbac412b570fe97', 'AutoScalingGroupName', 'My-Auto-Scaling-Group', 'ImageId', 'ami-05d916989a5f3a6ec', 'InstanceType', 't3.micro', 'device', 'nvme0n1p1', 'fstype', 'ext4', {'label': 'i-0bbbac412b570fe97'}], ['CWAgent', 'disk_used_percent', 'path', '/', 'InstanceId', 'i-086cf0da9f8cc8adf', 'AutoScalingGroupName', 'My-Auto-Scaling-Group', 'ImageId', 'ami-05d916989a5f3a6ec', 'InstanceType', 't3.micro', 'device', 'nvme0n1p1', 'fstype', 'ext4', {'label': 'i-086cf0da9f8cc8adf'}], ['CWAgent', 'disk_used_percent', 'path', '/', 'InstanceId', 'i-0b656d2ffd94573e0', 'AutoScalingGroupName', 'My-Auto-Scaling-Group', 'ImageId', 'ami-05d916989a5f3a6ec', 'InstanceType', 't3.micro', 'device', 'nvme0n1p1', 'fstype', 'ext4', {'label': 'i-0b656d2ffd94573e0'}]]


    # insert these data to static widget
    all_widgets = copy.deepcopy(STATIC_WIDGET_TEMPLATE)
    all_widgets['widgets'][0]['properties']['metrics'] = cpu_metric
    all_widgets['widgets'][1]['properties']['metrics'] = memory_metric
    all_widgets['widgets'][2]['properties']['metrics'] = disk_metric

    print('all widgets')
    print(all_widgets)

    cw.put_dashboard(all_widgets)

However, If i copy the result from all_widgets and updated it manually in dashboard source cloudwatch console, it works

is this kind of bug or something ? thanks

[tim-finnigan]

Hi @fhmlsml thanks for reaching out. I don't know what the issue might be here — do you get an error when running your code? If you view the logs (via adding boto3.set_stream_logger('') to your script) does that provide more information? I'm not sure if you have any updates since opening this discussion but I think we'll just need to narrow down where the issue could be occurring.

[github-actions[bot]]

Hello! Reopening this discussion to make it searchable.

[LordHumungous]

This is also an issue for me. Logging the boto3 interaction, as stated above, shows no issues. The response comes back with a 200 and no validation issues. Cloudtrail shows no validation issues with the requests to the API. I can clearly see the payload sending in new metrics inside the widgets but AWS will not update the dashboard.

[LordHumungous]

So this morning I made an unexpected discovery. For my problem at least, the issue was you have to do a full page refresh to show the updated dashboard in the AWS Console. You can't use the widget refresh button by the timeframe filter. I was thinking, I guess incorrectly, that by hitting the widget refresh button it would pull in the new metrics. It does not. I have to do a full page triggered by the browser's refresh button for AWS to update all the widgets. Once I figured that out, the updates appeared and were showing their data correctly.