describe_instances not returning latest state of ec2-instance

[jaimin-sl]

Describe the bug

I have a event rule which calls a lambda when a ec2-instance's state is changed. Rule's instance id list is configured such that it only has on demand instances. Now when I stop/run my instance it changes states as follows
stopped -> pending -> running
running -> stopping -> stopped

In each transition my lambda get's called. Sometime when I do describe_instances(instance_id) (instance_id is provided by event rule) it returns old state of ec2-instance.

For example if I have ran my ec2-instance, lambda should log pending and running in two consecutive logs.
Instead sometimes it logs stopped and pending.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

lambda should log current state of ec2-instance

Current Behavior

Lambda is logging old state of ec2-instance

Reproduction Steps

I am not sure how to reproduce this as sometime this happens and sometimes not,
but here is the minimal code I am using

def lambda_handler(event, context):
    if event['source'] != "aws.ec2":
        print(f"ERROR: Got triggered by {event['Source']}!!")
        return

    instance_id = event['detail']['instance-id']

    #instance data
    instance = ec2_client.describe_instances(InstanceIds=[instance_id])

    if len(instance["Reservations"][0]["Instances"]) != 1:
        print(f'Instance list length is invalid, Length: {len(instance["Reservations"][0]["Instances"])}')
        return

    instance_status = instance["Reservations"][0]["Instances"][0]["State"]["Name"]
    print(f"For {instance_id} state is {instance_status}")

Possible Solution

No response

Additional Information/Context

No response

SDK version used

Python: python3.10, boto3-1.35.95

Environment details (OS name and version, etc.)

Amazon Linux 2023

[RyanFitzSimmonsAK]

Hi @jaimin-sl, thanks for reaching out. I think this is a result of EC2 being eventually consistent. I'd recommend following the guidance in the linked document, and continuing to describe the instance until it reaches the state you're expecting. Please let me know if you have any follow-up questions.

[jaimin-sl]

Thanks for the help. Eventually I did above. I thought may be there is some other way then running a busy loop on describe_instances.