Layer instrumentation with Boto3/Botocore >1.36 breaking

[tombarton]

Recently updated Boto3/botocore to use 1.36 whilst instrumenting the Lambda using the Serverless framework.

Expected Behavior

Lambda boots as expected with DataDog instrumentation enabled.

Actual Behavior

Lambda immediately errors with the following error:

[ERROR] Runtime.ImportModuleError: Unable to import module 'datadog_lambda.handler': cannot import name 'DEFAULT_CHECKSUM_ALGORITHM' from 'botocore.httpchecksum' (/var/lang/lib/python3.11/site-packages/botocore/httpchecksum.py)

Mismatched versions of Botocore cause this: boto/boto3#4399

Steps to Reproduce the Problem

1. Instrument basic Python Lambda using Serverless framework and install Boto3 at 1.36.0 or higher.
2. Execute Lambda

Specifications

• Datadog Lambda Layer version: 104
• Python version: 3.11

Stacktrace

[ERROR] Runtime.ImportModuleError: Unable to import module 'datadog_lambda.handler': cannot import name 'DEFAULT_CHECKSUM_ALGORITHM' from 'botocore.httpchecksum' (/var/lang/lib/python3.11/site-packages/botocore/httpchecksum.py)

[purple4reina]

Hey @tombarton thanks for reaching out. Interestingly, we don't actually package boto3 or botocore in the Datadog Python Lambda Layer since it's already available in the runtime. Can you test this again without the Datadog layer and see if the import error still happens?

[tombarton]

Hey @purple4reina, I saw that earlier which confused things somewhat. The Lambda works perfectly with the DD layers disabled.

[tombarton]

I've tried to re-create a minimum but have sadly failed. At this point, I can only presume it must be something to do with cross dependencies in our repository. As soon as I bump boto3 from 1.35.99 to 1.36.0, I get the above error with the DataDog layers enabled. If I disable the DataDog layers, everything works perfectly. If I downgrade Boto3 to ~1.35, I can enable the DataDog layers.

The issue is due to the mismatch of botocore versions. If I change the Lambda handler to use the DataDog Lambda wrapper, it returns a Botocore version of 1.34.145, followed by the subsequent error:

TELEMETRY	Name: datadog-agent	State: Subscribed	Types: [Extension, Function, Platform]
botocore.__version__ 1.34.145
boto3.__version__ 1.36.0
EXTENSION	Name: datadog-agent	State: Ready	Events: [SHUTDOWN, INVOKE]
INIT_REPORT Init Duration: 10008.04 ms	Phase: init	Status: timeout
TELEMETRY	Name: datadog-agent	State: Already subscribed	Types: [Extension, Function, Platform]
botocore.__version__ 1.34.145
boto3.__version__ 1.36.0
[ERROR] Runtime.ImportModuleError: Unable to import module 'datadog_lambda.handler': cannot import name 'DEFAULT_CHECKSUM_ALGORITHM' from 'botocore.httpchecksum' (/var/lang/lib/python3.11/site-packages/botocore/httpchecksum.py)

If I keep the layers enabled but remove the DataDog wrapper, the botocore version logged differs:

TELEMETRY	Name: datadog-agent	State: Subscribed	Types: [Extension, Function, Platform]
botocore.__version__ 1.36.9
boto3.__version__ 1.36.0
EXTENSION	Name: datadog-agent	State: Ready	Events: [SHUTDOWN, INVOKE]
INIT_REPORT Init Duration: 10006.75 ms	Phase: init	Status: timeout
TELEMETRY	Name: datadog-agent	State: Already subscribed	Types: [Extension, Function, Platform]
botocore.__version__ 1.36.9
boto3.__version__ 1.36.0
EXTENSION	Name: datadog-agent	State: Ready	Events: [INVOKE, SHUTDOWN]
START RequestId: b433580a-7d9d-40c0-82c4-53fb868644c3 Version: $LATEST

[tombarton]

OK, I've been able to re-create this issue now in a tiny repository. DataDog Data Streams is the cause of the error. When you toggle this on with the environment variable (DD_DATA_STREAMS_ENABLED), the botocore version is fixed to 1.34.145, overriding what is set. Once disabled, botocore lists the correct version.

@purple4reina is this something you can help with or should it be addressed in a different repository?

Repository: https://github.com/tombarton/datadog-python-boto3-layer

DD_DATA_STREAMS_ENABLED enabled:

INIT_START Runtime Version: python:3.11.v46	Runtime Version ARN: arn:aws:lambda:us-east-1::runtime:84bc5fe9641102b252ebbfd80d16f2791f7bcca59e26ce68d95066317adf4503
TELEMETRY	Name: datadog-agent	State: Subscribed	Types: [Extension, Function, Platform]
botocore.__version__ 1.34.145
boto3.__version__ 1.36.9
EXTENSION	Name: datadog-agent	State: Ready	Events: [INVOKE, SHUTDOWN]
[ERROR] ImportError: cannot import name 'DEFAULT_CHECKSUM_ALGORITHM' from 'botocore.httpchecksum' (/var/lang/lib/python3.11/site-packages/botocore/httpchecksum.py)
Traceback (most recent call last):
  File "./python/lib/python3.11/site-packages/datadog_lambda/wrapper.py", line 242, in __call__
  File "/var/task/handler.py", line 16, in hello
    client = boto3.client("s3")
  File "./python/lib/python3.11/site-packages/ddtrace/contrib/internal/aws_lambda/patch.py", line 119, in __call__
  File "/var/task/handler.py", line 16, in hello
    client = boto3.client("s3")
  File "/tmp/sls-py-req/boto3/__init__.py", line 92, in client
    return _get_default_session().client(*args, **kwargs)
  File "/tmp/sls-py-req/boto3/session.py", line 297, in client
    return self._session.create_client(
  File "/var/lang/lib/python3.11/site-packages/botocore/session.py", line 997, in create_client
    client = client_creator.create_client(
  File "/var/lang/lib/python3.11/site-packages/botocore/client.py", line 147, in create_client
    cls = self._create_client_class(service_name, service_model)
  File "/var/lang/lib/python3.11/site-packages/botocore/client.py", line 201, in _create_client_class
    self._event_emitter.emit(
  File "/var/lang/lib/python3.11/site-packages/botocore/hooks.py", line 412, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/var/lang/lib/python3.11/site-packages/botocore/hooks.py", line 256, in emit
    return self._emit(event_name, kwargs)
  File "/var/lang/lib/python3.11/site-packages/botocore/hooks.py", line 239, in _emit
    response = handler(**kwargs)
  File "/tmp/sls-py-req/boto3/utils.py", line 65, in _handler
    module = import_module(module)
  File "/tmp/sls-py-req/boto3/utils.py", line 56, in import_module
    __import__(name)
  File "./python/lib/python3.11/site-packages/ddtrace/internal/module.py", line 295, in _exec_module
  File "./python/lib/python3.11/site-packages/datadog_lambda/cold_start.py", line 119, in wrapped_method
  File "/tmp/sls-py-req/boto3/s3/inject.py", line 18, in <module>
    from boto3.s3.transfer import (
  File "./python/lib/python3.11/site-packages/ddtrace/internal/module.py", line 295, in _exec_module
  File "./python/lib/python3.11/site-packages/datadog_lambda/cold_start.py", line 119, in wrapped_method
  File "/tmp/sls-py-req/boto3/s3/transfer.py", line 135, in <module>
    from s3transfer.futures import NonThreadedExecutor
  File "./python/lib/python3.11/site-packages/ddtrace/internal/module.py", line 295, in _exec_module
  File "./python/lib/python3.11/site-packages/datadog_lambda/cold_start.py", line 119, in wrapped_method
  File "/tmp/sls-py-req/s3transfer/futures.py", line 22, in <module>
    from s3transfer.utils import FunctionContainer, TaskSemaphore
  File "./python/lib/python3.11/site-packages/ddtrace/internal/module.py", line 295, in _exec_module
  File "./python/lib/python3.11/site-packages/datadog_lambda/cold_start.py", line 119, in wrapped_method
  File "/tmp/sls-py-req/s3transfer/utils.py", line 29, in <module>
    from botocore.httpchecksum import DEFAULT_CHECKSUM_ALGORITHM, AwsChunkedWrapper

DD_DATA_STREAMS_ENABLED disabled:

INIT_START Runtime Version: python:3.11.v46	Runtime Version ARN: arn:aws:lambda:us-east-1::runtime:84bc5fe9641102b252ebbfd80d16f2791f7bcca59e26ce68d95066317adf4503
TELEMETRY	Name: datadog-agent	State: Subscribed	Types: [Extension, Function, Platform]
botocore.__version__ 1.36.9
boto3.__version__ 1.36.9
EXTENSION	Name: datadog-agent	State: Ready	Events: [INVOKE, SHUTDOWN]
START RequestId: 6afa9b39-0d8e-4640-be5f-8d782a3121c2 Version: $LATES