feat: add observability for lambda-to-lambda invocations

[fdaciuk]

This Pull Request enables the visualization of Lambda-to-Lambda invocations in the Service Map of Elastic Search and provides detailed insights into the "spans" and "transactions" used by the internal Lambda function. The way we use to identify Lambda-to-Lambda invocations is by checking for the presence of the traceparent and tracestate in the invocation payload.

I welcome any insights or recommendations to make it even more robust and user-friendly.

Checklist

• Implement code
• Add tests
• Update TypeScript typings
• Update documentation
• Add CHANGELOG.asciidoc entry
• Commit message follows commit guidelines

[cla-checker-service]

💚 CLA has been signed

[trentm]

@fdaciuk Thanks for the PR.

Can you show some simple Lambda function code that uses this? That would help me understand a bit better.
Does your code that is doing the Lambda Invoke API call (https://docs.aws.amazon.com/lambda/latest/dg/API_Invoke.html) manually pass in traceparent and tracestate in the payload?

[fdaciuk]

Hi @trentm. Exactly! I don't know if there is better way to do this, but that update worked for my use case.

I'm invoking the lambda like this:

  const command = new InvokeCommand({
    FunctionName: "my-lambda",
    InvocationType: "RequestResponse",
    Payload: new TextEncoder().encode(JSON.stringify({ traceparent, tracestate })),
  })

[trentm]

There isn't a current way to do this ... except possibly Amazon's X-Ray support might enable making the connection. We aren't currently doing anything with X-Ray in our Lambda support.

Providing some way for this manual distributed tracing in Lambda is interesting. I like the idea.

I wonder about instead using ClientContext in the InvokeCommand call. Theoretically, if the APM agent automatically instrumented @aws-sdk/client-lambda -- which I'm guessing you are using here -- then we could (perhaps optionally) support automatically adding the trace-context info (traceparent et al) to the ClientContext or Payload.

I'm not sure if ClientContext and Payload, IIUC, are unstructured. Or does the provided ClientContext need to be a JSON object that can be added to the context object passed to the Lambda function? If these are unstructured, in general, then it would be a hard sell to have a default feature added to the APM agent that tried to impose some structure on it.

[fdaciuk]

I'm not sure if ClientContext and Payload, IIUC, are unstructured. Or does the provided ClientContext need to be a JSON object that can be added to the context object passed to the Lambda function?

That's a good question. I'm not sure, but love the idea that this might be automatic. We're creating a library on our company to abstract the use of InvokeCommand and inject traceparent and tracestate into Payload, so if we had an option to do this from elastic-apm-node, even manually, would be great =)

[trentm]

Just a heads up that I won't be able to get to this soon. :/

[fdaciuk]

No problem. For now, we're using a forked version of this repo with these updates. I hope it's not a problem until you decide whether or not it's a valuable feature in this lib =)

[trentm]

Was this change to avoid the extra span necessary for the Service Map connection to work? Or was this just to avoid having an addition span in the trace waterfall?

[fdaciuk]

Both I guess. If this span is created when a lambda is invoked, I can see two entries on Service Map: one empty (created by this span) and other with the full tracing (created from the code send in lib/lambda.js.

This code just removes the empty one.

[trentm]

cla/check