Add support for collecting and sending function logs to APM Server

[lahsivjar]

Adds support for collecting function logs using AWS Logs API and shipping it to APM-Server with collected agent metadata.

Closes: #256

[apmmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-09-19T03:26:40.652+0000

• Duration: 5 min 14 sec

Test stats 🧪

Test	Results
Failed	0
Passed	164
Skipped	2
Total	166

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[lahsivjar]

/test

[axw]

LGTM, just a few nitpicks. Nice work!

[lahsivjar]

@axw Apologies for the force push 🙇

This is the commit added for addressing the review comments.

[axw]

Looks great, thanks for the updates :)

[kruskall]

Tested with BC2 (and 8.6.0-SNAPSHOT) by provisioning the environment with the terraform script (SKIP_DESTROY=1 has been used to inspect the cluster) with TF_VAR_ess_version=8.5.0-SNAPSHOT.

Verified that the logs are visible in the logs tab in the Observability section.

Made sure that ELASTIC_APM_LAMBDA_CAPTURE_LOGS=false disables logs collection.

Possible concern: #322
I don't think the above issue is a blocker according to the definition of a blocker so I'm going to label this PR as test-plan-ok

[RyanHealey1307]

I'm aware this is still experimental, but some feedback...
I've tested this for a python lambda, using the latest APM layer, and when you invocate the function, the logs don't go through to APM until the next invocation. Eg, if my lambda is run at 10:09 and then again at 10:13, the logs from 10:09 are not visible until next next invocation at 10:13. My timeout is set to 5 mins and it's a very simple function, so I can't be a timeout issue.