feat: add smoke testing

[kruskall]

Add a terraform script and an example js function.
The function is pushed to aws with the following layers:

• apm js agent
• locally built lambda extension

Log level and aws region can be configured using variables.

Closes #217

[kruskall]

First iteration, ready for review!

Feedback or tip on how to integrate with CI welcomed 🙇

[apmmachine]

💔 Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-09-14T23:31:53.158+0000

• Duration: 7 min 41 sec

Steps errors

Expand to view the steps failures

make build

• Took 0 min 6 sec . View more details here
• Description: make build

make build

• Took 0 min 1 sec . View more details here
• Description: make build

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[axw]

This is a great start. Before closing #217 (the description says this PR closes it), we should make invocations to the function and check the results in Elasticsearch.

In standup I mentioned that I think we should use an ephemeral deployment for this. Perhaps for that we can reuse some of the Terraform modules defined in the apm-server repo, like this:

module "ec_deployment" {
  source = "github.com/elastic/apm-server//testing/infra/terraform/modules/ec_deployment?depth=1"
  deployment_name_prefix = "apm-aws-lambda"
  integrations_server = true
  apm_server_expvar = false
  apm_server_pprof = false
}

(@marclop WDYT about other projects using the apm-server modules like that?)

We could then compose that and the module you have defined to orchestrate the creation of the ESS deployment and Lambda. After provisioning we would run a script that sends requests to the Lambda URL, and then poll Elasticsearch for results.

I've been intending to propose that we take this approach for the agents too, and stop using apm-integration-testing. CC @felixbarny @cachedout

[kruskall]

In standup I mentioned that I think we should use an ephemeral deployment for this. Perhaps for that we can reuse some of the Terraform modules defined in the apm-server repo, like this:

module "ec_deployment" {
source = "github.com/elastic/apm-server//testing/infra/terraform/modules/ec_deployment?depth=1"
deployment_name_prefix = "apm-aws-lambda"
integrations_server = true
apm_server_expvar = false
apm_server_pprof = false
}

We could then compose that and the module you have defined to orchestrate the creation of the ESS deployment and Lambda. After provisioning we would run a script that sends requests to the Lambda URL, and then poll Elasticsearch for results.

Could you clarify this ? Looking at the module in the apm-server repository I don't think we will end up using much of the available inputs. Is there a specific feature we're planning to use ?

[axw]

Could you clarify this ? Looking at the module in the apm-server repository I don't think we will end up using much of the available inputs. Is there a specific feature we're planning to use ?

My initial thought was also to use the ec provider directly, which I assume is what you're thinking here. The main thing we would get from using ec_deployment from the apm-server repo is extraction of the generated secret token from the APM integration policy.

[axw]

Looking good @kruskall, just a few changes I'd like to see.

In addition to the comments I left, can we:

• add .gitignore entries for terraform state?
• add outputs for the Lambda function URL, and ESS deployment details (Elasticsearch + Kibana URLs, username and password)?

Perhaps we should default the AWS region to gcp-us-west-2, since it's where we do the rest of our cloud-based testing. Although it's crossing regions, the amount of traffic will be minimal - and running the deployment in the dedicated CFT us-west-2 region means we won't be contending with customers for ESS resources.

How will we invoke the Lambda function from some test script after provisioning? Should we be configuring a function URL, and emitting it in output?

[axw]

Looking good, thanks for the updates @kruskall.

I think we should change the default ESS region, but let me know if you have any concerns about that. Ideally I think we would use the defaults for ess_region and deployment_template as defined in ec_deployment, to keep things defined in one place.

In order to close #217 we will need to actually run some invocations and verify trace events end up in Elasticsearch as expected. I suggest we get this PR finished up and follow up with a test script that does those bits. Can you please remove the "Closes" from the description?

[kruskall]

Ok, I added the script to verify the traces are in Elasticsearch, I think this is ready for another review

[axw]

Thanks, this is great. LGTM with the region explicitly passed to the AWS CLI.

[marclop]

oops, my review didn't make it pre-merge, I guess we can see if it's worth as a follow up

[marclop]

By default, we'll end up using the latest SNAPSHOT versions, they may be broken and cause the smoke tests to fail. If we want to avoid that, then we need to provide an expression for a released version, such as:

Suggested change

	deployment_template = var.ess_deployment_template
	deployment_template = var.ess_deployment_template
	stack_version = "\\d.\\d.\\d?(\\d)$" // use the latest released version.

[axw]

++ I've opened #307 to track this