[APM][Hosts] Rollout Default APM to remaining IDAs

[timmc-edx]

Goal is to be able to turn on instrumentation for all services at once for some limited time period, and then disable. This is to gather APM host data for the contract.

AC:

• Ensure as many APM hosts as possible are enabled in DD for some agreed upon window (TBD with Shannon).
  • Confirm APM Hosts seen in DD.
• Use New Relic to find data for missing services.
• Leave some documentation trail

Some implementation steps:

• Deploy ddtrace-run to Kubernetes IDAs
• Deploy ddtrace-run to EC2 IDAs
• See additional todo items down below.

Notes:

• Communications:
  • Expect a perf hit
  • Disable pymongo integration
• Goal is to get a coordinated view of all traffic at some point.
• Need a list of services and infrastructure type.
• Docs can go under https://2u-internal.atlassian.net/wiki/spaces/SRE/pages/861372422/New+Relic+-+Datadog+Migration
• Need a planned date for turning on/off.
• We will do what we can for teams on this to keep this unblocked.
• It would be great to know how we want to tag the services appropriately.
• Do we need a separate ticket for Ruby instrumentation? Completed by SRE

Deployment process for Kubernetes web applications:

• Add ddtrace to the internal dockerfiles (https://github.com/edx/internal-dockerfiles/pull/102 and https://github.com/edx/internal-dockerfiles/pull/103)
  • pip install ddtrace
  • Ensure Dockerfile does USER root before and USER app afterwards, at least if the base Dockerfile is using USER app. Otherwise there will be a permissions error. (The existing pip install newrelic only "works" because newrelic is already installed.)
• Update service's Helm chart to use django-ida version 0.8.27 or higher and set datadog.enabled=true in the configs, or 0.9.0 to enable by default.

Additional Notes:

• See and update Tracking spreadsheet
• Use New Relic list of services to check if migration spreadsheet is complete: See https://onenr.io/0dQeorq8NRe
• Here is a link for NR hosts per service: https://onenr.io/0BR6Z5bg1RO. You may use this to help prioritize/filter services with low host counts.
• Find Tim’s past Slack threads communicating changes for example communications, including potential impact to performance, etc.

Containers

• [Diana] Confirm helm version upgrade status.
• Confirm auto deployment to Stage, and whether we see data in DD.
• Add DD links to spreadsheet. Do we need a different link per environment?
• Coordinate with owners for GoCD deployment to Prod per service.
• Crons and workers? Where did this go and how far did we get? See Tim’s Slack messages like this one.
• Ensure Kafka consumer workers have APM. (Note that Kafka Spans will be ticketed separately. This is just about APM Host counts.)

EC2

• Check status of all ec2 services
  • Follow-up with Nadeem on this Slack thread. Was Nadeem able to complete this?
• Confirm DD and get links in spreadsheet per service.

Skipped hosts:

• For services we are skipping, find relative usage from New Relic and add to spreadsheet.
• Mark in spreadsheet what will not be enabled

Notes for Skipped Hosts:

• New Relic hosts by hour for all services: https://onenr.io/08woe7Zvpjx
• New Relic hosts by hour for example specific services:
• Faceted to check work: https://onenr.io/0Zw0mgN8ORv
• Unfaceted: https://onenr.io/0LREVYlgoja
• Note: DD needs 99th percentile maximum hour, which is 8th hour of month, 4th for half month, etc.

[robrap]

Done as much as needed for this ticket. Thanks team!