[APM][Tags] Configure Datadog request resource names

[timmc-edx]

We need to settle on a resource_name format for Python APM. This has implications for trace metrics, browsing the UI, and monitoring.

As of 2024-05-15 we're using a handler-only format but we need to make a final decision.

A/C:

• Final choice decided on and communicated to teams
• Deployment (or notification to teams of how to deploy), including communications warning of the change
  • EC2: feat!: Switch to method+handler Datadog resource name format configuration#9
  • K8s: https://github.com/edx/helm-charts/pull/136
  • Followup once k8s deploys are validated: https://github.com/edx/bootstrap-k8s-app/pull/82
• High-priority ticket created for adding full Django view information as a custom span tag
  • Add viewfunc custom span tag on Django requests #641

Resources

• https://ddtrace.readthedocs.io/en/latest/integrations.html#id45 documents the options.

This is applied to the root span as the resource_name tag. HTTP method and urlpattern are also available on the root span as custom span tags http.method and http.route but there doesn't seem to be any other way to get the handler info.

Pros and cons

• urlpattern format:
  • Available in default config (always accompanied by HTTP method).
  • Can be pretty simple like ^/ (for the home page) or really nasty like ^courses/(?P<course_id>[^/+]+(/|\+)[^/+]+(/|\+)[^/?]+)/xblock/(?P<usage_id>(?:i4x://?[^/]+/[^/]+/[^/]+/[^@]+(?:@[^/]+)?)|(?:[^/]+))/handler/(?P<handler>[^/]*)(?:/(?P<suffix>.*))?$ (xblock_handler).
  • Really bad for querying and for dashboards.
• handler format:
  • Available with or without HTTP method using DD_DJANGO_USE_HANDLER_RESOURCE_FORMAT=true or DD_DJANGO_USE_LEGACY_RESOURCE_FORMAT=true respectively.
  • Can be a dotted module path to a function, e.g. lms.djangoapps.branding.views.index (the view function index in the module lms.djangoapps.branding.views) or to a class, e.g. enterprise.api.v1.views.enterprise_customer:EnterpriseCustomerViewSet.
  • Problem: View classes can have any number of methods, and Datadog doesn't tell us which one was invoked. EnterpriseCustomerViewSet has list, retrieve, dashboard_list, enroll_learners_in_courses, basic_list, toggle_universal_link, and partial_update. These can generally be distinguished by method + urlpattern, but not by handler class alone.
    • While we could make up for this by adding our own custom span tag in a new middleware, it wouldn't be available for certain kinds of monitors.
    • New Relic gives the full name, e.g. WebTransaction/Function/enterprise.api.v1.views.enterprise_customer:EnterpriseCustomerViewSet.enroll_learners_in_courses
    • Private ticket with Datadog regarding lack of complete Django viewfunc information: https://help.datadoghq.com/hc/en-us/requests/1694229
• method in format:
  • Available with either handler or urlpattern.
  • If used with handler, would require parsing to get just the handler out of the resource name.
  • Helps disambiguate views when DD elides the full viewfunc.

Decision

• We will switch to the DD_DJANGO_USE_HANDLER_RESOURCE_FORMAT=true resource name format (method + handler):
  • urlpattern resource names are just too hard to read, query, and pronounce for (for some of our IDAs and views), so we definitely want a handler-based format.
  • The only upside of urlpattern is being able to discriminate between different views in a viewset. We can capture some of that by including method in the resource name; this will help when the view functions are just read, create, delete, etc. It does not help when the view functions are distinct operations such as retirement_queue and cleanup.
  • Teams with more complicated viewsets will be able to use Trace Analytics in their monitors (using @http.route, @django.view, and similar tags) to be more specific than the resource name. They will have less granularity in the built-in dashboards and in certain other features.
• We will provide a new middleware feature for IDAs that will add a tag on Django service entry spans that includes the full view function name. This will make Trace Analytics and dashboard queries more uniform and pleasant to write, rather than relying on http.route and django.view.
  • Issue for this: Add viewfunc custom span tag on Django requests #641
• We will ask Datadog to fill in this missing info via a new flag. If/when they do this, teams will have a choice of whether to adopt it (except for LMS, which would just have this feature enabled, and teams would just have to adjust their boards and runbooks).

[timmc-edx]

We'll use the spreadsheet to ensure deployments happen.

[timmc-edx]

Verified in edxapp-lms and demographics in stage.