Discrepancies between logged requests by the Kong and metrics in the Datadog

[piotr-maslanka]

Hello.
Recently we set up Datadog plugin and noticed some discrepancies between Kong requests logs and Datadog metrics provided by it.

Data gathered in our logs:

Data from DD metrics:

As you can see we have two major problems:

• some types of requests are not available in the metrics at all (fe. ones with status 201, 308 etc.)
• number of requests in the logs and in the metrics are not the same

Our Kong version is 3.2 (upgraded from 3.1 to mitigate this issue: #10044)
Metrics plugin definition that we use:

apiVersion: configuration.konghq.com/v1
kind: KongClusterPlugin
metadata:
 name: datadog-plugin
 annotations:
  kubernetes.io/ingress.class: kong
 labels:
  global: "true"
config:
 host: localhost
 port: 8125
 flush_timeout: 2
 retry_count: 10
 prefix: kong
 metrics:
  - name: request_count
    stat_type: meter
    consumer_identifier: username
plugin: datadog

We already changed stat_type from counter to meter since it seems to better represent the actual number of requests after sending metrics to DD (counter was automatically converted to rate).

Additionally, we also use these plugins: jwt, request-transformer, response-transformer, our own custom plugin.

Is it some kind of bug, misconfiguration on our side or maybe there is some kind of logic regarding which requests should be provided in the metrics (and which shouldn't)?

[StarlightIbuki]

Given this is an old ticket, please ignore me if it's already fixed for you.

I noticed that the missing statuses are:
201 308 400 414
Which are somehow special statuses or with small counts;
and meanwhile, 502 with count 0 shows up in the DD metrics.

The DD logs look like a fraction of Kong logs.

Could you share more details, like what period both logs are counted? DD metrics could be filtered and scoped and maybe it's just about how we operate the DD dashboard.