Add a metrics endpoint for Prometheus

[nolwennz]

Describe the feature or potential improvement

Expose an endpoint with technical metrics (response time, number of requests..?) to monitor Langfuse deployment

Additional information

Discussed on Discord :)

[marcklingen]

usually AWS/GCP stats could be helpful already for these high-level metrics (# requests + latencies)

not sure if building an internal OSS observability logic which exposes metrics is worth the effort. Maybe migrating to OTel and allowing to add a custom collector is a good approach here. No timeline on this though as it'd require to change the observability stack used to operate Langfuse.

can you link to discord thread here?

[mindsdb-devops]

What I would like in a metrics endpoint is not for debugging langfuse (which is what OTEL would typically be for) but would be for usage-related metrics. For example:

• Evals metrics namely success metrics (which you call "scores") so that we can actively monitor the success ratio and quality of our service offering, and make alerts if the threshold drops below a certain value (in Prometheus). These should properly have all the labels (Prometheus Labels) needed to ensure selecting just a specific eval or model etc is possible, so we can choose our own aggregation(s) via queries.
• Daily metrics type of data. Although with Prometheus, "daily" doesn't quite make sense, your time-frame of metrics could be on a per-minute increment of time all captured into Prometheus so we can monitor the usage and cost of our system. A prometheus query could then later aggregate an entire day's worth of data if someone wanted to get a comparable daily overview via Prometheus.
• etc

Overall, what we're looking for is to take the data collected by LangFuse and to "action" and review it in a metrics and observability system. I am not sure that it adds tons of value for LangFuse to invent their own metrics/graphing/alerting stack, but simply exposing various datum to Prometheus allows for medium to large self-hosted installations of LangFuse to capture, alert, and action on these metrics.

PS. For what the original author of this is asking for, I do recommend separately to simply implement OTEL and provide some env vars for configuring OTEL to send traces/metrics to. I personally believe the Prometheus metrics should be things that you can't/don't typically capture via OTEL, for custom metrics. These can be different techs means for different dedicated purposes. Prom for precision usage metrics (aka, data inside/about the traces) for actioning usage data (via Prometheus alerts) and OTEL for debugging and optimizing LangFuse (aka, data about how LangFuse is running).

[marcklingen]

thank you for sharing these detailed notes, super helpful!