Leveraging Azure OpenAI Diagnostic Settings?

[Manouchehri]

Azure OpenAI has this very cool (and terribly named) feature called diagnostic setting that logs stats/metrics and events. It'd be neat if Langfuse supported creating a project (and/or dashboard) that allowed feeding in this data.

When allLogs is enabled, you get logs that show you the durationMs that Microsoft themselves measured. (The correlationId is the same as the apim-request-id response header.) I believe you'll also see content violations here, though I didn't test it.

{ "callerIpAddress": "172.70.34.***", "category": "RequestResponse", "correlationId": "2976f95a-1f61-42bf-81cc-346eab82305f", "durationMs": 64, "event": "ShoeboxCallResult", "location": "southindia", "operationName": "ChatCompletions_Create", "properties": "{\"apiName\":\"Azure OpenAI API version 2023-12-01-preview\",\"requestTime\":638422766685683999,\"requestLength\":245,\"responseTime\":638422766686328681,\"responseLength\":794,\"objectId\":\"\",\"streamType\":\"Non-Streaming\",\"modelDeploymentName\":\"gpt-35-turbo\",\"modelName\":\"gpt-35-turbo\",\"modelVersion\":\"1106\"}", "resourceId": "/SUBSCRIPTIONS/00000000-11111-2222-333333333333/RESOURCEGROUPS/OPENAI-GROUP/PROVIDERS/MICROSOFT.COGNITIVESERVICES/ACCOUNTS/DEMO-EXAMPLE", "resultSignature": "200", "time": "2024-01-31T05:44:29.9950000Z", "Tenant": "southindia"}
{ "callerIpAddress": "172.70.135.**", "category": "RequestResponse", "correlationId": "c2458ed3-fdc7-4532-8009-309e5b65d11a", "durationMs": 74, "event": "ShoeboxCallResult", "location": "southindia", "operationName": "ChatCompletions_Create", "properties": "{\"apiName\":\"Azure OpenAI API version 2023-12-01-preview\",\"requestTime\":638422767452440515,\"requestLength\":245,\"responseTime\":638422767453181628,\"responseLength\":794,\"objectId\":\"\",\"streamType\":\"Non-Streaming\",\"modelDeploymentName\":\"gpt-35-turbo\",\"modelName\":\"gpt-35-turbo\",\"modelVersion\":\"1106\"}", "resourceId": "/SUBSCRIPTIONS/00000000-11111-2222-333333333333/RESOURCEGROUPS/OPENAI-GROUP/PROVIDERS/MICROSOFT.COGNITIVESERVICES/ACCOUNTS/DEMO-EXAMPLE", "resultSignature": "200", "time": "2024-01-31T05:45:46.0940000Z", "Tenant": "southindia"}
{ "callerIpAddress": "172.70.34.***", "category": "RequestResponse", "correlationId": "3f66466b-0c0a-4371-92c8-df5c242b6ae7", "durationMs": 274, "event": "ShoeboxCallResult", "location": "southindia", "operationName": "ChatCompletions_Create", "properties": "{\"apiName\":\"Azure OpenAI API version 2023-12-01-preview\",\"requestTime\":638422767727327584,\"requestLength\":245,\"responseTime\":638422767730076182,\"responseLength\":794,\"objectId\":\"\",\"streamType\":\"Non-Streaming\",\"modelDeploymentName\":\"gpt-35-turbo\",\"modelName\":\"gpt-35-turbo\",\"modelVersion\":\"1106\"}", "resourceId": "/SUBSCRIPTIONS/00000000-11111-2222-333333333333/RESOURCEGROUPS/OPENAI-GROUP/PROVIDERS/MICROSOFT.COGNITIVESERVICES/ACCOUNTS/DEMO-EXAMPLE", "resultSignature": "200", "time": "2024-01-31T05:46:14.1090000Z", "Tenant": "southindia"}

And AllMetrics look like this, where you can see token stats, latency, and request count per minute.

{ "count": 1, "total": 34, "minimum": 34, "maximum": 34, "average": 34, "resourceId": "/SUBSCRIPTIONS/00000000-11111-2222-333333333333/RESOURCEGROUPS/OPENAI-GROUP/PROVIDERS/MICROSOFT.COGNITIVESERVICES/ACCOUNTS/DEMO-EXAMPLE", "time": "2024-01-31T04:55:00.0000000Z", "metricName": "TokenTransaction", "timeGrain": "PT1M"}
{ "count": 1, "total": 32, "minimum": 32, "maximum": 32, "average": 32, "resourceId": "/SUBSCRIPTIONS/00000000-11111-2222-333333333333/RESOURCEGROUPS/OPENAI-GROUP/PROVIDERS/MICROSOFT.COGNITIVESERVICES/ACCOUNTS/DEMO-EXAMPLE", "time": "2024-01-31T04:55:00.0000000Z", "metricName": "ProcessedPromptTokens", "timeGrain": "PT1M"}
{ "count": 1, "total": 2, "minimum": 2, "maximum": 2, "average": 2, "resourceId": "/SUBSCRIPTIONS/00000000-11111-2222-333333333333/RESOURCEGROUPS/OPENAI-GROUP/PROVIDERS/MICROSOFT.COGNITIVESERVICES/ACCOUNTS/DEMO-EXAMPLE", "time": "2024-01-31T04:55:00.0000000Z", "metricName": "GeneratedTokens", "timeGrain": "PT1M"}
{ "count": 2, "total": 2, "average": 1, "resourceId": "/SUBSCRIPTIONS/00000000-11111-2222-333333333333/RESOURCEGROUPS/OPENAI-GROUP/PROVIDERS/MICROSOFT.COGNITIVESERVICES/ACCOUNTS/DEMO-EXAMPLE", "time": "2024-01-31T04:58:00.0000000Z", "metricName": "AzureOpenAIRequests", "timeGrain": "PT1M"}
{ "count": 1, "total": 1, "average": 1, "resourceId": "/SUBSCRIPTIONS/00000000-11111-2222-333333333333/RESOURCEGROUPS/OPENAI-GROUP/PROVIDERS/MICROSOFT.COGNITIVESERVICES/ACCOUNTS/DEMO-EXAMPLE", "time": "2024-01-31T04:59:00.0000000Z", "metricName": "AzureOpenAIRequests", "timeGrain": "PT1M"}
{ "count": 2, "total": 2, "minimum": 1, "maximum": 1, "average": 1, "resourceId": "/SUBSCRIPTIONS/00000000-11111-2222-333333333333/RESOURCEGROUPS/OPENAI-GROUP/PROVIDERS/MICROSOFT.COGNITIVESERVICES/ACCOUNTS/DEMO-EXAMPLE", "time": "2024-01-31T04:58:00.0000000Z", "metricName": "SuccessfulCalls", "timeGrain": "PT1M"}
{ "count": 1, "total": 1, "minimum": 1, "maximum": 1, "average": 1, "resourceId": "/SUBSCRIPTIONS/00000000-11111-2222-333333333333/RESOURCEGROUPS/OPENAI-GROUP/PROVIDERS/MICROSOFT.COGNITIVESERVICES/ACCOUNTS/DEMO-EXAMPLE", "time": "2024-01-31T04:59:00.0000000Z", "metricName": "SuccessfulCalls", "timeGrain": "PT1M"}
{ "count": 2, "total": 424, "minimum": 212, "maximum": 212, "average": 212, "resourceId": "/SUBSCRIPTIONS/00000000-11111-2222-333333333333/RESOURCEGROUPS/OPENAI-GROUP/PROVIDERS/MICROSOFT.COGNITIVESERVICES/ACCOUNTS/DEMO-EXAMPLE", "time": "2024-01-31T04:58:00.0000000Z", "metricName": "DataIn", "timeGrain": "PT1M"}
{ "count": 1, "total": 201, "minimum": 201, "maximum": 201, "average": 201, "resourceId": "/SUBSCRIPTIONS/00000000-11111-2222-333333333333/RESOURCEGROUPS/OPENAI-GROUP/PROVIDERS/MICROSOFT.COGNITIVESERVICES/ACCOUNTS/DEMO-EXAMPLE", "time": "2024-01-31T04:59:00.0000000Z", "metricName": "DataIn", "timeGrain": "PT1M"}
{ "count": 2, "total": 1524, "minimum": 762, "maximum": 762, "average": 762, "resourceId": "/SUBSCRIPTIONS/00000000-11111-2222-333333333333/RESOURCEGROUPS/OPENAI-GROUP/PROVIDERS/MICROSOFT.COGNITIVESERVICES/ACCOUNTS/DEMO-EXAMPLE", "time": "2024-01-31T04:58:00.0000000Z", "metricName": "DataOut", "timeGrain": "PT1M"}
{ "count": 1, "total": 762, "minimum": 762, "maximum": 762, "average": 762, "resourceId": "/SUBSCRIPTIONS/00000000-11111-2222-333333333333/RESOURCEGROUPS/OPENAI-GROUP/PROVIDERS/MICROSOFT.COGNITIVESERVICES/ACCOUNTS/DEMO-EXAMPLE", "time": "2024-01-31T04:59:00.0000000Z", "metricName": "DataOut", "timeGrain": "PT1M"}
{ "count": 2, "total": 228693, "minimum": 109455, "maximum": 119238, "average": 114346.5, "resourceId": "/SUBSCRIPTIONS/00000000-11111-2222-333333333333/RESOURCEGROUPS/OPENAI-GROUP/PROVIDERS/MICROSOFT.COGNITIVESERVICES/ACCOUNTS/DEMO-EXAMPLE", "time": "2024-01-31T04:58:00.0000000Z", "metricName": "Latency", "timeGrain": "PT1M"}
{ "count": 1, "total": 143619, "minimum": 143619, "maximum": 143619, "average": 143619, "resourceId": "/SUBSCRIPTIONS/00000000-11111-2222-333333333333/RESOURCEGROUPS/OPENAI-GROUP/PROVIDERS/MICROSOFT.COGNITIVESERVICES/ACCOUNTS/DEMO-EXAMPLE", "time": "2024-01-31T04:59:00.0000000Z", "metricName": "Latency", "timeGrain": "PT1M"}

Semi-ish related to BerriAI/litellm#1721.