Observability: Add pre-built dashboards for basic infrastructure and model metrics

[rddefauw]

Describe the feature

When using a model from Bedrock or hosted in SageMaker, provide a pre-built CloudWatch dashboard that shows the essential infrastructure and model metrics. For Bedrock, this would include things like invocations, latency, and token count. For SageMaker we would have invocations and latency, and perhaps in future provide an extension to track token count. For multi-tenant use cases, we could also track per-tenant dimensions to help with cost allocation, to the extent that the services support this.

Other components like vector databases would support both infrastructure metrics (e..g, CPU overhead on an OpenSearch collection) and metrics like number of retrieval requests.

Use Case

As with any workload, observability is key to understanding workload behavior. In the case of GenAI, surfacing the basic metrics via CloudWatch dashboards and alarms helps the operator understand performance problems (e.g., unusually high invocation latency) and more general trends (average number of model requests).

Proposed Solution

Provide L2 constructs for dashboards for major components, like models hosted in Bedrock or SageMaker, or vector databases in AOSS. The simplest usage might be:

new BedrockModelDashboard(this, "ModelDashboard", { modelId: "anthropic.claude-3-haiku-20240307-v1:0", })

One potential drawback is that the native metrics for Bedrock don't support dimensions beyond model ID. If a single account is hosting multiple workloads in the same region, the Bedrock metrics would be aggregated across all workloads. If in future Bedrock offers more dimensions, we can take advantage of them. We can also consider more advanced constructs that provide their own way to meter things like token count at a more granular level.

Other Information

I have a reference implementation available in a private repo, and plan to publish it soon. It includes pre-baked CloudWatch dashboards for Bedrock and AOSS.

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

[krokoko]

Hi @rddefauw , thank you for this feature request ! This might be a good fit for https://github.com/cdklabs/cdk-monitoring-constructs, what do you think ? I see there is an open feature request to support AOSS

[rddefauw]

I think the problem is not supporting a specific service like AOSS, but knowing which metrics are relevant for a GenAI workload. I think that fits better here.

There are also some more interesting metrics, like collecting human feedback, that I can see coming as a follow-up, and that would require deeper modifications into the application stack.

[krokoko]

Hi @rddefauw , thanks again for this feature request ! Could you please have a look at the draft PR mentioned above (#651) ?
For a first iteration, we can publish the Bedrock dashboard, and I'll create separate tickets for the other component level dashboards (AOSS, SageMaker endpoint) if that works

[rddefauw]

Bedrock dashboard looks good. Agree on creating tickets for other components.