Track model metadata

[SmittieC]

It can really help users out if OCS knows a few things about selected models so that we can build guardrails that will ultimately help lower frustration and make the platform for robust. See this thread as an example where it would have been useful if OCS had known what the model's token limit is.

Model metadata to track

• Token limit
• Rates / cost
• Support for function calling

---

Currently we store model token limits on the Experiment or Pipeline Node but the values are specific to the LLM model so should be stored along with the model name.

We must also allow users to create new 'models' via the UI e.g. fine tuned models.

The token limits should then be accessed through the LlmService objects.

[SmittieC]

If we know the token limit for a specific model, we can

• Do do input token count and disallow users to input messages larger than that which the model can handle (on webusers though)
• Do proper limiting and/or estimation for what the max token limit should be. Currently users can set this to any number, regardless of the model's context limit.

[SmittieC]

Some thoughts on this can be found here cc @snopoke @stephherbers @bderenzi

[SmittieC]

Langfuse has a nice way of doing it. Under the Tracing tab -> Models. This might serve as inspiration

[marcklingen]

Langfuse has a nice way of doing it. Under the Tracing tab -> Models. This might serve as inspiration

fyi, this is currently changing in Langfuse as more detailed model price details are necessary (cached/video/audio tokens). will probably be released next week