Programmatically update a StructuredPrompt (prompt template + output schema) in Langsmith

[arne-rustad]

Feature request

It would be great to be able to commit a StructuredPrompt to Langsmith. Currently, it is possible to create a StructuredPrompt in Langsmith using the UI and it can be pulled down as a StructuredPrompt and used directly in code. However, if I try to commit a StructuredPrompt programatically using the Python SDK, then I get an error message (at least if I define the output schema as a Pydantic object).

Code to reproduce

from langsmith import Client
from dotenv import load_dotenv

load_dotenv()

client = Client(
    api_key=os.getenv("LANGSMITH_API_KEY"),
    api_url=os.getenv("LANGCHAIN_ENDPOINT"),
)

from langchain_core.prompts import ChatPromptTemplate
from pydantic import BaseModel, Field

messages = [
    ("human", "Hello, how are you?"),
    ("ai", "I'm doing well, thanks!"),
    ("human", "Write me a poem. Please"),
]

class example_schema(BaseModel):
    poem: str = Field(..., title="Poem", description="The poem generated by the AI.")
    poem_quality: str = Field(..., title="Poem quality", description="The quality of the poem generated by the AI.")

If I just push a ChatPromptTemplate it works

example_prompt = ChatPromptTemplate.from_messages(
    messages=messages
)
client.push_prompt("test_prompt", object=example_prompt)

However, with a StructuredPrompt it fails.

from langchain_core.prompts.structured import StructuredPrompt

structured_prompt = StructuredPrompt.from_messages_and_schema(
    messages=messages,
    schema=example_schema,
)
client.push_prompt("test_prompt", object=structured_prompt)

The error I get is this:

LangSmithError: Failed to POST /commits/-/test_prompt in LangSmith API. HTTPError('400 Client Error: Bad Request for url: https://eu.api.smith.langchain.com/commits/-/test_prompt', '{"detail":"Trying to load an object that doesn\'t implement serialization: {\'lc\': 1, \'type\': \'not_implemented\', \'id\': [\'__main__\', \'example_schema\'], \'repr\': \\"<class \'__main__.example_schema\'>\\"}"}') (edited)

Motivation

A lot of the time it is more useful to store the Pydantic models in the actual code, even though it is tightly connected to the prompt. Perhaps because the model is used more than once in the code or just for type hinting etc. However, it would still be very useful to be able to utilize the Langsmith functionality and UI for iterating on the prompt by being able to test changes side-by-side and even test over whole datasets (great feature by the way). Currently, to use this functionality I need to manually recreate the Pydantic object in the UI, which is quite time-consuming, at least in the beginning when I am also making rapid changes to the extraction object (for instance due to parts of the extraction object being reasoning steps in order to increase quality of extractions).

The frustration is increased due to the current limitations of editing output schemas in the Langsmith UI (it is not possible to change the sequence of properties without deleting and recreating). Since the order of extracted properties often matter (the LLM needs to extract the properties in a certain order to be the most efficient) it is time-consuming to make edits in the UI when we can't drag-and-drop. However, this deserves it own issue. Just wanted to mention it as it increases the frustration of not being able to programatically set a StructuredPrompt.

[Kabichev]

This would be very useful!