Replies: 2 comments 4 replies
-
|
Hey @OSS-GR! I'm here to help you with any bug or question you have. Let's get to the bottom of this together! To stream the response in a structured output using
Here is an example of how to do it: from langchain_community.chat_models import ChatBedrock
from langchain_core.pydantic_v1 import BaseModel
from typing import List, TypedDict, Annotated
class CitationSubModel(TypedDict):
number: int
id: str
class FinalAnswerModel(TypedDict):
answer: Annotated[str, ..., "The answer to the user question using the citations"]
citations: Annotated[List[CitationSubModel], ..., "A dictionary that includes the numbering and the id references for the citations to be used to answer"]
llm = ChatBedrock(
model_id="your_model_id",
aws_access_key_id="your_access_key",
aws_secret_access_key="your_secret_key",
region_name="your_region",
streaming=True # Enable streaming
)
structured_llm = llm.with_structured_output(FinalAnswerModel)
for chunk in structured_llm.stream({"context": context, "task": state["task"], "citation": citation_instruction, "coding_instructions": coding_instructions}):
print(chunk)In this example, the Additionally, the structured_llm = llm.with_structured_output(
FinalAnswerModel,
method="function_calling", # or "json_mode"
include_raw=True # Include raw response along with parsed output
)This configuration allows you to handle the structured output more flexibly and avoid potential recursion errors. The For more detailed examples and alternative methods, you can refer to the LangChain documentation on structured output: This should help you stream the response in a structured output without encountering recursion errors [1][2][3][4][5]. |
Beta Was this translation helpful? Give feedback.
-
|
I am able to stream with the exact code I have using ChatAzureOpenAI, so I am not sure why it does not work with ChatBedrock if both stream and with_structured_output are both implemented. Is this a bug? To answer your response, I already have streaming=True enabled in my definition of ChatBedrock. Here is that part of the code: model_kwargs = {
"max_tokens": 4096,
"temperature": 0,
"top_p": 1,
}
# print("---Streaming---")
model = ChatBedrock(
client=bedrock_runtime,
streaming=streaming, ## SET TO TRUE IN THIS CASE
model_id=model_id,
model_kwargs=model_kwargs,
) |
Beta Was this translation helpful? Give feedback.
-
|
The issue you're encountering is due to the fact that if self.bind_tools is BaseChatModel.bind_tools:
raise NotImplementedError(
"with_structured_output is not implemented for this model."
)Therefore, while you can stream responses using |
Beta Was this translation helpful? Give feedback.
-
@dosu def with_structured_output(
self,
schema: Union[Dict, Type[BaseModel]],
*,
include_raw: bool = False,
**kwargs: Any,
) -> Runnable[LanguageModelInput, Union[Dict, BaseModel]]:
"""Model wrapper that returns outputs formatted to match the given schema.
Args:
schema: The output schema as a dict or a Pydantic class. If a Pydantic class
then the model output will be an object of that class. If a dict then
the model output will be a dict. With a Pydantic class the returned
attributes will be validated, whereas with a dict they will not be.
include_raw: If False then only the parsed structured output is returned. If
an error occurs during model output parsing it will be raised. If True
then both the raw model response (a BaseMessage) and the parsed model
response will be returned. If an error occurs during output parsing it
will be caught and returned as well. The final output is always a dict
with keys "raw", "parsed", and "parsing_error".
Returns:
A Runnable that takes any ChatModel input. The output type depends on
include_raw and schema.
If include_raw is True then output is a dict with keys:
raw: BaseMessage,
parsed: Optional[_DictOrPydantic],
parsing_error: Optional[BaseException],
If include_raw is False and schema is a Dict then the runnable outputs a Dict.
If include_raw is False and schema is a Type[BaseModel] then the runnable
outputs a BaseModel.
Example: Pydantic schema (include_raw=False):
.. code-block:: python
from langchain_aws.chat_models.bedrock import ChatBedrock
from langchain_core.pydantic_v1 import BaseModel
class AnswerWithJustification(BaseModel):
'''An answer to the user question along with justification for the answer.'''
answer: str
justification: str
llm =ChatBedrock(
model_id="anthropic.claude-3-sonnet-20240229-v1:0",
model_kwargs={"temperature": 0.001},
) # type: ignore[call-arg]
structured_llm = llm.with_structured_output(AnswerWithJustification)
structured_llm.invoke("What weighs more a pound of bricks or a pound of feathers")
# -> AnswerWithJustification(
# answer='They weigh the same',
# justification='Both a pound of bricks and a pound of feathers weigh one pound. The weight is the same, but the volume or density of the objects may differ.'
# )
Example: Pydantic schema (include_raw=True):
.. code-block:: python
from langchain_aws.chat_models.bedrock import ChatBedrock
from langchain_core.pydantic_v1 import BaseModel
class AnswerWithJustification(BaseModel):
'''An answer to the user question along with justification for the answer.'''
answer: str
justification: str
llm =ChatBedrock(
model_id="anthropic.claude-3-sonnet-20240229-v1:0",
model_kwargs={"temperature": 0.001},
) # type: ignore[call-arg]
structured_llm = llm.with_structured_output(AnswerWithJustification, include_raw=True)
structured_llm.invoke("What weighs more a pound of bricks or a pound of feathers")
# -> {
# 'raw': AIMessage(content='', additional_kwargs={'tool_calls': [{'id': 'call_Ao02pnFYXD6GN1yzc0uXPsvF', 'function': {'arguments': '{"answer":"They weigh the same.","justification":"Both a pound of bricks and a pound of feathers weigh one pound. The weight is the same, but the volume or density of the objects may differ."}', 'name': 'AnswerWithJustification'}, 'type': 'function'}]}),
# 'parsed': AnswerWithJustification(answer='They weigh the same.', justification='Both a pound of bricks and a pound of feathers weigh one pound. The weight is the same, but the volume or density of the objects may differ.'),
# 'parsing_error': None
# }
Example: Dict schema (include_raw=False):
.. code-block:: python
from langchain_aws.chat_models.bedrock import ChatBedrock
schema = {
"name": "AnswerWithJustification",
"description": "An answer to the user question along with justification for the answer.",
"input_schema": {
"type": "object",
"properties": {
"answer": {"type": "string"},
"justification": {"type": "string"},
},
"required": ["answer", "justification"]
}
}
llm =ChatBedrock(
model_id="anthropic.claude-3-sonnet-20240229-v1:0",
model_kwargs={"temperature": 0.001},
) # type: ignore[call-arg]
structured_llm = llm.with_structured_output(schema)
structured_llm.invoke("What weighs more a pound of bricks or a pound of feathers")
# -> {
# 'answer': 'They weigh the same',
# 'justification': 'Both a pound of bricks and a pound of feathers weigh one pound. The weight is the same, but the volume and density of the two substances differ.'
# }
""" # noqa: E501
if self.beta_use_converse_api:
return self._as_converse.with_structured_output(
schema, include_raw=include_raw, **kwargs
)
if "claude-3" not in self._get_model():
ValueError(
f"Structured output is not supported for model {self._get_model()}"
)
tool_name = convert_to_anthropic_tool(schema)["name"]
llm = self.bind_tools([schema], tool_choice=tool_name)
if isinstance(schema, type) and issubclass(schema, BaseModel):
output_parser = ToolsOutputParser(
first_tool_only=True, pydantic_schemas=[schema]
)
else:
output_parser = ToolsOutputParser(first_tool_only=True, args_only=True)
if include_raw:
parser_assign = RunnablePassthrough.assign(
parsed=itemgetter("raw") | output_parser, parsing_error=lambda _: None
)
parser_none = RunnablePassthrough.assign(parsed=lambda _: None)
parser_with_fallback = parser_assign.with_fallbacks(
[parser_none], exception_key="parsing_error"
)
return RunnableMap(raw=llm) | parser_with_fallback
else:
return llm | output_parser
@property
def _as_converse(self) -> ChatBedrockConverse:
kwargs = {
k: v
for k, v in (self.model_kwargs or {}).items()
if k in ("stop", "stop_sequences", "max_tokens", "temperature", "top_p")
}
return ChatBedrockConverse(
model=self.model_id,
region_name=self.region_name,
credentials_profile_name=self.credentials_profile_name,
config=self.config,
provider=self.provider or "",
base_url=self.endpoint_url,
guardrail_config=(self.guardrails if self._guardrails_enabled else None), # type: ignore[call-arg]
**kwargs,
) |
Beta Was this translation helpful? Give feedback.
-
|
The issue you're encountering is due to the fact that if self.bind_tools is BaseChatModel.bind_tools:
raise NotImplementedError(
"with_structured_output is not implemented for this model."
)Therefore, while you can stream responses using |
Beta Was this translation helpful? Give feedback.
-
|
Same here! |
Beta Was this translation helpful? Give feedback.
-
Checked other resources
Commit to Help
Example Code
Beta Was this translation helpful? Give feedback.
All reactions