OutputParserException: Failed to parse GradeDocuments from completion {"binary_score": {"title": "Binary Score", "description": "Documents are relevant to the question, 'yes' or 'no'", "type": "string"}}. Got: 1 validation error for GradeDocuments binary_score

[Sridevi17j]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.document_loaders import WebBaseLoader
from langchain_community.vectorstores import Chroma
from langchain_openai import OpenAIEmbeddings
from langchain_community.llms import Ollama
from langchain_huggingface import HuggingFaceEmbeddings

model_name = "sentence-transformers/all-mpnet-base-v2"
model_kwargs = {'device': 'cpu'}
encode_kwargs = {'normalize_embeddings': False}
hf = HuggingFaceEmbeddings(
    model_name=model_name,
    model_kwargs=model_kwargs,
    encode_kwargs=encode_kwargs
)

llm = Ollama(model="gemma:2b")

urls = [
    "https://learndatasciencewithme.com/"
]

docs = [WebBaseLoader(url).load() for url in urls]
docs_list = [item for sublist in docs for item in sublist]

text_splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
    chunk_size=250, chunk_overlap=0
)
doc_splits = text_splitter.split_documents(docs_list)

# Add to vectorDB
vectorstore = Chroma.from_documents(
    documents=doc_splits,
    collection_name="rag-chroma",
    embedding=hf,
)
retriever = vectorstore.as_retriever()

from langchain_core.pydantic_v1 import BaseModel, Field
#from langchain_openai import Ollama
from langchain_experimental.llms.ollama_functions import OllamaFunctions

from langchain_core.prompts import ChatPromptTemplate

# Data model
class GradeDocuments(BaseModel):
    """Binary score for relevance check on retrieved documents."""
    binary_score: str = Field(description="Documents are relevant to the question, 'yes' or 'no'")

# LLM with function call
llm = OllamaFunctions(model="gemma:2b",temperature=0)
structured_llm_grader = llm.with_structured_output(GradeDocuments)

# Prompt
system = """You are a grader assessing relevance of a retrieved document to a user question. \n 
    It does not need to be a stringent test. The goal is to filter out erroneous retrievals. \n
    If the document contains keyword(s) or semantic meaning related to the user question, grade it as relevant. \n
    Give a binary score 'yes' or 'no' score to indicate whether the document is relevant to the question."""
grade_prompt = ChatPromptTemplate.from_messages(
    [
        ("system", system),
        ("human", "Retrieved document: \n\n {document} \n\n User question: {question}"),
    ]
)

retrieval_grader = grade_prompt | structured_llm_grader
question = "data science"
docs = retriever.get_relevant_documents(question)
doc_txt = docs[1].page_content
print(retrieval_grader.invoke({"question": question, "document": doc_txt}))

Description

While executing above code, i am getting below error
OutputParserException: Failed to parse GradeDocuments from completion {"binary_score": {"title": "Binary Score", "description": "Documents are relevant to the question, 'yes' or 'no'", "type": "string"}}. Got: 1 validation error for GradeDocuments
binary_score
str type expected (type=type_error.str)
Output is truncated. View as a scrollable element or open in a text editor. Adjust cell output settings...

Below is my retrieval grader output

ChatPromptTemplate(input_variables=['document', 'question'], messages=[SystemMessagePromptTemplate(prompt=PromptTemplate(input_variables=[], template="You are a grader assessing relevance of a retrieved document to a user question. \n \n It does not need to be a stringent test. The goal is to filter out erroneous retrievals. \n\n If the document contains keyword(s) or semantic meaning related to the user question, grade it as relevant. \n\n Give a binary score 'yes' or 'no' score to indicate whether the document is relevant to the question.")), HumanMessagePromptTemplate(prompt=PromptTemplate(input_variables=['document', 'question'], template='Retrieved document: \n\n {document} \n\n User question: {question}'))])
| RunnableBinding(bound=OllamaFunctions(model='gemma:2b', temperature=0.0), kwargs={'functions': [<class 'main.GradeDocuments'>], 'format': 'json'})
| RunnableLambda(parse_response)
| PydanticOutputParser(pydantic_object=<class 'main.GradeDocuments'>)

System Info

System Information

OS: Windows
OS Version: 10.0.22631
Python Version: 3.12.0 (tags/v3.12.0:0fb18b0, Oct 2 2023, 13:03:39) [MSC v.1935 64 bit (AMD64)]

Package Information

langchain_core: 0.2.1
langchain: 0.2.1
langchain_community: 0.2.1
langsmith: 0.1.63
langchain_experimental: 0.0.59
langchain_huggingface: 0.0.1
langchain_openai: 0.1.7
langchain_text_splitters: 0.2.0
langchainhub: 0.1.16
langgraph: 0.0.55

Packages not installed (Not Necessarily a Problem)

The following packages were not found:

langserve

[dosubot[bot]]

The OutputParserException you're encountering is due to the binary_score field in the GradeDocuments model expecting a string, but it seems to be receiving a different type. To resolve this issue, you can use the OutputFixingParser to handle and correct the parsing errors.

Here's how you can modify your code to use the OutputFixingParser:

1. Create an instance of OutputFixingParser using the from_llm method:

   • Pass the language model (llm), the original parser (parser), and optionally a prompt and the maximum number of retries.

2. Use the OutputFixingParser to parse the output:

   • The OutputFixingParser will attempt to parse the output and, if it encounters an OutputParserException, it will retry the parsing process using the provided language model and prompt.

Here is an example code snippet to illustrate this:

from langchain_core.language_models import YourLanguageModel
from langchain_core.output_parsers import YourOriginalParser
from langchain.output_parsers.fix import OutputFixingParser
from langchain_core.prompts import YourPromptTemplate
from langchain_core.exceptions import OutputParserException

# Initialize your language model, original parser, and prompt
llm = YourLanguageModel()
original_parser = YourOriginalParser()
prompt = YourPromptTemplate()

# Create an OutputFixingParser instance
fixing_parser = OutputFixingParser.from_llm(
    llm=llm,
    parser=original_parser,
    prompt=prompt,
    max_retries=3  # You can set the number of retries as needed
)

# Use the fixing parser to parse the completion
try:
    result = fixing_parser.parse(completion)
except OutputParserException as e:
    print(f"Failed to parse after retries: {e}")

This approach leverages the OutputFixingParser to handle and correct the parsing errors, ensuring that the binary_score field is correctly parsed as a string. Additionally, make sure to import OutputParserException from langchain_core.exceptions to handle the exception properly ^[1][2].

Details

About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

[ymcki]

I am also getting the same error at the following line

for output in app.stream(inputs):

of a jupyter notebook
https://github.com/langchain-ai/langgraph/blob/main/examples/rag/langgraph_self_rag.ipynb

[ymcki]

Since gemma 2 doesn't accept system prompt, I changed

grade_prompt = ChatPromptTemplate.from_messages(
[
("human", "%s \n\n Retrieved document: \n\n {document} \n\n User question: {question}" % system),
]
)

Now it is working for me.