How can I improve QnA with the bot using our knowledge base?

[sorthai]

Stack used -

• Using ConversationalRetrievalQAChain | 🦜️🔗 Langchain
• The knowledge base are bunch of pdfs → Embeddings are generated via openai ada → saved in Pinecone.
• When a user query comes, it goes with ConversationalRetrievalQAChain with chat history
• LLM used in langchain is openai turbo 3.5

Here are some examples of bad questions and answers -

Q: “Hi” or “Hi “who are you
A: Tells about itself using “system” instruction provided the prompt. But it also returns sources of the top chunks as sources returned by embeddings search as per the langchain. gpt-turbo LLM correctly understands that sources are not relevant, but langchain doesn’t get to know about this. By then it has already returned sources from embedding search.

Q: “Good morning”.
A: “I am not sure about this. Ask me about some-random-topic-from-knowledgebase-it-picks“.

Q: “Ok, thanks” as reply to some answer bot has given
A: Same reply as good morning because it doesn’t match from topics in knowledebase.

Q: “What are you” as first message
A: Replies correctly from “system” message already provided in prompt.

Q: “What are you” or “Who are you” as reply to some answer bot to a question like “Tell about Solar system”,
A: As it remembers chat history (client sends with each query), so, bot tells back about solar system. The expected reply should have been from the “system” instruction in the prompt.

The same type of questions are asked in openai Chatgpt turbo which doesn’t have our knowledge base, then it answers all correctly as per the query intention.

Code -

  const chain = ConversationalRetrievalQAChain.fromLLM(
    model,
    vectorstore.asRetriever(),
    {
      qaTemplate: QA_PROMPT,
      questionGeneratorTemplate: CONDENSE_PROMPT,
      returnSourceDocuments: true, //The number of source documents returned is 4 by default
    },
  );

Prompts -

const CONDENSE_PROMPT = `Given the following conversation and a follow up question, rephrase the follow up question to be a standalone question.

Chat History:
{chat_history}
Follow Up Input: {question}
Standalone question:`;

const QA_PROMPT = `You are a helpful teacher, your name is Dolphin. You are an AI assistant providing helpful answers based on the context to provide conversational answer without any prior knowledge. You are given the following extracted parts of a long document and a question.
You should only provide hyperlinks that reference the context below. Do NOT make up hyperlinks.
If you can't find the answer in the context below, just say "Hmm, I'm not sure". You can also ask the student to rephrase the question if you need more context. But don't try to make up an answer.
If the question is not related to the context, politely respond that you are tuned to only answer questions that are related to the context.
Answer in a concise or elaborate format as per the intent of the question. Use formating ** to bold, __ to italic & ~~ to cut wherever required. Format the answer using headings, paragraphs or points wherever applicable. 
=========
{context}
=========
Question: {question}
Answer:`;

[oddrationale]

You might consider using a conversational agent instead of a chain. An agent would be able to "decide" whether it should look up information in the datastore or just respond to the query directly.

[smwitkowski]

I've tried this before, but had little success in instructing the agent in a way so it uses a "lookup" tool.

Do you have any examples you could share that work well?

[oddrationale]

There's a good example in the docs: How to combine agents and vectorstores.

Basically, you use a RetrievalQA chain to connect to your datastore. Then add the RetrievalQA chain as a Tool to the agent with an appropriate description for when to use the tool.

Hope that helps!