Some quick, high level thoughts on improvements/changes

[hwchase17]

A lot has changed in the ~8 months since we launched LangChain. We're constantly adapting the library to best help developers build LLM applications. Part of this involves adding new integrations and chains, which is very visible. But another part is making larger changes to base interfaces that require more thought and feedback from the community, and are less visible.

We want to share some of the latter, less visible improvements/changes we're thinking about (and some we're already working on) in order to get feedback and to give everyone a clearer sense of where we see the library going. Note that these are just some higher level thoughts, and we'll follow up with more detailed plans for some of the larger changes shortly.

Documentation

Trying to keep documentation up-to-date in this fast moving field is a constant struggle. In the past month, we've revamped our doc structure, changed the reference guide style, and worked on improving docstrings to some of our more popular chains. However, there is a still a lot of ground to cover, and this is a continual effort. Since we will likely have to prioritize, feedback on which specific chains/components need better documentation is particularly helpful.

We think of documentation in a few buckets, and we’d love feedback on which ones could use the most work!

• Conceptual docs: different concepts in LangChain, and how they relate
• Reference docs: reference documentation for each component. A description of all the parameters and methods.
  • Here, we will spend a lot more time documenting the prompts. How they are used, how to pass in custom ones
• Examples: Notebooks/cookbooks showing how to us particular chains and components

Modularity

We want to make all parts of LangChain as modular as possible. An obvious example is making the individual modules as standalone as possible. A less obvious example is making subchains more modular.

Let’s consider the [SQLDatabaseChain](https://python.langchain.com/docs/modules/chains/popular/sqlite) as an example. This chain takes user input and constructs a SQL query by calling an LLM, runs that query against a database, then passes those results back to the LLM for summarization. While this is a nice end-to-end flow, we also want to make it as easy as possible to use any individual part of that chain on it's own - e.g., only use the LLMChain that constructs the query. We plan to do this by adding good documentation and constructors for those individual chains. Two ways we are thinking of doing this are adding a SQLDatabaseChain.create_query_chain method on the top level chain class, or add a create_sql_query_chain function. If you have suggestions or preferences on how we expose this, we’re all ears!

Customizability

We want to make it as easy as possible to customize chains. Both the Documentation and Modularity efforts should contribute this. Another big part of this is making it easy to customize prompts for chains. The defaults prompts as meant as general starters, we imagine them being customized for specific use cases, we need to make it easier to do that. We’re also working on things related the [LangChainHub](https://github.com/hwchase17/langchain-hub) to make it easier to discover and share custom prompts.

Base Abstractions

As the stack evolves and best practices for productionizing LLM applications emerge, we’re adapting our interfaces to best support those. We want to put more effort into clarifying these interfaces (documentation) and then also cleaning up any tech debt from previous lives. This is made difficult by the fact that we don’t JUST want to support OpenAI - we want to encourage usage and experimentation of many models. We LOVE feedback and suggestions here on specific ways in which any abstractions seem too restrictive or out of date.

Debugging

When you get an unexpected result, it can often be difficult to debug what exactly went wrong. This is true of complex LLM applications in general. We want to make sure LangChain has best in class debugging abilities. In this vein, we’ve:

1. Added a langchain.debug = True option to print out information to the terminal
2. Added a robust Callback system and integrated with many observability solutions

We are also working on a separate platform offering that will help with this.

Architecture

There is a lot in LangChain. We see several distinct features:

1. Core abstractions for common components needed for building LLM applications (LLMs, vector stores, etc)
2. Integrations for those components (OpenAI LLM, Pinecone vector store, etc)
3. Core logic (Prompt templates, Output Parser, LLMChain, MapReduceChain, etc)
4. Agent Framework (Agent Executor, etc)
5. Use case-specific chains that are more robust (ConversationalRetrievalQA, RetrievalQA, etc)
6. Use case-specific chains that are more experimental (PAL, etc)

We’ve been trying to add as much as possible so that it’s as easy as possible to get started. But we’re now actively thinking of ways to split these functionalities into multiple packages to keep a each one lighter and more focused. Our current frontrunner proposal is to split things into:

• Core library with base abstractions, core logic
• Integrations library with integrations for all the components in Core
• Moving all use case-specific chains to their own packages, grouped by use case

We are debating heavily whether Core and Integrations should be left in the same library for now or separated. This is an area that we’re extremely interested in community feedback - specifically, (1) what parts of LangChain you’d like to be separate so you can use them in a standalone manner, (2) how you’d like new use-case specific chains/agents to be added.

We are actively thinking about this and listening to feedback, and will share more concrete thoughts and plans here next week.

[carterjfulcher]

Glad to see an emphasis on documentation, and appreciate everything you and the team have done to make Langchain great!

[houfu]

whether Core and Integrations should be left in the same library for now or separated

Should be separated. In the past, there weren't that many, but now there are so many it makes the documentation harder to read.

[VeryKumar]

The defaults prompts as meant as general starters, we imagine them being customized for specific use cases, we need to make it easier to do that.

I have already customized prompts for a few chains, but because of the fast pace of development keeping up with changes in components while also having custom components quickly becomes difficult. Wish there was an easy way to separate prompts from code changes in any component.

Also, wish there were an easy way to stitch code with LLM chains without having to wrap them in tools. Tools work great when I want the LLM to choose what to do, but there are time when I want a chain to be tied to custom code and that isn't easy enough yet.

[ericfeunekes]

Mostly commenting so that I see @VeryKumar's response.

Also not sure if it's similar but my recent use case was taking the response from a retriever and running it through a custom function to augment the results. I did it by subclassing the retriever and overriding the method I was using.

I'm not sure if this could have been done with a sequential chain, but I think the work you're planning to focus on base abstractions could be helpful if some of those abstractions implement base transformations like a compose.

[hwchase17]

@ericfeunekes thanks for sharing. that does sound like its probably doable with a sequential chain, but its probably not that easy currently

out of curiosity, what are the biggest benefits your gaining from having it be a single chain (as opposed to just code)? We want to make sure to maintain those benefits when we make any changes

[ericfeunekes]

@hwchase17 I alluded to this in my other comment on the main thread, but having known interfaces allows me to focus on higher level concepts like how I'm going to compose the parts of a chain into larger chains to achieve a goal. Which is I think why langchain is so popular to begin with.

If you can implement those abstract classes and interfaces well so that they can plug and play at any level of composition then you'll allow the users to focus on what they want to achieve rather than how they want to achieve it.

I wrote custom code because right now I'm not entirely sure whether I can plug a multi query retriever into a sequential chain. Will it work? Maybe, but I'll only know when I test it. Whereas if I know that a multi-query retriever returns a list of some interface object (like a dataclass) and langchain has separate map and reduce interfaces, then I could take the next LLM or function call, plug the first into a map on the results from the retriever and then reduce using a function or LLM call.

I guess my point is that having a consistent set of basic interfaces that can be composed is the main benefit I would see from this exercise. Particularly if the interface is the same at any level of composition. Eg I know that if I do prompt > multi-query (which is itself a prompt > LLM > func) > map func > reduce LLM, the output interface should be the same as if I did a single LLM call. Then I can build any tool, to be used anywhere.

[hwchase17]

makes sense! thanks for elaborating

[VeryKumar]

@hwchase17 sure -- since it seems to be a popular one we can look at the sql_database prompt:

Only use the following tables:
{table_info}

Question: {input}"""

_DEFAULT_TEMPLATE = """Given an input question, first create a syntactically correct {dialect} query to run, then look at the results of the query and return the answer. Unless the user specifies in his question a specific number of examples he wishes to obtain, always limit your query to at most {top_k} results. You can order the results by a relevant column to return the most interesting examples in the database.

Never query for all the columns from a specific table, only ask for a the few relevant columns given the question.

Pay attention to use only the column names that you can see in the schema description. Be careful to not query for columns that do not exist. Also, pay attention to which column is in which table.

Use the following format:

Question: Question here
SQLQuery: SQL Query to run
SQLResult: Result of the SQLQuery
Answer: Final answer here

This is the default, but I was running into an error where I did not want the last output to be Answer, but rather I wanted to check the SQLQuery with a sql_validator, which I coded. I naively tried to augment my description of my sql_database chain (using it as a tool), but what ended up working was to change the prompt located in lib/python3.9/site-packages/langchain/chains/sql_database/prompt.py from:

Question: "Question here" SQLQuery: "SQL Query to run" SQLResult: "Result of the SQLQuery" Answer: "Final answer here"

to
Question: "Question here" SQLQuery: "SQL Query to run"

So I

• Achieved the output I wanted (sql query)
• Used my custom code (sql validator)
• Got it to work reliably

With the trade offs of

• Having to manually keep track of new improvements to the sql_database chain prompt
• Unsure what level of abstraction my custom code should sit in order to limit refactors

Let me know If I can explain it more simply

[xleven]

Thanks for the awesome package and glad to provide some feedback.

• Conceptual docs: different concepts in LangChain, and how they relate

• Reference docs: reference documentation for each component. A description of all the parameters and methods.

  • Here, we will spend a lot more time documenting the prompts. How they are used, how to pass in custom ones

• Examples: Notebooks/cookbooks showing how to us particular chains and components

Conceptual guides, along with examples, can be very helpful for beginners, while a comprehensive reference is a must-have for daily development. This saves you from having to dive into the source code every time, which I kind of appreciated as it taught me a lot.

---

Two ways we are thinking of doing this are adding a SQLDatabaseChain.create_query_chain method on the top level chain class, or add a create_sql_query_chain function. If you have suggestions or preferences on how we expose this, we’re all ears!

In my opinion, using class methods is preferable in most cases. It seems a create_xx_class function can create uncertainty within modules by introducing something that appears out of nowhere. Plus class methods are good for documenting(?)

---

We’re also working on things related the [LangChainHub](https://github.com/hwchase17/langchain-hub) to make it easier to discover and share custom prompts.

(2) how you’d like new use-case specific chains/agents to be added.

Above statements gave me an idea. Perhaps it would be beneficial for the core library and case-specific uses if most of the specific chains/agents could be serialized into JSON/YAML files. This would allow users to tune certain keys from outside, such as llm_paras and prompt, and then load with a simple load_chain('lc:some_chain.json'/'local_chain.json'). However, implementing this idea may present challenges in the design and implementation of the architecture, and may not be feasible at all. Forgive me if it does not make sense.

[djpecot]

Agree on the class method point.

In fact, I feel like this may be the way to go forward for customizing prompts and other component customizations, especially for use case-specific chains like ConversationalRetrievalQA, RetrievalQA, etc

[ericfeunekes]

I agree with the comments on modularity and separating the core components. To me one question is what is the lowest level unit (or units) of work?

It seems like there are three: (1) prompt (2) LLM call (3) function call. Taking your sql example, theres a prompt, an LLM call to create the query, and a function that uses that query to make a call to the database.

If you use these three as interfaces that always accept and return a single pydantic model (or dataclass, or whatever) between them then you should be able to compose any combination of these to create whatever you need. Arguably the LLM call is just a function call, but I think it's sufficiently important to be differentiated. Further, these should be composable in any order, so I could make a function call that passes output to a prompt, the prompt to an LLM, and then to another function that parses the results.

You could also break down the composition into map (list > list), reduce (list > single output), and compose (single output > single output). Eg stuff is just a specific implementation of reduce for Documents.

Separating this into a core library with a focus on interface stability and modularity would allow for much clearer documentation and allow people to build custom logic without worrying about changes.

[giladbarnea]

Having to import and construct a wide variety of objects, specific to each use case, feels like having to learn the library's implementation details in order to do basic tasks.

The experience would be much better if we could import less stuff and care less about specifics, and instead use more primitives and flat APIs with a quiet confidence that the specific objects are constructed behind the scenes, even "magically".

Less Chains, less Memories, less Agents etc.

I bet you could run an LLM against the codebase and ask it to group the classes by semantic similarity, and visually see the clusters. These clusters are made of multiple objects, but should probably be exposed to the user by 1-2 objects at most.

[TouristShaun]

The focus on customizability and the plans for its expansion are much appreciated. The agnostic approach to LLMs is one of the most valuable aspects of LangChain, and I sincerely hope this remains a core part of the philosophy moving forward.

There may be criticism from those who find the system overly complex, but the long-term vision is clear: providing a framework that saves developers significant time by enabling hot-swapping of components as needs and technologies evolve. Those who hard-code solutions for specific LLMs may find themselves in difficult situations when they need to switch. OpenAI might be leading now, but the landscape is constantly changing, just look at the Claude 2 release this week! Flexibility is key in this rapidly evolving field.

[385olt]

I don't know if this is useful in any way, but I wanted to share some ideas and maybe get some feedback on them. I think that a package like LangChain should not focus on particular use-cases, but rather provide building blocks which can be combined/composed in multiple ways to obtain the desired behavior. The particular arrangement of these "building blocks" are called chains and are not included in the package. The users can build the chains themselves by following detailed tutorials. It is easier to illustrate using an example. Below is a mock-up implementation of the SQLDatabaseChain using this approach:

from prompt_builder import system, input, output, code_only
from chain_builder import compose, run
from database import DB
from logging import logger
from llms import llm

sql_cmd_prompt = system("Given an input question, [...]") + # system prompt, instruction
                 code_only +                                # specification, return only code
                 input("Tables", "{tables}") +                   
                 input("Question", "{question}") +
                 output("question", sql_query = "SQL Query") # returns a dict of the form {"prompt": "[generated prompt]", "output": "sql_query", "question": "[user's question]"}

db = DB(
        input = "sql_query", 
        output = "sql_result", 
        pass_along = "question"
     ) # receives dict, runs the command from dict["sql_query"], returns dict {"sql_result": "[result in string form]", "question": "[user's question]"}

answer_prompt = system("Answer the question [...]") +
                input("SQL result", "{sql_result}") +
                input("Question", "{question}") +
                output(answer = "Answer") # returns a dict of the form {"prompt": "[generated prompt]", "output": "answer"}

sql_database_chain = compose(
            sql_cmd_prompt,
            llm(temp=0), 
            logger,  # logger receives a dict, displays it, and passes the dict along
            db,
            answer_prompt, 
            llm,
        )

run(
        input = {
            "question": "How many employees are there?", 
            "tables": "CREATE TABLE [...]"
        }, 
        chain = sql_database_chain,
    ) # returns a dict {"answer": "[...]"}

So in this code, LangChain provides only the things that are imported: tools to build prompts, tools to build chains, integrations (db, llm). If the user has a specific use-case in mind, he can follow a tutorial to build the chain where all the building blocks are explained and optimal prompts are provided.

This is how this code is supposed to work: first, we define the elements that will go into our chain: sql_cmd_prompt, db, answer_prompt. Then, we compose the elements we defined (and the ones we take from LangChain, like llm, db, logger) into a chain. Then, we run the chain by providing it with an input. The input goes through the chain as follows:

$$\text{input} \rightarrow \text{sql\_cmd\_prompt} \rightarrow \text{llm(temp=0)} \rightarrow \text{logger} \rightarrow \text{db} \rightarrow \text{answer\_prompt} \rightarrow \text{llm} \rightarrow \text{output}$$

The prompt template sql_cmd_prompt receives the input dictionary and builds the prompt taking into the account the optimal format for the given llm, for example, if llm is Llama, then the system prompt would be prepended with ### System:\n, if its MPT, then it would be formatted as <|im_start|>system\n{system_prompt}<|im_end|>. The prompt template returns a dict with the prompt and uses the output block to pass along the question input and tell the model which key it should use to store its output (here its sql_query). The resulting dict is fed to llm which returns a dict with prompt removed and sql_query added with its output in it. And the chain keeps going. The logger element simply displays the dict and passes it along to the next element in the chain.

This is, of course, a mock-up example of how things could work and technical details are irrelevant.

Doing things this way keeps LangChain small, highly modular and extensible. The chains themselves are transparent and highly customizable. What do you think? Any form of feedback would be highly appreciated.

[hinthornw]

This seems very aligned with what we have in mind :)

[385olt]

Great! Is there any work going on in this direction? Is there a possibility to follow or contribute?

[mpskex]

That would be awesome! We can plug in more SQL database backends now to LangChain!