Continual learning with LLM

[sonichi]

Reason: to support continual learning with LLM, such as:

• Use historical interaction log to improve the performance of an agent with the accumulation of interaction sessions.
• Summarize data that don't fit in the context window at once.
• Personalized chat.
• Long chat.

One possible solution is:
Implement a continual learning agent and a teaching agent such that:

• The teaching agent feeds learning goal and learning data to the learning agent. The learning agent maintains learning results of a particular form.
• The learning agent can use LLM to update learning results after each batch of learning data.
• When LLM is used for learning in the learning agent, it only needs the current learning results and the new batch of learning data. So the learning can be performed online and continually.
• The teaching agent allows human input similar to user proxy agent.
• Both the learning agent and the teaching agent can be serialized and deserialized to continue the learning process.
• Optionally, the learning agent tells the teaching agent the maximal number of tokens of the learning data to receive.
• Optionally, the teaching agent tells the learning agent the minimal number of tokens needed for the next batch of learning data.
• Optionally, the teaching agent can provide feedback to the learning agent, including: numeric value about how good is the current learning result(s); and textual feedback.

Tasks

Beta Give feedback

1. Review PR Agent notebook example with human feedback; Support shell command and multiple code blocks; Improve the system message for assistant agent; Improve utility functions for config lists; reuse docker image  #1056
2. Implement a basic solution for continue learning with LLMs and make it compatible with current design #1073
   1 of 3
   +0
   
3. Write user code example #1072
   1 of 5
   +0
   

[qingyun-wu]

This implementation could be used to answer the following research question: How to use historical interaction log to improve the performance of an agent with the accumulation of interaction sessions?

[qingyun-wu]

One possible implementation is to just have a learning component in the AssistantAgent and a teaching component in the UserProxyAgent.