This quickstart guide will help you set up and use the agent routing system. Follow the steps below to get started.
First, download lmos-router dependency from maven central
implementation("org.eclipse.lmos:lmos-router-llm:x.y.z")
then create the agent routing specifications using SimpleAgentRoutingSpecProvider
and AgentRoutingSpecBuilder
.
val agentRoutingSpecsProvider = SimpleAgentRoutingSpecProvider()
.add(
AgentRoutingSpecBuilder()
.name("offer-agent")
.description("This agent is responsible for offer management")
.version("1.0.0")
.address(Address(uri = "/agents/offer-agent"))
.build()
)
.add(
AgentRoutingSpecBuilder()
.name("service-agent")
.description("This agent is responsible for service management")
.version("1.0.0")
.address(Address(uri = "/agents/service-agent"))
.build()
)
Next, initialize the LLMAgentRoutingSpecsResolver
with the agentRoutingSpecsProvider
.
val agentRoutingSpecResolver = LLMAgentRoutingSpecsResolver(
agentRoutingSpecsProvider,
modelClient = DefaultModelClient(
DefaultModelClientProperties(openAiApiKey = "your-openai-api-key") //Defaults to System.getenv("OPENAI_API_KEY")
)
)
Set up the context and input messages that will be used to resolve the appropriate agent.
val context = Context(listOf(AssistantMessage("Hello")))
val input = UserMessage("Can you help me find a new phone?")
Finally, use the agentRoutingSpecResolver
to resolve the appropriate agent based on the context and input messages.
val result = agentRoutingSpecResolver.resolve(context, input)
The result
should return offer-agent
, indicating that the "offer-agent" is responsible for handling the user's request. Now you can use the address uri to route the user to the appropriate agent.
For spring cloud gateway, refer to the Demo.
This project routes user queries to the most suitable agent based on their capabilities using Language Model (LLM), Vector-based approaches, and a new Hybrid approach.
The Intelligent Agent Routing System directs user queries to the best-suited agent based on their capabilities using three methods:
- LLM-based approach: Uses a language model to understand and match queries with agent capabilities.
- Vector-based approach: Uses vector embeddings to find the most similar agent based on the query.
- Hybrid approach: Extracts abstract requirements from the query using an LLM and then searches for an agent using semantic similarity.
Uses advanced language models like OpenAI's GPT-4o mini to understand the context and semantics of user queries.
Pros:
- Understands complex queries and context.
- Flexible and adaptable to various scenarios.
- Utilizes state-of-the-art NLP techniques.
Cons:
- Expensive due to commercial language model costs.
- Higher response times.
- Dependent on external APIs with potential rate limits.
Uses vector embeddings to represent queries and agent capabilities, comparing them using cosine similarity.
Pros:
- Fast and efficient for large-scale data.
- Scalable to handle more agents and queries.
- Independent of external APIs.
Cons:
- Limited in understanding complex queries.
- Requires initial setup and regular updates.
- Needs maintenance for embedding updates.
Extracts abstract requirements from the query using an LLM and then searches for an agent using semantic similarity.
Pros:
- Balances the strengths of both LLM and Vector-based approaches.
- Better understanding of complex queries than vector-based alone.
- More efficient than LLM-based alone.
Cons:
- Still dependent on external APIs for LLM.
- Requires integration of both LLM and vector-based systems.
Feature | LLM-Based Approach | Vector-Based Approach | Hybrid Approach |
---|---|---|---|
Contextual Understanding | High | Moderate | High |
Flexibility | High | Moderate | High |
Efficiency | Moderate | High | High |
Scalability | Moderate | High | High |
Cost | High | Low | High |
Latency | Higher | Lower | High |
Dependency | High | Low | High |
Setup Complexity | Low | High | High |
Maintenance | Low | High | High |
Contains foundational classes and interfaces:
- ChatMessage: Represents different types of chat messages.
- Context: Represents the conversation context.
- AgentRoutingSpec: Represents agent routing specifications.
- AgentRoutingSpecsProvider: Interface for providing agent routing specifications.
- AgentRoutingSpecsResolver: Interface for resolving agent routing specifications.
- Result: Utility class for handling success and failure cases.
Handles agent routing specifications using a language model:
- DefaultModelClient: Client for calling the OpenAI model.
- LLMAgentRoutingSpecsResolver: Resolves agent routing specifications using a language model.
- ModelPromptProvider: Provides prompts for the language model.
Handles agent routing specifications using vector embeddings:
- DefaultEmbeddingClient: Client for embedding text using a local service.
- OpenAIEmbeddingClient: Client for embedding text using the OpenAI API.
- VectorAgentRoutingSpecsResolver: Resolves agent routing specifications using vector similarity search.
- VectorSearchClient: Interface for searching similar vectors.
- VectorSeedClient: Interface for seeding vectors.
Combines LLM and vector-based approaches:
- HybridAgentRoutingSpecsResolver: Resolves agent routing specifications using a hybrid approach.
Spring Boot starter for the LLM-based agent routing system:
- LLMAgentRoutingSpecsResolverAutoConfiguration: Auto-configuration for the LLM-based agent routing system.
- LLMAgentRoutingSpecsResolverProperties: Configuration properties for the LLM-based agent routing system.
- LLMAgentRoutingSpecsResolverService: Service for resolving agent routing specifications.
Spring Boot starter for the Vector-based agent routing system:
- VectorAgentRoutingSpecsResolverAutoConfiguration: Auto-configuration for the Vector-based agent routing system.
- VectorAgentRoutingSpecsResolverProperties: Configuration properties for the Vector-based agent routing system.
- VectorAgentRoutingSpecsResolverService: Service for resolving agent routing specifications.
- VectorSeedService: Service for seeding vectors.
- VectorSearchService: Service for searching similar vectors.
Spring Boot starter for the Hybrid-based agent routing system:
- HybridAgentRoutingSpecsResolverAutoConfiguration: Auto-configuration for the Hybrid-based agent routing system.
- HybridAgentRoutingSpecsResolverProperties: Configuration properties for the Hybrid-based agent routing system.
- HybridAgentRoutingSpecsResolverService: Service for resolving agent routing specifications.
Sample Spring Boot application demonstrating the system:
- AgentsApplication: Main application class.
- AgentsController: REST controller for handling agent responses.
- SuperRouteGatewayApplication: Spring Cloud Gateway application for routing requests.
Evaluates the performance of the LLM-based, Vector-based, and Hybrid resolvers:
- LLM-based Resolver: Processes 2000 samples.
- Vector-based Resolver: Processes 5000 samples.
- Hybrid Resolver: To be added.
Refer to the Benchmarks for detailed instructions.
The benchmarks include confusion matrices and accuracy metrics for all methods.
You can download the dependencies from maven central by adding the following dependencies to your project:
implementation("org.eclipse.lmos:lmos-router-llm-spring-boot-starter:x.y.z")
Or using Maven:
<dependency>
<groupId>org.eclipse.lmos</groupId>
<artifactId>lmos-router-llm-spring-boot-starter</artifactId>
<version>x.y.z</version>
</dependency>
implementation("org.eclipse.lmos:lmos-router-vector-spring-boot-starter:x.y.z")
Or using Maven:
<dependency>
<groupId>org.eclipse.lmos</groupId>
<artifactId>lmos-router-vector-spring-boot-starter</artifactId>
<version>x.y.z</version>
</dependency>
implementation("org.eclipse.lmos:lmos-router-hybrid-spring-boot-starter:x.y.z")
Or using Maven:
<dependency>
<groupId>org.eclipse.lmos</groupId>
<artifactId>lmos-router-hybrid-spring-boot-starter</artifactId>
<version>x.y.z</version>
</dependency>
If you are not using Spring Boot, you can add the following dependencies:
implementation("org.eclipse.lmos:lmos-router-llm:x.y.z")
implementation("org.eclipse.lmos:lmos-router-vector:x.y.z")
implementation("org.eclipse.lmos:lmos-router-hybrid:x.y.z")
or you can build the project from source:
- Clone the repository:
git clone https://github.com/eclipse-lmos/lmos-router.git
cd lmos-router
-
Set environment variables: (If running Flow tests, they can be enabled by setting gradle project property
runFlowTests=true
)OPENAI_API_KEY
: Your OpenAI API key.VECTOR_SEED_JSON_FILE_PATH
: Path to the JSON file containing seed vectors.
-
Build the project:
./gradlew build
To run the demo:
Refer to the Demo for detailed instructions.
Contributions are welcome! Please read the contributing guidelines for more information.
This project has adopted the Contributor Covenant in version 2.1 as our code of conduct. Please see the details in our CodeOfConduct.md. All contributors must abide by the code of conduct.
By participating in this project, you agree to abide by its Code of Conduct at all times.
Copyright (c) 2024 Deutsche Telekom AG.
Sourcecode licensed under the Apache License, Version 2.0 (the "License"); you may not use this project except in compliance with the License.
This project follows the REUSE standard for software licensing.
Each file contains copyright and license information, and license texts can be found in the ./LICENSES folder. For more information visit https://reuse.software/.
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the LICENSE for the specific language governing permissions and limitations under the License.