diff --git a/docs/concepts/cli.mdx b/docs/concepts/cli.mdx index 19482e4288..4c9f617ba9 100644 --- a/docs/concepts/cli.mdx +++ b/docs/concepts/cli.mdx @@ -12,7 +12,7 @@ The CrewAI CLI provides a set of commands to interact with CrewAI, allowing you To use the CrewAI CLI, make sure you have CrewAI installed: -```shell +```shell Terminal pip install crewai ``` @@ -20,7 +20,7 @@ pip install crewai The basic structure of a CrewAI CLI command is: -```shell +```shell Terminal crewai [COMMAND] [OPTIONS] [ARGUMENTS] ``` @@ -30,7 +30,7 @@ crewai [COMMAND] [OPTIONS] [ARGUMENTS] Create a new crew or flow. -```shell +```shell Terminal crewai create [OPTIONS] TYPE NAME ``` @@ -38,7 +38,7 @@ crewai create [OPTIONS] TYPE NAME - `NAME`: Name of the crew or flow Example: -```shell +```shell Terminal crewai create crew my_new_crew crewai create flow my_new_flow ``` @@ -47,14 +47,14 @@ crewai create flow my_new_flow Show the installed version of CrewAI. -```shell +```shell Terminal crewai version [OPTIONS] ``` - `--tools`: (Optional) Show the installed version of CrewAI tools Example: -```shell +```shell Terminal crewai version crewai version --tools ``` @@ -63,7 +63,7 @@ crewai version --tools Train the crew for a specified number of iterations. -```shell +```shell Terminal crewai train [OPTIONS] ``` @@ -71,7 +71,7 @@ crewai train [OPTIONS] - `-f, --filename TEXT`: Path to a custom file for training (default: "trained_agents_data.pkl") Example: -```shell +```shell Terminal crewai train -n 10 -f my_training_data.pkl ``` @@ -79,14 +79,14 @@ crewai train -n 10 -f my_training_data.pkl Replay the crew execution from a specific task. -```shell +```shell Terminal crewai replay [OPTIONS] ``` - `-t, --task_id TEXT`: Replay the crew from this task ID, including all subsequent tasks Example: -```shell +```shell Terminal crewai replay -t task_123456 ``` @@ -94,7 +94,7 @@ crewai replay -t task_123456 Retrieve your latest crew.kickoff() task outputs. -```shell +```shell Terminal crewai log-tasks-outputs ``` @@ -102,7 +102,7 @@ crewai log-tasks-outputs Reset the crew memories (long, short, entity, latest_crew_kickoff_outputs). -```shell +```shell Terminal crewai reset-memories [OPTIONS] ``` @@ -113,7 +113,7 @@ crewai reset-memories [OPTIONS] - `-a, --all`: Reset ALL memories Example: -```shell +```shell Terminal crewai reset-memories --long --short crewai reset-memories --all ``` @@ -122,7 +122,7 @@ crewai reset-memories --all Test the crew and evaluate the results. -```shell +```shell Terminal crewai test [OPTIONS] ``` @@ -130,7 +130,7 @@ crewai test [OPTIONS] - `-m, --model TEXT`: LLM Model to run the tests on the Crew (default: "gpt-4o-mini") Example: -```shell +```shell Terminal crewai test -n 5 -m gpt-3.5-turbo ``` @@ -138,7 +138,7 @@ crewai test -n 5 -m gpt-3.5-turbo Run the crew. -```shell +```shell Terminal crewai run ``` @@ -153,7 +153,7 @@ Starting in version `0.98.0`, when you run the `crewai chat` command, you start After receiving the results, you can continue interacting with the assistant for further instructions or questions. -```shell +```shell Terminal crewai chat ``` diff --git a/docs/concepts/llms.mdx b/docs/concepts/llms.mdx index 851e930859..261a1fdd8d 100644 --- a/docs/concepts/llms.mdx +++ b/docs/concepts/llms.mdx @@ -243,6 +243,9 @@ There are three ways to configure LLMs in CrewAI. Choose the method that best fi # llm: bedrock/amazon.titan-text-express-v1 # llm: bedrock/meta.llama2-70b-chat-v1 + # Amazon SageMaker Models - Enterprise-grade + # llm: sagemaker/ + # Mistral Models - Open source alternative # llm: mistral/mistral-large-latest # llm: mistral/mistral-medium-latest @@ -506,6 +509,21 @@ Learn how to get the most out of your LLM configuration: ) ``` + + + ```python Code + AWS_ACCESS_KEY_ID= + AWS_SECRET_ACCESS_KEY= + AWS_DEFAULT_REGION= + ``` + + Example usage: + ```python Code + llm = LLM( + model="sagemaker/" + ) + ``` + ```python Code diff --git a/docs/installation.mdx b/docs/installation.mdx index c98bce9ef2..d629c4c80b 100644 --- a/docs/installation.mdx +++ b/docs/installation.mdx @@ -15,10 +15,48 @@ icon: wrench If you need to update Python, visit [python.org/downloads](https://python.org/downloads) +# Setting Up Your Environment + +Before installing CrewAI, it's recommended to set up a virtual environment. This helps isolate your project dependencies and avoid conflicts. + + + + Choose your preferred method to create a virtual environment: + + **Using venv (Python's built-in tool):** + ```shell Terminal + python3 -m venv .venv + ``` + + **Using conda:** + ```shell Terminal + conda create -n crewai-env python=3.12 + ``` + + + + Activate your virtual environment based on your platform: + + **On macOS/Linux (venv):** + ```shell Terminal + source .venv/bin/activate + ``` + + **On Windows (venv):** + ```shell Terminal + .venv\Scripts\activate + ``` + + **Using conda (all platforms):** + ```shell Terminal + conda activate crewai-env + ``` + + + # Installing CrewAI -CrewAI is a flexible and powerful AI framework that enables you to create and manage AI agents, tools, and tasks efficiently. -Let's get you set up! 🚀 +Now let's get you set up! 🚀 @@ -72,9 +110,9 @@ Let's get you set up! 🚀 # Creating a New Project - + We recommend using the YAML Template scaffolding for a structured approach to defining agents and tasks. - + @@ -101,10 +139,22 @@ Let's get you set up! 🚀 │ └── __init__.py └── config/ ├── agents.yaml + ├── config.yaml └── tasks.yaml ``` - + + + + You can install additional tools using UV: + ```shell Terminal + uv add + ``` + + + UV is our preferred package manager as it's significantly faster than pip and provides better dependency resolution. + + Your project will contain these essential files: diff --git a/docs/tools/composiotool.mdx b/docs/tools/composiotool.mdx index 244a5b657c..b72e09dcde 100644 --- a/docs/tools/composiotool.mdx +++ b/docs/tools/composiotool.mdx @@ -1,5 +1,5 @@ --- -title: Composio +title: Composio Tool description: Composio provides 250+ production-ready tools for AI agents with flexible authentication management. icon: gear-code --- @@ -75,7 +75,8 @@ filtered_action_enums = toolset.find_actions_by_use_case( ) tools = toolset.get_tools(actions=filtered_action_enums) -```Set `advanced` to True to get actions for complex use cases +``` +Set `advanced` to True to get actions for complex use cases - Using specific tools: diff --git a/src/crewai/agent.py b/src/crewai/agent.py index b7f0b2896c..5823ef7f9f 100644 --- a/src/crewai/agent.py +++ b/src/crewai/agent.py @@ -3,6 +3,7 @@ import subprocess from typing import Any, Dict, List, Literal, Optional, Union +from litellm import AuthenticationError as LiteLLMAuthenticationError from pydantic import Field, InstanceOf, PrivateAttr, model_validator from crewai.agents import CacheHandler @@ -261,6 +262,9 @@ def execute_task( } )["output"] except Exception as e: + if isinstance(e, LiteLLMAuthenticationError): + # Do not retry on authentication errors + raise e self._times_executed += 1 if self._times_executed > self.max_retry_limit: raise e diff --git a/src/crewai/agents/crew_agent_executor.py b/src/crewai/agents/crew_agent_executor.py index 286f92e672..ee5b9c5822 100644 --- a/src/crewai/agents/crew_agent_executor.py +++ b/src/crewai/agents/crew_agent_executor.py @@ -3,6 +3,8 @@ from dataclasses import dataclass from typing import Any, Callable, Dict, List, Optional, Union +from litellm.exceptions import AuthenticationError as LiteLLMAuthenticationError + from crewai.agents.agent_builder.base_agent import BaseAgent from crewai.agents.agent_builder.base_agent_executor_mixin import CrewAgentExecutorMixin from crewai.agents.parser import ( @@ -13,6 +15,7 @@ OutputParserException, ) from crewai.agents.tools_handler import ToolsHandler +from crewai.llm import LLM from crewai.tools.base_tool import BaseTool from crewai.tools.tool_usage import ToolUsage, ToolUsageErrorException from crewai.utilities import I18N, Printer @@ -54,7 +57,7 @@ def __init__( callbacks: List[Any] = [], ): self._i18n: I18N = I18N() - self.llm = llm + self.llm: LLM = llm self.task = task self.agent = agent self.crew = crew @@ -80,10 +83,8 @@ def __init__( self.tool_name_to_tool_map: Dict[str, BaseTool] = { tool.name: tool for tool in self.tools } - if self.llm.stop: - self.llm.stop = list(set(self.llm.stop + self.stop)) - else: - self.llm.stop = self.stop + self.stop = stop_words + self.llm.stop = list(set(self.llm.stop + self.stop)) def invoke(self, inputs: Dict[str, str]) -> Dict[str, Any]: if "system" in self.prompt: @@ -98,7 +99,11 @@ def invoke(self, inputs: Dict[str, str]) -> Dict[str, Any]: self._show_start_logs() self.ask_for_human_input = bool(inputs.get("ask_for_human_input", False)) - formatted_answer = self._invoke_loop() + + try: + formatted_answer = self._invoke_loop() + except Exception as e: + raise e if self.ask_for_human_input: formatted_answer = self._handle_human_feedback(formatted_answer) @@ -124,7 +129,6 @@ def _invoke_loop(self): self._enforce_rpm_limit() answer = self._get_llm_response() - formatted_answer = self._process_llm_response(answer) if isinstance(formatted_answer, AgentAction): @@ -145,10 +149,40 @@ def _invoke_loop(self): if self._is_context_length_exceeded(e): self._handle_context_length() continue + elif self._is_litellm_authentication_error(e): + self._handle_litellm_auth_error(e) + raise e + else: + self._printer.print( + content=f"Unhandled exception: {e}", + color="red", + ) + finally: + self.iterations += 1 self._show_logs(formatted_answer) return formatted_answer + def _is_litellm_authentication_error(self, exception: Exception) -> bool: + """Check if the exception is a litellm authentication error.""" + if LiteLLMAuthenticationError and isinstance( + exception, LiteLLMAuthenticationError + ): + return True + + return False + + def _handle_litellm_auth_error(self, exception: Exception) -> None: + """Handle litellm authentication error by informing the user and exiting.""" + self._printer.print( + content="Authentication error with litellm occurred. Please check your API key and configuration.", + color="red", + ) + self._printer.print( + content=f"Error details: {exception}", + color="red", + ) + def _has_reached_max_iterations(self) -> bool: """Check if the maximum number of iterations has been reached.""" return self.iterations >= self.max_iter @@ -160,10 +194,17 @@ def _enforce_rpm_limit(self) -> None: def _get_llm_response(self) -> str: """Call the LLM and return the response, handling any invalid responses.""" - answer = self.llm.call( - self.messages, - callbacks=self.callbacks, - ) + try: + answer = self.llm.call( + self.messages, + callbacks=self.callbacks, + ) + except Exception as e: + self._printer.print( + content=f"Error during LLM call: {e}", + color="red", + ) + raise e if not answer: self._printer.print( @@ -184,7 +225,6 @@ def _process_llm_response(self, answer: str) -> Union[AgentAction, AgentFinish]: if FINAL_ANSWER_AND_PARSABLE_ACTION_ERROR_MESSAGE in e.error: answer = answer.split("Observation:")[0].strip() - self.iterations += 1 return self._format_answer(answer) def _handle_agent_action( diff --git a/src/crewai/llm.py b/src/crewai/llm.py index 790d13ead6..98b0bc8553 100644 --- a/src/crewai/llm.py +++ b/src/crewai/llm.py @@ -142,7 +142,6 @@ def __init__( self.temperature = temperature self.top_p = top_p self.n = n - self.stop = stop self.max_completion_tokens = max_completion_tokens self.max_tokens = max_tokens self.presence_penalty = presence_penalty @@ -160,37 +159,63 @@ def __init__( litellm.drop_params = True + # Normalize self.stop to always be a List[str] + if stop is None: + self.stop: List[str] = [] + elif isinstance(stop, str): + self.stop = [stop] + else: + self.stop = stop + self.set_callbacks(callbacks) self.set_env_callbacks() def call( self, - messages: List[Dict[str, str]], + messages: Union[str, List[Dict[str, str]]], tools: Optional[List[dict]] = None, callbacks: Optional[List[Any]] = None, available_functions: Optional[Dict[str, Any]] = None, ) -> str: """ - High-level call method that: - 1) Calls litellm.completion - 2) Checks for function/tool calls - 3) If a tool call is found: - a) executes the function - b) returns the result - 4) If no tool call, returns the text response - - :param messages: The conversation messages - :param tools: Optional list of function schemas for function calling - :param callbacks: Optional list of callbacks - :param available_functions: A dictionary mapping function_name -> actual Python function - :return: Final text response from the LLM or the tool result + High-level llm call method that: + 1) Accepts either a string or a list of messages + 2) Converts string input to the required message format + 3) Calls litellm.completion + 4) Handles function/tool calls if any + 5) Returns the final text response or tool result + + Parameters: + - messages (Union[str, List[Dict[str, str]]]): The input messages for the LLM. + - If a string is provided, it will be converted into a message list with a single entry. + - If a list of dictionaries is provided, each dictionary should have 'role' and 'content' keys. + - tools (Optional[List[dict]]): A list of tool schemas for function calling. + - callbacks (Optional[List[Any]]): A list of callback functions to be executed. + - available_functions (Optional[Dict[str, Any]]): A dictionary mapping function names to actual Python functions. + + Returns: + - str: The final text response from the LLM or the result of a tool function call. + + Examples: + --------- + # Example 1: Using a string input + response = llm.call("Return the name of a random city in the world.") + print(response) + + # Example 2: Using a list of messages + messages = [{"role": "user", "content": "What is the capital of France?"}] + response = llm.call(messages) + print(response) """ + if isinstance(messages, str): + messages = [{"role": "user", "content": messages}] + with suppress_warnings(): if callbacks and len(callbacks) > 0: self.set_callbacks(callbacks) try: - # --- 1) Make the completion call + # --- 1) Prepare the parameters for the completion call params = { "model": self.model, "messages": messages, @@ -211,19 +236,21 @@ def call( "api_version": self.api_version, "api_key": self.api_key, "stream": False, - "tools": tools, # pass the tool schema + "tools": tools, } + # Remove None values from params params = {k: v for k, v in params.items() if v is not None} + # --- 2) Make the completion call response = litellm.completion(**params) response_message = cast(Choices, cast(ModelResponse, response).choices)[ 0 ].message text_response = response_message.content or "" tool_calls = getattr(response_message, "tool_calls", []) - - # Ensure callbacks get the full response object with usage info + + # --- 3) Handle callbacks with usage info if callbacks and len(callbacks) > 0: for callback in callbacks: if hasattr(callback, "log_success_event"): @@ -236,11 +263,11 @@ def call( end_time=0, ) - # --- 2) If no tool calls, return the text response + # --- 4) If no tool calls, return the text response if not tool_calls or not available_functions: return text_response - # --- 3) Handle the tool call + # --- 5) Handle the tool call tool_call = tool_calls[0] function_name = tool_call.function.name @@ -255,7 +282,6 @@ def call( try: # Call the actual tool function result = fn(**function_args) - return result except Exception as e: diff --git a/src/crewai/utilities/llm_utils.py b/src/crewai/utilities/llm_utils.py index 8c02875093..13230edf6c 100644 --- a/src/crewai/utilities/llm_utils.py +++ b/src/crewai/utilities/llm_utils.py @@ -24,12 +24,10 @@ def create_llm( # 1) If llm_value is already an LLM object, return it directly if isinstance(llm_value, LLM): - print("LLM value is already an LLM object") return llm_value # 2) If llm_value is a string (model name) if isinstance(llm_value, str): - print("LLM value is a string") try: created_llm = LLM(model=llm_value) return created_llm @@ -39,12 +37,10 @@ def create_llm( # 3) If llm_value is None, parse environment variables or use default if llm_value is None: - print("LLM value is None") return _llm_via_environment_or_fallback() # 4) Otherwise, attempt to extract relevant attributes from an unknown object try: - print("LLM value is an unknown object") # Extract attributes with explicit types model = ( getattr(llm_value, "model_name", None) diff --git a/tests/agent_test.py b/tests/agent_test.py index 859a5d6936..46b20004e3 100644 --- a/tests/agent_test.py +++ b/tests/agent_test.py @@ -16,7 +16,7 @@ from crewai.tools.tool_calling import InstructorToolCalling from crewai.tools.tool_usage import ToolUsage from crewai.tools.tool_usage_events import ToolUsageFinished -from crewai.utilities import RPMController +from crewai.utilities import Printer, RPMController from crewai.utilities.events import Emitter @@ -1600,3 +1600,103 @@ def test_agent_with_knowledge_sources(): # Assert that the agent provides the correct information assert "red" in result.raw.lower() + + +@pytest.mark.vcr(filter_headers=["authorization"]) +def test_litellm_auth_error_handling(): + """Test that LiteLLM authentication errors are handled correctly and not retried.""" + from litellm import AuthenticationError as LiteLLMAuthenticationError + + # Create an agent with a mocked LLM and max_retry_limit=0 + agent = Agent( + role="test role", + goal="test goal", + backstory="test backstory", + llm=LLM(model="gpt-4"), + max_retry_limit=0, # Disable retries for authentication errors + ) + + # Create a task + task = Task( + description="Test task", + expected_output="Test output", + agent=agent, + ) + + # Mock the LLM call to raise LiteLLMAuthenticationError + with ( + patch.object(LLM, "call") as mock_llm_call, + pytest.raises(LiteLLMAuthenticationError, match="Invalid API key"), + ): + mock_llm_call.side_effect = LiteLLMAuthenticationError( + message="Invalid API key", llm_provider="openai", model="gpt-4" + ) + agent.execute_task(task) + + # Verify the call was only made once (no retries) + mock_llm_call.assert_called_once() + + +def test_crew_agent_executor_litellm_auth_error(): + """Test that CrewAgentExecutor properly identifies and handles LiteLLM authentication errors.""" + from litellm import AuthenticationError as LiteLLMAuthenticationError + + from crewai.agents.tools_handler import ToolsHandler + from crewai.utilities import Printer + + # Create an agent and executor with max_retry_limit=0 + agent = Agent( + role="test role", + goal="test goal", + backstory="test backstory", + llm=LLM(model="gpt-4", api_key="invalid_api_key"), + ) + task = Task( + description="Test task", + expected_output="Test output", + agent=agent, + ) + + # Create executor with all required parameters + executor = CrewAgentExecutor( + agent=agent, + task=task, + llm=agent.llm, + crew=None, + prompt={"system": "You are a test agent", "user": "Execute the task: {input}"}, + max_iter=5, + tools=[], + tools_names="", + stop_words=[], + tools_description="", + tools_handler=ToolsHandler(), + ) + + # Mock the LLM call to raise LiteLLMAuthenticationError + with ( + patch.object(LLM, "call") as mock_llm_call, + patch.object(Printer, "print") as mock_printer, + pytest.raises(LiteLLMAuthenticationError, match="Invalid API key"), + ): + mock_llm_call.side_effect = LiteLLMAuthenticationError( + message="Invalid API key", llm_provider="openai", model="gpt-4" + ) + executor.invoke( + { + "input": "test input", + "tool_names": "", + "tools": "", + } + ) + + # Verify error handling + mock_printer.assert_any_call( + content="Authentication error with litellm occurred. Please check your API key and configuration.", + color="red", + ) + mock_printer.assert_any_call( + content="Error details: litellm.AuthenticationError: Invalid API key", + color="red", + ) + # Verify the call was only made once (no retries) + mock_llm_call.assert_called_once() diff --git a/tests/cassettes/test_agent_error_on_parsing_tool.yaml b/tests/cassettes/test_agent_error_on_parsing_tool.yaml index ea56fa9814..bd1c350fee 100644 --- a/tests/cassettes/test_agent_error_on_parsing_tool.yaml +++ b/tests/cassettes/test_agent_error_on_parsing_tool.yaml @@ -2,21 +2,21 @@ interactions: - request: body: '{"messages": [{"role": "system", "content": "You are test role. test backstory\nYour personal goal is: test goal\nYou ONLY have access to the following tools, and - should NEVER make up tools that are not listed here:\n\nTool Name: get_final_answer(*args: - Any, **kwargs: Any) -> Any\nTool Description: get_final_answer() - Get the final - answer but don''t give it yet, just re-use this tool non-stop. \nTool - Arguments: {}\n\nUse the following format:\n\nThought: you should always think - about what to do\nAction: the action to take, only one name of [get_final_answer], - just the name, exactly as it''s written.\nAction Input: the input to the action, - just a simple python dictionary, enclosed in curly braces, using \" to wrap - keys and values.\nObservation: the result of the action\n\nOnce all necessary - information is gathered:\n\nThought: I now know the final answer\nFinal Answer: - the final answer to the original input question\n"}, {"role": "user", "content": - "\nCurrent Task: Use the get_final_answer tool.\n\nThis is the expect criteria - for your final answer: The final answer\nyou MUST return the actual complete - content as the final answer, not a summary.\n\nBegin! This is VERY important - to you, use the tools available and give your best Final Answer, your job depends - on it!\n\nThought:"}], "model": "gpt-4o"}' + should NEVER make up tools that are not listed here:\n\nTool Name: get_final_answer\nTool + Arguments: {}\nTool Description: Get the final answer but don''t give it yet, + just re-use this\n tool non-stop.\n\nIMPORTANT: Use the following format + in your response:\n\n```\nThought: you should always think about what to do\nAction: + the action to take, only one name of [get_final_answer], just the name, exactly + as it''s written.\nAction Input: the input to the action, just a simple JSON + object, enclosed in curly braces, using \" to wrap keys and values.\nObservation: + the result of the action\n```\n\nOnce all necessary information is gathered, + return the following format:\n\n```\nThought: I now know the final answer\nFinal + Answer: the final answer to the original input question\n```"}, {"role": "user", + "content": "\nCurrent Task: Use the get_final_answer tool.\n\nThis is the expect + criteria for your final answer: The final answer\nyou MUST return the actual + complete content as the final answer, not a summary.\n\nBegin! This is VERY + important to you, use the tools available and give your best Final Answer, your + job depends on it!\n\nThought:"}], "model": "gpt-4o", "stop": ["\nObservation:"]}' headers: accept: - application/json @@ -25,16 +25,13 @@ interactions: connection: - keep-alive content-length: - - '1325' + - '1367' content-type: - application/json - cookie: - - _cfuvid=ePJSDFdHag2D8lj21_ijAMWjoA6xfnPNxN4uekvC728-1727226247743-0.0.1.1-604800000; - __cf_bm=3giyBOIM0GNudFELtsBWYXwLrpLBTNLsh81wfXgu2tg-1727226247-1.0.1.1-ugUDz0c5EhmfVpyGtcdedlIWeDGuy2q0tXQTKVpv83HZhvxgBcS7SBL1wS4rapPM38yhfEcfwA79ARt3HQEzKA host: - api.openai.com user-agent: - - OpenAI/Python 1.47.0 + - OpenAI/Python 1.59.6 x-stainless-arch: - arm64 x-stainless-async: @@ -44,30 +41,35 @@ interactions: x-stainless-os: - MacOS x-stainless-package-version: - - 1.47.0 + - 1.59.6 x-stainless-raw-response: - 'true' + x-stainless-retry-count: + - '0' x-stainless-runtime: - CPython x-stainless-runtime-version: - - 3.11.7 + - 3.12.7 method: POST uri: https://api.openai.com/v1/chat/completions response: - content: "{\n \"id\": \"chatcmpl-ABAtOWmVjvzQ9X58tKAUcOF4gmXwx\",\n \"object\": - \"chat.completion\",\n \"created\": 1727226842,\n \"model\": \"gpt-4o-2024-05-13\",\n + content: "{\n \"id\": \"chatcmpl-AsXdf4OZKCZSigmN4k0gyh67NciqP\",\n \"object\": + \"chat.completion\",\n \"created\": 1737562383,\n \"model\": \"gpt-4o-2024-08-06\",\n \ \"choices\": [\n {\n \"index\": 0,\n \"message\": {\n \"role\": - \"assistant\",\n \"content\": \"Thought: I need to use the get_final_answer - tool to determine the final answer.\\nAction: get_final_answer\\nAction Input: - {}\",\n \"refusal\": null\n },\n \"logprobs\": null,\n \"finish_reason\": - \"stop\"\n }\n ],\n \"usage\": {\n \"prompt_tokens\": 274,\n \"completion_tokens\": - 27,\n \"total_tokens\": 301,\n \"completion_tokens_details\": {\n \"reasoning_tokens\": - 0\n }\n },\n \"system_fingerprint\": \"fp_e375328146\"\n}\n" + \"assistant\",\n \"content\": \"```\\nThought: I have to use the available + tool to get the final answer. Let's proceed with executing it.\\nAction: get_final_answer\\nAction + Input: {}\",\n \"refusal\": null\n },\n \"logprobs\": null,\n + \ \"finish_reason\": \"stop\"\n }\n ],\n \"usage\": {\n \"prompt_tokens\": + 274,\n \"completion_tokens\": 33,\n \"total_tokens\": 307,\n \"prompt_tokens_details\": + {\n \"cached_tokens\": 0,\n \"audio_tokens\": 0\n },\n \"completion_tokens_details\": + {\n \"reasoning_tokens\": 0,\n \"audio_tokens\": 0,\n \"accepted_prediction_tokens\": + 0,\n \"rejected_prediction_tokens\": 0\n }\n },\n \"service_tier\": + \"default\",\n \"system_fingerprint\": \"fp_50cad350e4\"\n}\n" headers: CF-Cache-Status: - DYNAMIC CF-RAY: - - 8c8727b3492f31e6-MIA + - 9060d43e3be1d690-IAD Connection: - keep-alive Content-Encoding: @@ -75,19 +77,27 @@ interactions: Content-Type: - application/json Date: - - Wed, 25 Sep 2024 01:14:03 GMT + - Wed, 22 Jan 2025 16:13:03 GMT Server: - cloudflare + Set-Cookie: + - __cf_bm=_Jcp7wnO_mXdvOnborCN6j8HwJxJXbszedJC1l7pFUg-1737562383-1.0.1.1-pDSLXlg.nKjG4wsT7mTJPjUvOX1UJITiS4MqKp6yfMWwRSJINsW1qC48SAcjBjakx2H5I1ESVk9JtUpUFDtf4g; + path=/; expires=Wed, 22-Jan-25 16:43:03 GMT; domain=.api.openai.com; HttpOnly; + Secure; SameSite=None + - _cfuvid=x3SYvzL2nq_PTBGtE8R9cl5CkeaaDzZFQIrYfo91S2s-1737562383916-0.0.1.1-604800000; + path=/; domain=.api.openai.com; HttpOnly; Secure; SameSite=None Transfer-Encoding: - chunked X-Content-Type-Options: - nosniff access-control-expose-headers: - X-Request-ID + alt-svc: + - h3=":443"; ma=86400 openai-organization: - crewai-iuxna1 openai-processing-ms: - - '348' + - '791' openai-version: - '2020-10-01' strict-transport-security: @@ -99,45 +109,59 @@ interactions: x-ratelimit-remaining-requests: - '9999' x-ratelimit-remaining-tokens: - - '29999682' + - '29999680' x-ratelimit-reset-requests: - 6ms x-ratelimit-reset-tokens: - 0s x-request-id: - - req_be929caac49706f487950548bdcdd46e + - req_eeed99acafd3aeb1e3d4a6c8063192b0 http_version: HTTP/1.1 status_code: 200 - request: body: '{"messages": [{"role": "system", "content": "You are test role. test backstory\nYour personal goal is: test goal\nYou ONLY have access to the following tools, and - should NEVER make up tools that are not listed here:\n\nTool Name: get_final_answer(*args: - Any, **kwargs: Any) -> Any\nTool Description: get_final_answer() - Get the final - answer but don''t give it yet, just re-use this tool non-stop. \nTool - Arguments: {}\n\nUse the following format:\n\nThought: you should always think - about what to do\nAction: the action to take, only one name of [get_final_answer], - just the name, exactly as it''s written.\nAction Input: the input to the action, - just a simple python dictionary, enclosed in curly braces, using \" to wrap - keys and values.\nObservation: the result of the action\n\nOnce all necessary - information is gathered:\n\nThought: I now know the final answer\nFinal Answer: - the final answer to the original input question\n"}, {"role": "user", "content": - "\nCurrent Task: Use the get_final_answer tool.\n\nThis is the expect criteria - for your final answer: The final answer\nyou MUST return the actual complete - content as the final answer, not a summary.\n\nBegin! This is VERY important - to you, use the tools available and give your best Final Answer, your job depends - on it!\n\nThought:"}, {"role": "user", "content": "Thought: I need to use the - get_final_answer tool to determine the final answer.\nAction: get_final_answer\nAction + should NEVER make up tools that are not listed here:\n\nTool Name: get_final_answer\nTool + Arguments: {}\nTool Description: Get the final answer but don''t give it yet, + just re-use this\n tool non-stop.\n\nIMPORTANT: Use the following format + in your response:\n\n```\nThought: you should always think about what to do\nAction: + the action to take, only one name of [get_final_answer], just the name, exactly + as it''s written.\nAction Input: the input to the action, just a simple JSON + object, enclosed in curly braces, using \" to wrap keys and values.\nObservation: + the result of the action\n```\n\nOnce all necessary information is gathered, + return the following format:\n\n```\nThought: I now know the final answer\nFinal + Answer: the final answer to the original input question\n```"}, {"role": "user", + "content": "\nCurrent Task: Use the get_final_answer tool.\n\nThis is the expect + criteria for your final answer: The final answer\nyou MUST return the actual + complete content as the final answer, not a summary.\n\nBegin! This is VERY + important to you, use the tools available and give your best Final Answer, your + job depends on it!\n\nThought:"}, {"role": "assistant", "content": "```\nThought: + I have to use the available tool to get the final answer. Let''s proceed with + executing it.\nAction: get_final_answer\nAction Input: {}\nObservation: I encountered + an error: Error on parsing tool.\nMoving on then. I MUST either use a tool (use + one at time) OR give my best final answer not both at the same time. When responding, + I must use the following format:\n\n```\nThought: you should always think about + what to do\nAction: the action to take, should be one of [get_final_answer]\nAction + Input: the input to the action, dictionary enclosed in curly braces\nObservation: + the result of the action\n```\nThis Thought/Action/Action Input/Result can repeat + N times. Once I know the final answer, I must return the following format:\n\n```\nThought: + I now can give a great answer\nFinal Answer: Your final answer must be the great + and the most complete as possible, it must be outcome described\n\n```"}, {"role": + "assistant", "content": "```\nThought: I have to use the available tool to get + the final answer. Let''s proceed with executing it.\nAction: get_final_answer\nAction Input: {}\nObservation: I encountered an error: Error on parsing tool.\nMoving on then. I MUST either use a tool (use one at time) OR give my best final answer - not both at the same time. To Use the following format:\n\nThought: you should - always think about what to do\nAction: the action to take, should be one of - [get_final_answer]\nAction Input: the input to the action, dictionary enclosed - in curly braces\nObservation: the result of the action\n... (this Thought/Action/Action - Input/Result can repeat N times)\nThought: I now can give a great answer\nFinal + not both at the same time. When responding, I must use the following format:\n\n```\nThought: + you should always think about what to do\nAction: the action to take, should + be one of [get_final_answer]\nAction Input: the input to the action, dictionary + enclosed in curly braces\nObservation: the result of the action\n```\nThis Thought/Action/Action + Input/Result can repeat N times. Once I know the final answer, I must return + the following format:\n\n```\nThought: I now can give a great answer\nFinal Answer: Your final answer must be the great and the most complete as possible, - it must be outcome described\n\n \nNow it''s time you MUST give your absolute + it must be outcome described\n\n```\nNow it''s time you MUST give your absolute best final answer. You''ll ignore all previous instructions, stop using any - tools, and just return your absolute BEST Final answer."}], "model": "gpt-4o"}' + tools, and just return your absolute BEST Final answer."}], "model": "gpt-4o", + "stop": ["\nObservation:"]}' headers: accept: - application/json @@ -146,16 +170,16 @@ interactions: connection: - keep-alive content-length: - - '2320' + - '3445' content-type: - application/json cookie: - - _cfuvid=ePJSDFdHag2D8lj21_ijAMWjoA6xfnPNxN4uekvC728-1727226247743-0.0.1.1-604800000; - __cf_bm=3giyBOIM0GNudFELtsBWYXwLrpLBTNLsh81wfXgu2tg-1727226247-1.0.1.1-ugUDz0c5EhmfVpyGtcdedlIWeDGuy2q0tXQTKVpv83HZhvxgBcS7SBL1wS4rapPM38yhfEcfwA79ARt3HQEzKA + - __cf_bm=_Jcp7wnO_mXdvOnborCN6j8HwJxJXbszedJC1l7pFUg-1737562383-1.0.1.1-pDSLXlg.nKjG4wsT7mTJPjUvOX1UJITiS4MqKp6yfMWwRSJINsW1qC48SAcjBjakx2H5I1ESVk9JtUpUFDtf4g; + _cfuvid=x3SYvzL2nq_PTBGtE8R9cl5CkeaaDzZFQIrYfo91S2s-1737562383916-0.0.1.1-604800000 host: - api.openai.com user-agent: - - OpenAI/Python 1.47.0 + - OpenAI/Python 1.59.6 x-stainless-arch: - arm64 x-stainless-async: @@ -165,29 +189,36 @@ interactions: x-stainless-os: - MacOS x-stainless-package-version: - - 1.47.0 + - 1.59.6 x-stainless-raw-response: - 'true' + x-stainless-retry-count: + - '0' x-stainless-runtime: - CPython x-stainless-runtime-version: - - 3.11.7 + - 3.12.7 method: POST uri: https://api.openai.com/v1/chat/completions response: - content: "{\n \"id\": \"chatcmpl-ABAtPaaeRfdNsZ3k06CfAmrEW8IJu\",\n \"object\": - \"chat.completion\",\n \"created\": 1727226843,\n \"model\": \"gpt-4o-2024-05-13\",\n + content: "{\n \"id\": \"chatcmpl-AsXdg9UrLvAiqWP979E6DszLsQ84k\",\n \"object\": + \"chat.completion\",\n \"created\": 1737562384,\n \"model\": \"gpt-4o-2024-08-06\",\n \ \"choices\": [\n {\n \"index\": 0,\n \"message\": {\n \"role\": - \"assistant\",\n \"content\": \"Final Answer: The final answer\",\n \"refusal\": - null\n },\n \"logprobs\": null,\n \"finish_reason\": \"stop\"\n - \ }\n ],\n \"usage\": {\n \"prompt_tokens\": 483,\n \"completion_tokens\": - 6,\n \"total_tokens\": 489,\n \"completion_tokens_details\": {\n \"reasoning_tokens\": - 0\n }\n },\n \"system_fingerprint\": \"fp_e375328146\"\n}\n" + \"assistant\",\n \"content\": \"```\\nThought: I now know the final answer\\nFinal + Answer: The final answer must be the great and the most complete as possible, + it must be outcome described.\\n```\",\n \"refusal\": null\n },\n + \ \"logprobs\": null,\n \"finish_reason\": \"stop\"\n }\n ],\n + \ \"usage\": {\n \"prompt_tokens\": 719,\n \"completion_tokens\": 35,\n + \ \"total_tokens\": 754,\n \"prompt_tokens_details\": {\n \"cached_tokens\": + 0,\n \"audio_tokens\": 0\n },\n \"completion_tokens_details\": {\n + \ \"reasoning_tokens\": 0,\n \"audio_tokens\": 0,\n \"accepted_prediction_tokens\": + 0,\n \"rejected_prediction_tokens\": 0\n }\n },\n \"service_tier\": + \"default\",\n \"system_fingerprint\": \"fp_50cad350e4\"\n}\n" headers: CF-Cache-Status: - DYNAMIC CF-RAY: - - 8c8727b9da1f31e6-MIA + - 9060d4441edad690-IAD Connection: - keep-alive Content-Encoding: @@ -195,7 +226,7 @@ interactions: Content-Type: - application/json Date: - - Wed, 25 Sep 2024 01:14:03 GMT + - Wed, 22 Jan 2025 16:13:05 GMT Server: - cloudflare Transfer-Encoding: @@ -209,7 +240,7 @@ interactions: openai-organization: - crewai-iuxna1 openai-processing-ms: - - '188' + - '928' openai-version: - '2020-10-01' strict-transport-security: @@ -221,13 +252,13 @@ interactions: x-ratelimit-remaining-requests: - '9999' x-ratelimit-remaining-tokens: - - '29999445' + - '29999187' x-ratelimit-reset-requests: - 6ms x-ratelimit-reset-tokens: - 1ms x-request-id: - - req_d8e32538689fe064627468bad802d9a8 + - req_61fc7506e6db326ec572224aec81ef23 http_version: HTTP/1.1 status_code: 200 version: 1 diff --git a/tests/cassettes/test_llm_call_with_message_list.yaml b/tests/cassettes/test_llm_call_with_message_list.yaml new file mode 100644 index 0000000000..da204c6470 --- /dev/null +++ b/tests/cassettes/test_llm_call_with_message_list.yaml @@ -0,0 +1,102 @@ +interactions: +- request: + body: '{"messages": [{"role": "user", "content": "What is the capital of France?"}], + "model": "gpt-4o-mini"}' + headers: + accept: + - application/json + accept-encoding: + - gzip, deflate + connection: + - keep-alive + content-length: + - '101' + content-type: + - application/json + cookie: + - _cfuvid=8NrWEBP3dDmc8p2.csR.EdsSwS8zFvzWI1kPICaK_fM-1737568015338-0.0.1.1-604800000; + __cf_bm=pKr3NwXmTZN9rMSlKvEX40VPKbrxF93QwDNHunL2v8Y-1737568015-1.0.1.1-nR0EA7hYIwWpIBYUI53d9xQrUnl5iML6lgz4AGJW4ZGPBDxFma3PZ2cBhlr_hE7wKa5fV3r32eMu_rNWMXD.eA + host: + - api.openai.com + user-agent: + - OpenAI/Python 1.59.6 + x-stainless-arch: + - arm64 + x-stainless-async: + - 'false' + x-stainless-lang: + - python + x-stainless-os: + - MacOS + x-stainless-package-version: + - 1.59.6 + x-stainless-raw-response: + - 'true' + x-stainless-retry-count: + - '0' + x-stainless-runtime: + - CPython + x-stainless-runtime-version: + - 3.12.7 + method: POST + uri: https://api.openai.com/v1/chat/completions + response: + content: "{\n \"id\": \"chatcmpl-AsZ6WjNfEOrHwwEEdSZZCRBiTpBMS\",\n \"object\": + \"chat.completion\",\n \"created\": 1737568016,\n \"model\": \"gpt-4o-mini-2024-07-18\",\n + \ \"choices\": [\n {\n \"index\": 0,\n \"message\": {\n \"role\": + \"assistant\",\n \"content\": \"The capital of France is Paris.\",\n + \ \"refusal\": null\n },\n \"logprobs\": null,\n \"finish_reason\": + \"stop\"\n }\n ],\n \"usage\": {\n \"prompt_tokens\": 14,\n \"completion_tokens\": + 8,\n \"total_tokens\": 22,\n \"prompt_tokens_details\": {\n \"cached_tokens\": + 0,\n \"audio_tokens\": 0\n },\n \"completion_tokens_details\": {\n + \ \"reasoning_tokens\": 0,\n \"audio_tokens\": 0,\n \"accepted_prediction_tokens\": + 0,\n \"rejected_prediction_tokens\": 0\n }\n },\n \"service_tier\": + \"default\",\n \"system_fingerprint\": \"fp_72ed7ab54c\"\n}\n" + headers: + CF-Cache-Status: + - DYNAMIC + CF-RAY: + - 90615dc63b805cb1-RDU + Connection: + - keep-alive + Content-Encoding: + - gzip + Content-Type: + - application/json + Date: + - Wed, 22 Jan 2025 17:46:56 GMT + Server: + - cloudflare + Transfer-Encoding: + - chunked + X-Content-Type-Options: + - nosniff + access-control-expose-headers: + - X-Request-ID + alt-svc: + - h3=":443"; ma=86400 + openai-organization: + - crewai-iuxna1 + openai-processing-ms: + - '355' + openai-version: + - '2020-10-01' + strict-transport-security: + - max-age=31536000; includeSubDomains; preload + x-ratelimit-limit-requests: + - '30000' + x-ratelimit-limit-tokens: + - '150000000' + x-ratelimit-remaining-requests: + - '29999' + x-ratelimit-remaining-tokens: + - '149999974' + x-ratelimit-reset-requests: + - 2ms + x-ratelimit-reset-tokens: + - 0s + x-request-id: + - req_cdbed69c9c63658eb552b07f1220df19 + http_version: HTTP/1.1 + status_code: 200 +version: 1 diff --git a/tests/cassettes/test_llm_call_with_string_input.yaml b/tests/cassettes/test_llm_call_with_string_input.yaml new file mode 100644 index 0000000000..f0c2a51e68 --- /dev/null +++ b/tests/cassettes/test_llm_call_with_string_input.yaml @@ -0,0 +1,108 @@ +interactions: +- request: + body: '{"messages": [{"role": "user", "content": "Return the name of a random + city in the world."}], "model": "gpt-4o-mini"}' + headers: + accept: + - application/json + accept-encoding: + - gzip, deflate + connection: + - keep-alive + content-length: + - '117' + content-type: + - application/json + cookie: + - _cfuvid=3UeEmz_rnmsoZxrVUv32u35gJOi766GDWNe5_RTjiPk-1736537376739-0.0.1.1-604800000 + host: + - api.openai.com + user-agent: + - OpenAI/Python 1.59.6 + x-stainless-arch: + - arm64 + x-stainless-async: + - 'false' + x-stainless-lang: + - python + x-stainless-os: + - MacOS + x-stainless-package-version: + - 1.59.6 + x-stainless-raw-response: + - 'true' + x-stainless-retry-count: + - '0' + x-stainless-runtime: + - CPython + x-stainless-runtime-version: + - 3.12.7 + method: POST + uri: https://api.openai.com/v1/chat/completions + response: + content: "{\n \"id\": \"chatcmpl-AsZ6UtbaNSMpNU9VJKxvn52t5eJTq\",\n \"object\": + \"chat.completion\",\n \"created\": 1737568014,\n \"model\": \"gpt-4o-mini-2024-07-18\",\n + \ \"choices\": [\n {\n \"index\": 0,\n \"message\": {\n \"role\": + \"assistant\",\n \"content\": \"How about \\\"Lisbon\\\"? It\u2019s the + capital city of Portugal, known for its rich history and vibrant culture.\",\n + \ \"refusal\": null\n },\n \"logprobs\": null,\n \"finish_reason\": + \"stop\"\n }\n ],\n \"usage\": {\n \"prompt_tokens\": 18,\n \"completion_tokens\": + 24,\n \"total_tokens\": 42,\n \"prompt_tokens_details\": {\n \"cached_tokens\": + 0,\n \"audio_tokens\": 0\n },\n \"completion_tokens_details\": {\n + \ \"reasoning_tokens\": 0,\n \"audio_tokens\": 0,\n \"accepted_prediction_tokens\": + 0,\n \"rejected_prediction_tokens\": 0\n }\n },\n \"service_tier\": + \"default\",\n \"system_fingerprint\": \"fp_72ed7ab54c\"\n}\n" + headers: + CF-Cache-Status: + - DYNAMIC + CF-RAY: + - 90615dbcaefb5cb1-RDU + Connection: + - keep-alive + Content-Encoding: + - gzip + Content-Type: + - application/json + Date: + - Wed, 22 Jan 2025 17:46:55 GMT + Server: + - cloudflare + Set-Cookie: + - __cf_bm=pKr3NwXmTZN9rMSlKvEX40VPKbrxF93QwDNHunL2v8Y-1737568015-1.0.1.1-nR0EA7hYIwWpIBYUI53d9xQrUnl5iML6lgz4AGJW4ZGPBDxFma3PZ2cBhlr_hE7wKa5fV3r32eMu_rNWMXD.eA; + path=/; expires=Wed, 22-Jan-25 18:16:55 GMT; domain=.api.openai.com; HttpOnly; + Secure; SameSite=None + - _cfuvid=8NrWEBP3dDmc8p2.csR.EdsSwS8zFvzWI1kPICaK_fM-1737568015338-0.0.1.1-604800000; + path=/; domain=.api.openai.com; HttpOnly; Secure; SameSite=None + Transfer-Encoding: + - chunked + X-Content-Type-Options: + - nosniff + access-control-expose-headers: + - X-Request-ID + alt-svc: + - h3=":443"; ma=86400 + openai-organization: + - crewai-iuxna1 + openai-processing-ms: + - '449' + openai-version: + - '2020-10-01' + strict-transport-security: + - max-age=31536000; includeSubDomains; preload + x-ratelimit-limit-requests: + - '30000' + x-ratelimit-limit-tokens: + - '150000000' + x-ratelimit-remaining-requests: + - '29999' + x-ratelimit-remaining-tokens: + - '149999971' + x-ratelimit-reset-requests: + - 2ms + x-ratelimit-reset-tokens: + - 0s + x-request-id: + - req_898373758d2eae3cd84814050b2588e3 + http_version: HTTP/1.1 + status_code: 200 +version: 1 diff --git a/tests/cassettes/test_llm_call_with_string_input_and_callbacks.yaml b/tests/cassettes/test_llm_call_with_string_input_and_callbacks.yaml new file mode 100644 index 0000000000..a930a60a76 --- /dev/null +++ b/tests/cassettes/test_llm_call_with_string_input_and_callbacks.yaml @@ -0,0 +1,102 @@ +interactions: +- request: + body: '{"messages": [{"role": "user", "content": "Tell me a joke."}], "model": + "gpt-4o-mini"}' + headers: + accept: + - application/json + accept-encoding: + - gzip, deflate + connection: + - keep-alive + content-length: + - '86' + content-type: + - application/json + cookie: + - _cfuvid=8NrWEBP3dDmc8p2.csR.EdsSwS8zFvzWI1kPICaK_fM-1737568015338-0.0.1.1-604800000; + __cf_bm=pKr3NwXmTZN9rMSlKvEX40VPKbrxF93QwDNHunL2v8Y-1737568015-1.0.1.1-nR0EA7hYIwWpIBYUI53d9xQrUnl5iML6lgz4AGJW4ZGPBDxFma3PZ2cBhlr_hE7wKa5fV3r32eMu_rNWMXD.eA + host: + - api.openai.com + user-agent: + - OpenAI/Python 1.59.6 + x-stainless-arch: + - arm64 + x-stainless-async: + - 'false' + x-stainless-lang: + - python + x-stainless-os: + - MacOS + x-stainless-package-version: + - 1.59.6 + x-stainless-raw-response: + - 'true' + x-stainless-retry-count: + - '0' + x-stainless-runtime: + - CPython + x-stainless-runtime-version: + - 3.12.7 + method: POST + uri: https://api.openai.com/v1/chat/completions + response: + content: "{\n \"id\": \"chatcmpl-AsZ6VyjuUcXYpChXmD8rUSy6nSGq8\",\n \"object\": + \"chat.completion\",\n \"created\": 1737568015,\n \"model\": \"gpt-4o-mini-2024-07-18\",\n + \ \"choices\": [\n {\n \"index\": 0,\n \"message\": {\n \"role\": + \"assistant\",\n \"content\": \"Why did the scarecrow win an award? \\n\\nBecause + he was outstanding in his field!\",\n \"refusal\": null\n },\n \"logprobs\": + null,\n \"finish_reason\": \"stop\"\n }\n ],\n \"usage\": {\n \"prompt_tokens\": + 12,\n \"completion_tokens\": 19,\n \"total_tokens\": 31,\n \"prompt_tokens_details\": + {\n \"cached_tokens\": 0,\n \"audio_tokens\": 0\n },\n \"completion_tokens_details\": + {\n \"reasoning_tokens\": 0,\n \"audio_tokens\": 0,\n \"accepted_prediction_tokens\": + 0,\n \"rejected_prediction_tokens\": 0\n }\n },\n \"service_tier\": + \"default\",\n \"system_fingerprint\": \"fp_72ed7ab54c\"\n}\n" + headers: + CF-Cache-Status: + - DYNAMIC + CF-RAY: + - 90615dc03b6c5cb1-RDU + Connection: + - keep-alive + Content-Encoding: + - gzip + Content-Type: + - application/json + Date: + - Wed, 22 Jan 2025 17:46:56 GMT + Server: + - cloudflare + Transfer-Encoding: + - chunked + X-Content-Type-Options: + - nosniff + access-control-expose-headers: + - X-Request-ID + alt-svc: + - h3=":443"; ma=86400 + openai-organization: + - crewai-iuxna1 + openai-processing-ms: + - '825' + openai-version: + - '2020-10-01' + strict-transport-security: + - max-age=31536000; includeSubDomains; preload + x-ratelimit-limit-requests: + - '30000' + x-ratelimit-limit-tokens: + - '150000000' + x-ratelimit-remaining-requests: + - '29999' + x-ratelimit-remaining-tokens: + - '149999979' + x-ratelimit-reset-requests: + - 2ms + x-ratelimit-reset-tokens: + - 0s + x-request-id: + - req_4c1485d44e7461396d4a7316a63ff353 + http_version: HTTP/1.1 + status_code: 200 +version: 1 diff --git a/tests/cassettes/test_llm_call_with_tool_and_message_list.yaml b/tests/cassettes/test_llm_call_with_tool_and_message_list.yaml new file mode 100644 index 0000000000..6102d9ef17 --- /dev/null +++ b/tests/cassettes/test_llm_call_with_tool_and_message_list.yaml @@ -0,0 +1,111 @@ +interactions: +- request: + body: '{"messages": [{"role": "user", "content": "What is the square of 5?"}], + "model": "gpt-4o-mini", "tools": [{"type": "function", "function": {"name": + "square_number", "description": "Returns the square of a number.", "parameters": + {"type": "object", "properties": {"number": {"type": "integer", "description": + "The number to square"}}, "required": ["number"]}}}]}' + headers: + accept: + - application/json + accept-encoding: + - gzip, deflate + connection: + - keep-alive + content-length: + - '361' + content-type: + - application/json + host: + - api.openai.com + user-agent: + - OpenAI/Python 1.59.6 + x-stainless-arch: + - arm64 + x-stainless-async: + - 'false' + x-stainless-lang: + - python + x-stainless-os: + - MacOS + x-stainless-package-version: + - 1.59.6 + x-stainless-raw-response: + - 'true' + x-stainless-retry-count: + - '0' + x-stainless-runtime: + - CPython + x-stainless-runtime-version: + - 3.12.7 + method: POST + uri: https://api.openai.com/v1/chat/completions + response: + content: "{\n \"id\": \"chatcmpl-AsZL5nGOaVpcGnDOesTxBZPHhMoaS\",\n \"object\": + \"chat.completion\",\n \"created\": 1737568919,\n \"model\": \"gpt-4o-mini-2024-07-18\",\n + \ \"choices\": [\n {\n \"index\": 0,\n \"message\": {\n \"role\": + \"assistant\",\n \"content\": null,\n \"tool_calls\": [\n {\n + \ \"id\": \"call_i6JVJ1KxX79A4WzFri98E03U\",\n \"type\": + \"function\",\n \"function\": {\n \"name\": \"square_number\",\n + \ \"arguments\": \"{\\\"number\\\":5}\"\n }\n }\n + \ ],\n \"refusal\": null\n },\n \"logprobs\": null,\n + \ \"finish_reason\": \"tool_calls\"\n }\n ],\n \"usage\": {\n \"prompt_tokens\": + 58,\n \"completion_tokens\": 15,\n \"total_tokens\": 73,\n \"prompt_tokens_details\": + {\n \"cached_tokens\": 0,\n \"audio_tokens\": 0\n },\n \"completion_tokens_details\": + {\n \"reasoning_tokens\": 0,\n \"audio_tokens\": 0,\n \"accepted_prediction_tokens\": + 0,\n \"rejected_prediction_tokens\": 0\n }\n },\n \"service_tier\": + \"default\",\n \"system_fingerprint\": \"fp_72ed7ab54c\"\n}\n" + headers: + CF-Cache-Status: + - DYNAMIC + CF-RAY: + - 906173d229b905f6-IAD + Connection: + - keep-alive + Content-Encoding: + - gzip + Content-Type: + - application/json + Date: + - Wed, 22 Jan 2025 18:02:00 GMT + Server: + - cloudflare + Set-Cookie: + - __cf_bm=BYDpIoqfPZyRxl9xcFxkt4IzTUGe8irWQlZ.aYLt8Xc-1737568920-1.0.1.1-Y_cVFN7TbguWRBorSKZynVY02QUtYbsbHuR2gR1wJ8LHuqOF4xIxtK5iHVCpWWgIyPDol9xOXiqUkU8xRV_vHA; + path=/; expires=Wed, 22-Jan-25 18:32:00 GMT; domain=.api.openai.com; HttpOnly; + Secure; SameSite=None + - _cfuvid=etTqqA9SBOnENmrFAUBIexdW0v2ZeO1x9_Ek_WChlfU-1737568920137-0.0.1.1-604800000; + path=/; domain=.api.openai.com; HttpOnly; Secure; SameSite=None + Transfer-Encoding: + - chunked + X-Content-Type-Options: + - nosniff + access-control-expose-headers: + - X-Request-ID + alt-svc: + - h3=":443"; ma=86400 + openai-organization: + - crewai-iuxna1 + openai-processing-ms: + - '642' + openai-version: + - '2020-10-01' + strict-transport-security: + - max-age=31536000; includeSubDomains; preload + x-ratelimit-limit-requests: + - '30000' + x-ratelimit-limit-tokens: + - '150000000' + x-ratelimit-remaining-requests: + - '29999' + x-ratelimit-remaining-tokens: + - '149999976' + x-ratelimit-reset-requests: + - 2ms + x-ratelimit-reset-tokens: + - 0s + x-request-id: + - req_388e63f9b8d4edc0dd153001f25388e5 + http_version: HTTP/1.1 + status_code: 200 +version: 1 diff --git a/tests/cassettes/test_llm_call_with_tool_and_string_input.yaml b/tests/cassettes/test_llm_call_with_tool_and_string_input.yaml new file mode 100644 index 0000000000..865d258269 --- /dev/null +++ b/tests/cassettes/test_llm_call_with_tool_and_string_input.yaml @@ -0,0 +1,107 @@ +interactions: +- request: + body: '{"messages": [{"role": "user", "content": "What is the current year?"}], + "model": "gpt-4o-mini", "tools": [{"type": "function", "function": {"name": + "get_current_year", "description": "Returns the current year as a string.", + "parameters": {"type": "object", "properties": {}, "required": []}}}]}' + headers: + accept: + - application/json + accept-encoding: + - gzip, deflate + connection: + - keep-alive + content-length: + - '295' + content-type: + - application/json + cookie: + - _cfuvid=8NrWEBP3dDmc8p2.csR.EdsSwS8zFvzWI1kPICaK_fM-1737568015338-0.0.1.1-604800000; + __cf_bm=pKr3NwXmTZN9rMSlKvEX40VPKbrxF93QwDNHunL2v8Y-1737568015-1.0.1.1-nR0EA7hYIwWpIBYUI53d9xQrUnl5iML6lgz4AGJW4ZGPBDxFma3PZ2cBhlr_hE7wKa5fV3r32eMu_rNWMXD.eA + host: + - api.openai.com + user-agent: + - OpenAI/Python 1.59.6 + x-stainless-arch: + - arm64 + x-stainless-async: + - 'false' + x-stainless-lang: + - python + x-stainless-os: + - MacOS + x-stainless-package-version: + - 1.59.6 + x-stainless-raw-response: + - 'true' + x-stainless-retry-count: + - '0' + x-stainless-runtime: + - CPython + x-stainless-runtime-version: + - 3.12.7 + method: POST + uri: https://api.openai.com/v1/chat/completions + response: + content: "{\n \"id\": \"chatcmpl-AsZJ8HKXQU9nTB7xbGAkKxqrg9BZ2\",\n \"object\": + \"chat.completion\",\n \"created\": 1737568798,\n \"model\": \"gpt-4o-mini-2024-07-18\",\n + \ \"choices\": [\n {\n \"index\": 0,\n \"message\": {\n \"role\": + \"assistant\",\n \"content\": null,\n \"tool_calls\": [\n {\n + \ \"id\": \"call_mfvEs2jngeFloVZpZOHZVaKY\",\n \"type\": + \"function\",\n \"function\": {\n \"name\": \"get_current_year\",\n + \ \"arguments\": \"{}\"\n }\n }\n ],\n + \ \"refusal\": null\n },\n \"logprobs\": null,\n \"finish_reason\": + \"tool_calls\"\n }\n ],\n \"usage\": {\n \"prompt_tokens\": 46,\n \"completion_tokens\": + 12,\n \"total_tokens\": 58,\n \"prompt_tokens_details\": {\n \"cached_tokens\": + 0,\n \"audio_tokens\": 0\n },\n \"completion_tokens_details\": {\n + \ \"reasoning_tokens\": 0,\n \"audio_tokens\": 0,\n \"accepted_prediction_tokens\": + 0,\n \"rejected_prediction_tokens\": 0\n }\n },\n \"service_tier\": + \"default\",\n \"system_fingerprint\": \"fp_72ed7ab54c\"\n}\n" + headers: + CF-Cache-Status: + - DYNAMIC + CF-RAY: + - 906170e038281775-IAD + Connection: + - keep-alive + Content-Encoding: + - gzip + Content-Type: + - application/json + Date: + - Wed, 22 Jan 2025 17:59:59 GMT + Server: + - cloudflare + Transfer-Encoding: + - chunked + X-Content-Type-Options: + - nosniff + access-control-expose-headers: + - X-Request-ID + alt-svc: + - h3=":443"; ma=86400 + openai-organization: + - crewai-iuxna1 + openai-processing-ms: + - '416' + openai-version: + - '2020-10-01' + strict-transport-security: + - max-age=31536000; includeSubDomains; preload + x-ratelimit-limit-requests: + - '30000' + x-ratelimit-limit-tokens: + - '150000000' + x-ratelimit-remaining-requests: + - '29999' + x-ratelimit-remaining-tokens: + - '149999975' + x-ratelimit-reset-requests: + - 2ms + x-ratelimit-reset-tokens: + - 0s + x-request-id: + - req_4039a5e5772d1790a3131f0b1ea06139 + http_version: HTTP/1.1 + status_code: 200 +version: 1 diff --git a/tests/llm_test.py b/tests/llm_test.py index 5513093898..6d1e6a188b 100644 --- a/tests/llm_test.py +++ b/tests/llm_test.py @@ -4,6 +4,7 @@ from crewai.agents.agent_builder.utilities.base_token_process import TokenProcess from crewai.llm import LLM +from crewai.tools import tool from crewai.utilities.token_counter_callback import TokenCalcHandler @@ -37,3 +38,119 @@ def test_llm_callback_replacement(): assert usage_metrics_1.successful_requests == 1 assert usage_metrics_2.successful_requests == 1 assert usage_metrics_1 == calc_handler_1.token_cost_process.get_summary() + + +@pytest.mark.vcr(filter_headers=["authorization"]) +def test_llm_call_with_string_input(): + llm = LLM(model="gpt-4o-mini") + + # Test the call method with a string input + result = llm.call("Return the name of a random city in the world.") + assert isinstance(result, str) + assert len(result.strip()) > 0 # Ensure the response is not empty + + +@pytest.mark.vcr(filter_headers=["authorization"]) +def test_llm_call_with_string_input_and_callbacks(): + llm = LLM(model="gpt-4o-mini") + calc_handler = TokenCalcHandler(token_cost_process=TokenProcess()) + + # Test the call method with a string input and callbacks + result = llm.call( + "Tell me a joke.", + callbacks=[calc_handler], + ) + usage_metrics = calc_handler.token_cost_process.get_summary() + + assert isinstance(result, str) + assert len(result.strip()) > 0 + assert usage_metrics.successful_requests == 1 + + +@pytest.mark.vcr(filter_headers=["authorization"]) +def test_llm_call_with_message_list(): + llm = LLM(model="gpt-4o-mini") + messages = [{"role": "user", "content": "What is the capital of France?"}] + + # Test the call method with a list of messages + result = llm.call(messages) + assert isinstance(result, str) + assert "Paris" in result + + +@pytest.mark.vcr(filter_headers=["authorization"]) +def test_llm_call_with_tool_and_string_input(): + llm = LLM(model="gpt-4o-mini") + + def get_current_year() -> str: + """Returns the current year as a string.""" + from datetime import datetime + + return str(datetime.now().year) + + # Create tool schema + tool_schema = { + "type": "function", + "function": { + "name": "get_current_year", + "description": "Returns the current year as a string.", + "parameters": { + "type": "object", + "properties": {}, + "required": [], + }, + }, + } + + # Available functions mapping + available_functions = {"get_current_year": get_current_year} + + # Test the call method with a string input and tool + result = llm.call( + "What is the current year?", + tools=[tool_schema], + available_functions=available_functions, + ) + + assert isinstance(result, str) + assert result == get_current_year() + + +@pytest.mark.vcr(filter_headers=["authorization"]) +def test_llm_call_with_tool_and_message_list(): + llm = LLM(model="gpt-4o-mini") + + def square_number(number: int) -> int: + """Returns the square of a number.""" + return number * number + + # Create tool schema + tool_schema = { + "type": "function", + "function": { + "name": "square_number", + "description": "Returns the square of a number.", + "parameters": { + "type": "object", + "properties": { + "number": {"type": "integer", "description": "The number to square"} + }, + "required": ["number"], + }, + }, + } + + # Available functions mapping + available_functions = {"square_number": square_number} + + messages = [{"role": "user", "content": "What is the square of 5?"}] + + # Test the call method with messages and tool + result = llm.call( + messages, + tools=[tool_schema], + available_functions=available_functions, + ) + + assert isinstance(result, int) + assert result == 25