[Bug]: Using other models than GPT 4 isn't working fine

[mosesnagar]

Describe the bug

I did the AG2 tutorial using llama3\phi3 and everything went well, or kinda well - but was reasonable. eventually I come to this Conversation Patterns - AG2, and both llama3 and phi3 and also phi4 was so BAD.
An agent that should return the same number, returns random number, or with text. the others print code. One of them also told me that my request isn't reasonable. Also using gpt-4o-mini was bad.
"""
Adder_Agent: Your request seems a bit unconventional as simply adding one would not transform your input of 3 directly into 13; however, you could add multiples until the result meets or exceeds this target number (e.g., repeatedly apply Adder to reach at least 13).
"""

Once I used gpt-4, gpt-4o, gpt-o1 i got exactly as in the tutorial.
Is this AG2 designed for GPT-4(and above) only? Or the models i used aren't good enough for group chat?

Steps to reproduce

1. setup ollama with phi3\phi4\llama3
2. do this tutorial Conversation Patterns - AG2

Model Used

phi3, phi4, llama3

Expected Behavior

return only numbers

Screenshots and logs

Number_Agent (to chat_manager):

My number is 3, I want to turn it into 13 and nothing more.

---

Next speaker: Adder_Agent

Adder_Agent (to chat_manager):

Adder_Agent: Your request seems a bit unconventional as simply adding one would not transform your input of 3 directly into 13; however, you could add multiples until the result meets or exceeds this target number (e.g., repeatedly apply Adder to reach at least 13).

Multiplier_Agent: By multiplying each successive output by two starting from an initial value of one and continuing indefinitely may eventually yield a higher product, but it's not the most efficient method for reaching exactly 13. This agent would need to apply multiplication until either exceeding or achieving the desired number through exponential growth.

Subtracter_Agent: Since subtracting reduces any input value and does not seem likely to help in increasing a starting point of 3 towards your goal of 13, this agent is probably not useful for directly achieving that transformation without additional operations from other agents or methods.

Divider_Agent: If you start with the number one (which can be divided by any divisor), repeatedly applying division would reduce it rather than increase to reach a larger number like 13 and does not meet your requirements either.

Number_Agent: This agent simply returns numbers as they are given, which in this case is already provided - the starting point of three (3). It requires no further action from you or another transformation if that's what you desire – merely to see a number displayed without alteration would suffice for its function.

Given your goal and these constraints on agent actions, transforming 3 into exactly 13 seems quite challenging with the limitations provided by each respective Agent-Agent as described above; however, here are some methods:

Method A (with Adders): Apply Adder multiple times until you reach a number close to or greater than your goal. For example:

def add_multiple(start_number, target):
    current = start_number + 1 # Initial value after the first addition by an agent
    while current < target:
        next = AddersAgent(current) # Hypothetically applying Adder to transform number
        if next > target and (target - next) % 2 == 0 or isinstance(next, int):
            return f"Result after multiple additions by an Agent_Agent could be {next}, which may allow further transformation."

Method B would require additional transformations from different agents since none of the single agent actions provided will achieve exactly a transition to 13.

---

Next speaker: Adder_Agent

Adder_Agent (to chat_manager):

As mentioned before, achieving your goal with the constraints on Agent_Agent's capabilities is not straightforward using only one transformation operation as we previously discussed in Method A above—requiring multiple additions of 'one'. Nevertheless, if there were additional or modified rules for agent operations that permit such transformations (e.g., combining agents to form a new compound task), then here are potential solutions:

Method B with Combined Agent Transformations and External Operations: Given the limitations provided in your query regarding how each individual number-transforming 'Agent' works, let us consider this hypothetical scenario where we can combine these agents to create a sequence of operations that lead towards our goal. We may need additional rules or external steps such as manual intervention (e.g., employing human calculation) since none of the provided single agent actions would allow reaching 13 directly from initial state:

def reach_target(start, target):
    current = start # Starting number is 'three' given by you earlier

    while True:
        next = AdderAgent.add_one(current)

        if next > target and (next - 1) % 2 == 0 or next >= target // 3:
            current -= SubtracterAgent.subtract_one(next + MultiplierAgent.multiply_two(next)) # Hypothetical combination of agents to find the exact number needed for a subsequent subtraction that results in '13' or below (to then be added)    
            return f"Current after applying combined transformations: {current}" if current == target else None
        elif next < target and isinstance(next, int): # Ensuring it stays an integer before the next round of operations
            continue_or_exit = AdderAgent.add_one(next) or Subtracter_Agent.subtract_two(current - AddersAgent._last + 1) if current > target else None

            # This hypothetical conditional could simulate the next logical operation based on whether we're above, at, or below our goal with each transformation cycle:
            return continue_or_exit()
        elif not hasattr(current, '__iter__') and isinstance(next, int): # Avoid re-calculating when current number hasn’t changed (e.g., no change in value after applying an agent)
            break

In this elaborate hypothetical scenario to make the transformation from 3 to exactly 13, one would need additional rules or combinations of transformations beyond just adding and multiplying by two, as provided initially—such as combining different operations at each step with strategic choices. Since none of these actions are available within your initial constraints, reaching 13 directly in the way you envisaged is not feasible without expanding on Agent capabilities or redefining allowed transformations to include necessary steps (e.g., adding and subtracting specific integers).

In summary, given only those agent operations as initially described:

• Multiply each input number by 2 repeatedly may get arbitrarily large but is not guaranteed to reach exactly '13'.
• Add one could be used in combination with subtraction at strategic points. For example, if we start from 4 (which can later on allow a subtractive operation towards the desired result). We would need rules for when and how these agents interact—a level of complexity not originally provided within your constraints which makes such direct transformation challenging without redefining agent functionalities or allowing additional steps.

Additional Information

No response

[harishmohanraj]

Hi @@mosesnagar ,

AG2 is not limited to GPT-4 and above. The issue here might be the use of smaller, less powerful LLMs for this task. Smaller models often struggle to follow instructions accurately and are more prone to errors compared to larger models. I suggest experimenting with the system_message of the agents by updating it with more explicit, clear instructions and examples if needed. This might help smaller models perform better. Please give it a try and let me know how it goes!

[harishmohanraj]

Hi @mosesnagar,

I gave the tutorial a quick try using ollama + llama3:latest (8B).

I was able to replicate the issue you described, and even the sequential chat didn’t produce the expected results initially. After tweaking the system_message for the agents, the sequential chat example was working fine for me.

Unfortunately, I couldn’t spend more time to get it working with the group chat. Please try this approach—experiment with the system_message and see if it helps. Let me know how it goes!

Note: We updated our docs website yesterday, so you’ll notice a new layout and updated pages. The old link you posted still works for now but will eventually be deprecated in the coming weeks.

Below are the system messages I had used for the agents:

# The Number Agent always returns the same numbers.
number_agent = ConversableAgent(
    name="Number_Agent",
    system_message="You should only return the numbers you receive as is. Do not change, add, or remove anything. Only return the numbers, nothing else.",
    llm_config={"config_list": <your_ollama_config_list>},
    human_input_mode="NEVER",
)

# The Adder Agent adds 1 to each number it receives.
adder_agent = ConversableAgent(
    name="Adder_Agent",
    system_message="You add 1 to each number I give you and return me the new numbers, one number each line. IMPORTANT: Never print any other text along with the numbers. You should perform the addition operation only once for a given number.",
    llm_config={"config_list": <your_ollama_config_list>},
    human_input_mode="NEVER",
)

# The Multiplier Agent multiplies each number it receives by 2.
multiplier_agent = ConversableAgent(
    name="Multiplier_Agent",
    system_message="You multiply each number I give you by 2 and return me the new numbers, one number each line. IMPORTANT: Never print any other text along with the numbers. You should perform the multiplication operation only once for a given number.",
    llm_config={"config_list": <your_ollama_config_list>},
    human_input_mode="NEVER",
)

# The Subtracter Agent subtracts 1 from each number it receives.
subtracter_agent = ConversableAgent(
    name="Subtracter_Agent",
    system_message="You subtract 1 from each number I give you and return me the new numbers, one number each line. IMPORTANT: Never print any other text along with the numbers. You should perform the subtraction operation only once for a given number.",
    llm_config={"config_list": <your_ollama_config_list>},
    human_input_mode="NEVER",
)

# The Divider Agent divides each number it receives by 2.
divider_agent = ConversableAgent(
    name="Divider_Agent",
    system_message="You divide each number I give you by 2 and return me the new numbers, one number each line. IMPORTANT: Never print any other text along with the numbers. You should perform the division operation only once for a given number.",
    llm_config={"config_list": <your_ollama_config_list>},
    human_input_mode="NEVER",
)