Granite 3.1 MoE tries to guess the result itself instead of using a tool

[akusok]

Playing around Granite models with Langchain and a custom tool. Granite 3.1 dense correctly uses the tool to do math. Granite 3.1 MoE stubbornly tries to approximate the result in its "head" (often incorrectly) and never calls a tool it is supposed to.
Thought you may want to know about this behavior.

Sample code:

@tool
def calculator(expression: str) -> str:
    """Calculator that runs mathetical expressions in `numexpr` and returns their result.

    Expression should be a single line mathematical expression
    that solves the problem.

    Examples:
        "37593 * 67" for "37593 times 67"
        "37593**(1/5)" for "37593^(1/5)"
    """
    local_dict = {"pi": math.pi, "e": math.e}
    return str(
        numexpr.evaluate(
            expression.strip(),
            global_dict={},  # restrict access to globals
            local_dict=local_dict,  # add common mathematical functions
        )
    )

tools = [calculator]

chat_model = ChatOllama(model="granite3.1-moe")
llm_with_tools = chat_model.bind_tools(tools)

# %%

query = "What is 3 * 12? Also, what is 112367 + 26449 / 2462?"

messages = [HumanMessage(query)]

# %%

result = llm_with_tools.invoke(messages)
messages.append(result)

print(result.tool_calls)  # granite MoE does not do any calls

# %%
# Generating tool messages from calls

for tool_call in result.tool_calls:
    tool_msg = calculator.invoke(tool_call)
    messages.append(tool_msg)

# %%
# Re-invoke the model with the tool messages

print(llm_with_tools.invoke(messages).content)

Example Granite MoE output with guessed answer

Correct Granite dense output

Exact models obtained from Ollama website

[akusok]

Granite MoE gives different guesses and sometimes explains how it computes the result in its thought process, but stable never calls the tool