Ollama not working with current paperqa version, but fixed

[Exemocaro]

Hi, I wanted to use ollama locally with paperqa and the code provided in the main README is not working, as it tries connecting to OpenAI no matter what I do. I managed to fix it by adding the agent=AgentSettings(agent_llm='ollama/llama3.2', agent_llm_config=local_llm_config), line as discussed in this disccusion, but I also tried to add a dummy key and adding #"api_key": "sk-no-key-required", to the litellm parameters.

My environment looks like this:

python=3.12.3
My requirements: requirements.txt

My code:

from paperqa import Settings, ask
from paperqa.settings import AgentSettings

#OPENAI_API_KEY="foo"

local_llm_config = {
    "model_list": [
        {
            "model_name": "ollama/llama3.2",
            "litellm_params": {
                "model": "ollama/llama3.2",
                "api_base": "http://localhost:11434",
                #"api_key": "sk-no-key-required",
            },
        }
    ]
}

answer_response = ask(
    "What manufacturing challenges are unique to bispecific antibodies?",
    settings=Settings(
        agent=AgentSettings(agent_llm='ollama/llama3.2', agent_llm_config=local_llm_config),
        llm="ollama/llama3.2",
        llm_config=local_llm_config,
        summary_llm="ollama/llama3.2",
        summary_llm_config=local_llm_config,
        embedding="ollama/mxbai-embed-large",
    ),
)

[dosubot[bot]]

Hey @Exemocaro! I'm here to help you out with any bugs, questions, or contributions you have for the project. Let's get that issue sorted out.

It seems like you've encountered a common issue where PaperQA attempts to connect to OpenAI despite configuring it to use a local model. You mentioned that you resolved this by adding the agent=AgentSettings(agent_llm='ollama/llama3.2', agent_llm_config=local_llm_config) line, which aligns with the configuration needed to ensure local models are used properly.

For further reference, there are similar discussions on this topic, such as the Local LLM using Ollama without OpenAI thread, which is still open and might provide additional insights or solutions ^[1].

Your approach of adding a dummy API key and configuring the AgentSettings seems to be a practical workaround to prevent PaperQA from defaulting to OpenAI services. If you continue to face issues, reviewing the settings in the AgentSettings class and ensuring all configurations are correctly set to use local models might help ^[2].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}