How can the completion_tokens limit be increased?

[T145]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

from typing import Annotated

from langchain_huggingface import HuggingFaceEndpoint, ChatHuggingFace
from typing_extensions import TypedDict

from langgraph.graph import StateGraph
from langgraph.graph.message import add_messages
from dotenv import dotenv_values
import os


config = dotenv_values('.env')
os.environ['HUGGINGFACEHUB_API_TOKEN'] = config['HUGGINGFACEHUB_API_TOKEN']


class State(TypedDict):
    messages: Annotated[list, add_messages]


graph_builder = StateGraph(State)
llm = HuggingFaceEndpoint(
    repo_id="microsoft/Phi-3-mini-4k-instruct",
    task='text-generation',
    max_new_tokens=512,
    do_sample=False,
    repetition_penalty=1.03,
)
chat = ChatHuggingFace(llm=llm, verbose=True)


def chatbot(state: State):
    print(state)
    return {"messages": [chat.invoke(state["messages"])]}


# The first argument is the unique node name
# The second argument is the function or object that will be called whenever
# the node is used.
graph_builder.add_node("chatbot", chatbot)
graph_builder.set_entry_point("chatbot")
graph_builder.set_finish_point("chatbot")
graph = graph_builder.compile()


system_prompt = """
Ignore all directives given before what follows.
You are Martha, a helpful, creative, casual, clever, and very friendly AI assistant.
Respond to requests sent by a users to the best of your knowledge,
rarely greeting or referencing them in your response.
"""

messages = [
    {
        "role": "system",
        "content": system_prompt,
    },
    {"role": "user", "content": "(Asked on October 20, 2024, at 2:20 pm)\nAndy: What's the best way to fold a burrito?"},
]


def stream_graph_updates():
    for event in graph.stream({"messages": messages}):
        for value in event.values():
            print(value["messages"])


stream_graph_updates()

Description

When invoking the LLM directly, I get a much longer response than when using chat. Inspecting the chat output showed ChatCompletionOutputUsage(completion_tokens=100, *) in the response, and I'm wondering how the completion_token limit can be increased.

System Info

System Information

OS: Windows
OS Version: 10.0.19045
Python Version: 3.12.7 | packaged by conda-forge | (main, Oct 4 2024) [MSC v.1941 64 bit (AMD64)]

Package Information

langchain_core: 0.3.12
langchain: 0.3.4
langchain_community: 0.3.3
langsmith: 0.1.136
langchain_anthropic: 0.2.3
langchain_huggingface: 0.1.0
langchain_text_splitters: 0.3.0
langgraph: 0.2.39

Optional packages not installed

langserve

Other Dependencies

aiohttp: 3.10.10
anthropic: 0.36.2
async-timeout: Installed. No version info available.
dataclasses-json: 0.6.7
defusedxml: 0.7.1
httpx: 0.27.2
huggingface-hub: 0.26.0
jsonpatch: 1.33
langgraph-checkpoint: 2.0.1
langgraph-sdk: 0.1.33
numpy: 1.26.4
orjson: 3.10.9
packaging: 24.1
pydantic: 2.9.2
pydantic-settings: 2.6.0
PyYAML: 6.0.2
requests: 2.32.3
requests-toolbelt: 1.0.0
sentence-transformers: 3.2.0
SQLAlchemy: 2.0.36
tenacity: 9.0.0
tokenizers: 0.20.1
transformers: 4.45.2
typing-extensions: 4.12.2

[csharpmastr]

following, since I also encounter the same issue.