text_completion output issues

[one-punch-dimash]

I have tried native LiteLLM library for text generation for text categorization based on the prompt instructions:

from lite_llm import text_completion

category = text_completion(
    model='openai/...',
    api_key='apikey123',
    temperature=0.2,
    max_retries=1,
    prompt=prompt_text
)

The output format is explained in the prompt, and must be a single name of a chosen category. LiteLLM text_completion gives weird responses, such as 3 different responses for Qwen, empty response text for Llama3.2, and the reason why weird is that I simultaneously tried to run this prompt on llama.index.llms.litellm's LiteLLM, and I get a correct response.

-------------Native LiteLLM-------------
'negative
positive
sarcastic'
-------------LlamaIndex LiteLLM-------------
'negative'

Correct answer is negative, however for models other than Qwen2.5 native LiteLLM gives an empty output.
Same prompt, same params, same models, but Llama index framework operates better than the native LiteLLM itself, which is weird (maybe I am wrong).

[krrishdholakia]

llama.index.llms.litellm's LiteLLM

llama index uses the same library 😄

[krrishdholakia]

@one-punch-dimash what do you see when running with litellm._turn_on_debug()

This should show the raw call being made to the llm api - which could help explain the difference

[one-punch-dimash]

Request to litellm:
litellm.completion(messages=[{'role': <MessageRole.USER: 'user'>, 'content': my_prompt}], stream=False, model='openai/Qwen2.5-72B-Instruct-GPTQ-Int8', temperature=0.2, api_key=API_KEY, api_base=API_BASE)

POST Request Sent from LiteLLM:

curl -X POST \
https://some_address.com/ \
-d '{'model': 'Qwen2.5-72B-Instruct-GPTQ-Int8', 'messages': [{'role': <MessageRole.USER: 'user'>, 'content': my_prompt}], 'temperature': 0.2, 'extra_body': {}}'

RAW RESPONSE:

{"id": "chatcmpl-...", "choices": [{"finish_reason": "stop", "index": 0, "logprobs": null, "message": {"content": "negative\nnegative\n\n### Output\nNegatives", "refusal": null, "role": "assistant", "audio": null, "function_call": null, "tool_calls": null}}], "created": 11111111, "model": "Qwen/Qwen2.5-72B-Instruct-GPTQ-Int8", "object": "chat.completion", "service_tier": null, "system_fingerprint": null, "usage": {"completion_tokens": 5, "prompt_tokens": 4034, "total_tokens": 4039, "completion_tokens_details": null, "prompt_tokens_details": null}, "prompt_logprobs": null}

I am getting text_completion response and seeing this:

print(response.choices[0])

Its' print text is:

TextChoices(stop_reason=None, prompt_logprobs=None, finish_reason='length', index=0, text='negative\nnegative\n\n### Output\nNegatives', logprobs=None)

The correct output should be 'negative' (case sensitive, i specify it in the prompt), however the model prints out several 'negatives'.

UPD: made some edits, for clarification