v1.59.5 issue with Tracing on Langfuse v3.13.0 + prompt #7938

qymab · 2025-01-23T08:55:51Z

After upgrading from v1.57.1 to v1.59.5, prompt traces are no longer being recorded in Langfuse, despite using the same Langfuse version (v3.13.0).

Here are my LiteLLM settings (Docker-based):

model_list:
  # groq
  - model_name: llama-3.3-70b-versatile
    litellm_params:
      model: groq/llama-3.3-70b-versatile

# liteLLM settings
litellm_settings:
  drop_params: true
  cache: true
  cache_params:
    type: redis
  #   supported_call_types: ["acompletion", "completion", "embedding", "aembedding"]
  success_callback: ["langfuse"]
  # failure_callback: ["langfuse"]

The text was updated successfully, but these errors were encountered:

ishaan-jaff · 2025-01-23T11:57:10Z

What's the client side request? Can you share more details

ishaan-jaff · 2025-01-23T15:49:15Z

this workers for me @qymab

public trace i just sent on latest https://us.cloud.langfuse.com/project/clvlhdfat0007vwb74m9lvfvi/traces/051c83e1-7949-40e4-b8e6-e5d0da99f76e?timestamp=2025-01-23T15%3A48%3A18.371Z

qymab · 2025-01-23T21:12:41Z

This is a docker logs for litellm container.

httpx.ConnectTimeout: timed out
 - LiteLLM:WARNING: logging_utils.py:113 - `logging_obj` not found - unable to track `llm_api_duration_ms
INFO:     52538 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     52540 - "POST /v1/chat/completions HTTP/1.1" 200 OK
 - LiteLLM:WARNING: logging_utils.py:113 - `logging_obj` not found - unable to track `llm_api_duration_ms
 - LiteLLM:WARNING: logging_utils.py:113 - `logging_obj` not found - unable to track `llm_api_duration_ms
INFO:     52554 - "POST /v1/chat/completions HTTP/1.1" 200 OK
 - LiteLLM:WARNING: logging_utils.py:113 - `logging_obj` not found - unable to track `llm_api_duration_ms
INFO:     52564 - "POST /v1/chat/completions HTTP/1.1" 200 OK
Giving up execute_task_with_backoff(...) after 3 tries (httpx.ConnectTimeout: timed out)
Giving up execute_task_with_backoff(...) after 3 tries (httpx.ConnectTimeout: timed out)
Giving up execute_task_with_backoff(...) after 3 tries (httpx.ConnectTimeout: timed out)
error uploading: timed out
Traceback (most recent call last):
  File "/usr/lib/python3.13/site-packages/httpx/_transports/default.py", line 69, in map_httpcore_exceptions
    yield
  File "/usr/lib/python3.13/site-packages/httpx/_transports/default.py", line 233, in handle_request
    resp = self._pool.handle_request(req)
  File "/usr/lib/python3.13/site-packages/httpcore/_sync/connection_pool.py", line 256, in handle_request
    raise exc from None
  File "/usr/lib/python3.13/site-packages/httpcore/_sync/connection_pool.py", line 236, in handle_request
    response = connection.handle_request(
        pool_request.request
    )
  File "/usr/lib/python3.13/site-packages/httpcore/_sync/connection.py", line 101, in handle_request
    raise exc
  File "/usr/lib/python3.13/site-packages/httpcore/_sync/connection.py", line 78, in handle_request
    stream = self._connect(request)
  File "/usr/lib/python3.13/site-packages/httpcore/_sync/connection.py", line 124, in _connect
    stream = self._network_backend.connect_tcp(**kwargs)
  File "/usr/lib/python3.13/site-packages/httpcore/_backends/sync.py", line 207, in connect_tcp
    with map_exceptions(exc_map):
         ~~~~~~~~~~~~~~^^^^^^^^^
  File "/usr/lib/python3.13/contextlib.py", line 162, in __exit__
    self.gen.throw(value)
    ~~~~~~~~~~~~~~^^^^^^^
  File "/usr/lib/python3.13/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ConnectTimeout: timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.13/site-packages/langfuse/task_manager.py", line 204, in upload
    self._upload_batch(batch)
    ~~~~~~~~~~~~~~~~~~^^^^^^^
  File "/usr/lib/python3.13/site-packages/langfuse/task_manager.py", line 244, in _upload_batch
    execute_task_with_backoff(batch)
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^
  File "/usr/lib/python3.13/site-packages/backoff/_sync.py", line 105, in retry
    ret = target(*args, **kwargs)
  File "/usr/lib/python3.13/site-packages/langfuse/task_manager.py", line 242, in execute_task_with_backoff
    raise e
  File "/usr/lib/python3.13/site-packages/langfuse/task_manager.py", line 230, in execute_task_with_backoff
    self._client.batch_post(batch=batch, metadata=metadata)
    ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/site-packages/langfuse/request.py", line 55, in batch_post
    res = self.post(**kwargs)
  File "/usr/lib/python3.13/site-packages/langfuse/request.py", line 67, in post
    res = self._session.post(
        url, content=data, headers=headers, timeout=self._timeout
    )
  File "/usr/lib/python3.13/site-packages/httpx/_client.py", line 1145, in post
    return self.request(
           ~~~~~~~~~~~~^
        "POST",
        ^^^^^^^
    ...<11 lines>...
        extensions=extensions,
        ^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/lib/python3.13/site-packages/httpx/_client.py", line 827, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
           ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/site-packages/httpx/_client.py", line 914, in send
    response = self._send_handling_auth(
        request,
    ...<2 lines>...
        history=[],
    )
  File "/usr/lib/python3.13/site-packages/httpx/_client.py", line 942, in _send_handling_auth
    response = self._send_handling_redirects(
        request,
        follow_redirects=follow_redirects,
        history=history,
    )
  File "/usr/lib/python3.13/site-packages/httpx/_client.py", line 979, in _send_handling_redirects
    response = self._send_single_request(request)
  File "/usr/lib/python3.13/site-packages/httpx/_client.py", line 1015, in _send_single_request
    response = transport.handle_request(request)
  File "/usr/lib/python3.13/site-packages/httpx/_transports/default.py", line 232, in handle_request
    with map_httpcore_exceptions():
         ~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/lib/python3.13/contextlib.py", line 162, in __exit__
    self.gen.throw(value)
    ~~~~~~~~~~~~~~^^^^^^^
  File "/usr/lib/python3.13/site-packages/httpx/_transports/default.py", line 86, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectTimeout: timed out

ishaan-jaff · 2025-01-24T05:39:09Z

does this issue go away when you downgrade to v1.57.1 ?

is this with self hosted langfuse @qymab ?

qymab · 2025-01-24T05:46:14Z

Yes, I have also noticed that the startup is slower after updating to above v1.57.1 image. With the newer version, I have to wait about 5 minutes after Docker starts before it is ready to accept prompts. I am using both LibreChat and Open WebUI for testing purposes.
Yes, I am using a self-hosted setup.

motin · 2025-01-24T08:27:10Z

I also noticed that langfuse tracing stopped working after upgrading to 1.59.5 - not using the proxy, just the litellm langfuse integration.

Recent commits related to logging that may be related:

motin · 2025-01-24T08:42:39Z

Btw, in litellm/utils.py, I notice that dynamic_success_callbacks is None now.
Langfuse package is v2.57.12 here, so it doesn't seem to be related to Langfuse v3.13.0.
Downgrading to 1.59.3 makes langfuse tracing work again (there is no release of 1.59.4)

The langfuse integration that works on 1.59.3 but not on 1.59.5 is simply:

litellm.success_callback = ["langfuse"]
litellm.failure_callback = ["langfuse"]

motin · 2025-01-24T17:06:07Z

May not be related, but even in 1.59.3, the langfuse callback is no longer executed when using context_window_fallback_dict and the fallback is activated (e.g. the model's context window is too small). If you happen to be improving test coverage for success callbacks, include this too in the test harness :)

ishaan-jaff · 2025-01-24T17:08:17Z

@motin - can you give me a way to repro ?

ishaan-jaff · 2025-01-24T17:09:22Z

ok - I will try with this

litellm.success_callback = ["langfuse"]
litellm.failure_callback = ["langfuse"]

ishaan-jaff · 2025-01-24T17:31:33Z

@motin - can you share a reproducible code snippet ? this seems to be working for me

what langfuse python sdk version are you using ?

motin · 2025-01-24T19:59:07Z

@ishaan-jaff Langfuse package is v2.57.12

Here is the shortest possible repro code I could get:

import os

os.environ["LANGFUSE_PUBLIC_KEY"] = "pk-lf-..."
os.environ["LANGFUSE_SECRET_KEY"] = "sk-lf-..."
os.environ["LANGFUSE_HOST"] = "https://our-self-hosted-domain" # Running Langfuse 2
os.environ["OPENAI_API_KEY"] = "..."
os.environ["LITELLM_LOG"] = "DEBUG"

from litellm import completion 
import litellm

litellm.success_callback = ["langfuse"]
litellm.failure_callback = ["langfuse"]

model_response = completion(
        model="gpt-4o-mini", 
        messages=[
           {"role": "user", "content": "hello from litellm"}
       ],
        stream=False,
)
print(model_response)

This code results in a working langfuse trace on 1.59.3 but not on 1.59.5

motin · 2025-01-24T20:05:55Z

And here is repro code for the other case, which also fails in 1.59.3:

This results in a langfuse trace:

model_response = completion(
        model="gpt-3.5-turbo", 
        messages=[
           {"role": "user", "content": "hello from litellm\n" * 500} # About 3k tokens, so it is good for for gpt-3.5-turbo
       ],
        stream=False,
)
model_response

As does this:

model_response = completion(
        model="gpt-4o-mini", 
        messages=[
           {"role": "user", "content": "hello from litellm\n" * 5000} # About 30k tokens, so it is good for for gpt-4o-mini
       ],
        stream=False,
)
model_response

But this doesn't, even though it completes fine, e.g. the fallback mechanism works, just not the langfuse trace:

model_response = completion(
        model="gpt-3.5-turbo", 
        messages=[
           {"role": "user", "content": "hello from litellm\n" * 5000} # About 30k tokens, so it is too large for gpt-3.5-turbo
       ],
        context_window_fallback_dict={
            "gpt-3.5-turbo": "gpt-4o-mini",
        },
        stream=False,
)
model_response

Let me know if this ought to be reported as a separate issue, not related to the other one.

krrishdholakia · 2025-01-25T22:53:03Z

testing with

litellm 1.59.7
langfuse 2.57.12

i'm able to repro this issue

krrishdholakia · 2025-01-25T23:07:40Z

I am able to repro the @motin issue - it was only for sync calls, as it had to do with a missing implementation for the sync success event on the new customlogger version of langfuse - updating our testing for this as well.

@qymab your call works fine for me

tested via proxy + clean google colab (screenshots below)

…alls Resolves #7938 (comment)

krrishdholakia · 2025-01-25T23:23:26Z

Unable to repro your issue @qymab - just ran your exact config - and i can see it logged to langfuse as well

REGARDING YOUR ERROR

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.13/site-packages/langfuse/task_manager.py", line 204, in upload
    self._upload_batch(batch)
    ~~~~~~~~~~~~~~~~~~^^^^^^^
  File "/usr/lib/python3.13/site-packages/langfuse/task_manager.py", line 244, in _upload_batch
    execute_task_with_backoff(batch)
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^
  File "/usr/lib/python3.13/site-packages/backoff/_sync.py", line 105, in retry
    ret = target(*args, **kwargs)
  File "/usr/lib/python3.13/site-packages/langfuse/task_manager.py", line 242, in execute_task_with_backoff
    raise e
  File "/usr/lib/python3.13/site-packages/langfuse/task_manager.py", line 230, in execute_task_with_backoff
    self._client.batch_post(batch=batch, metadata=metadata)
    ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/site-packages/langfuse/request.py", line 55, in batch_post
    res = self.post(**kwargs)
  File "/usr/lib/python3.13/site-packages/langfuse/request.py", line 67, in post
    res = self._session.post(
        url, content=data, headers=headers, timeout=self._timeout
    )
  File "/usr/lib/python3.13/site-packages/httpx/_client.py", line 1145, in post
    return self.request(
           ~~~~~~~~~~~~^
        "POST",
        ^^^^^^^
    ...<11 lines>...
        extensions=extensions,
        ^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/lib/python3.13/site-packages/httpx/_client.py", line 827, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
           ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/site-packages/httpx/_client.py", line 914, in send
    response = self._send_handling_auth(
        request,
    ...<2 lines>...
        history=[],
    )
  File "/usr/lib/python3.13/site-packages/httpx/_client.py", line 942, in _send_handling_auth
    response = self._send_handling_redirects(
        request,
        follow_redirects=follow_redirects,
        history=history,
    )
  File "/usr/lib/python3.13/site-packages/httpx/_client.py", line 979, in _send_handling_redirects
    response = self._send_single_request(request)
  File "/usr/lib/python3.13/site-packages/httpx/_client.py", line 1015, in _send_single_request
    response = transport.handle_request(request)
  File "/usr/lib/python3.13/site-packages/httpx/_transports/default.py", line 232, in handle_request
    with map_httpcore_exceptions():
         ~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/lib/python3.13/contextlib.py", line 162, in __exit__
    self.gen.throw(value)
    ~~~~~~~~~~~~~~^^^^^^^
  File "/usr/lib/python3.13/site-packages/httpx/_transports/default.py", line 86, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectTimeout: timed out

looks like your issue was caused when writing the log to langfuse, perhaps there was an error on the receiving server

* fix(base_utils.py): supported nested json schema passed in for anthropic calls * refactor(base_utils.py): refactor ref parsing to prevent infinite loop * test(test_openai_endpoints.py): refactor anthropic test to use bedrock * fix(langfuse_prompt_management.py): add unit test for sync langfuse calls Resolves #7938 (comment)

qymab · 2025-01-26T03:01:32Z

I opted to use the Docker container's hostname as the host instead of relying on the public URL

motin · 2025-01-27T07:31:00Z

@krrishdholakia Thanks! I did however notice that upgrading to 1.59.7 actually resolved my first issue, even without your fix included. Not sure if you referred to fixing the second issue related to context fallback dict, but I didn't seen the fallback in the new tests so I reported it separately here: #8014 - if already resolved by your fix, please close it with a comment stating as such.

ishaan-jaff changed the title ~~Upgrading to v1.59.5 Breaks Tracing with Langfuse v3.13.0~~ v1.59.5 issue with Tracing on Langfuse v3.13.0 + prompt Jan 23, 2025

motin mentioned this issue Jan 24, 2025

Langfuse client is disabled #7948

Open

krrishdholakia self-assigned this Jan 25, 2025

krrishdholakia added a commit that referenced this issue Jan 25, 2025

fix(langfuse_prompt_management.py): add unit test for sync langfuse c…

254fc81

…alls Resolves #7938 (comment)

krrishdholakia closed this as completed Jan 25, 2025

krrishdholakia mentioned this issue Jan 26, 2025

Supported nested json schema on anthropic calls via proxy + fix langfuse sync sdk issues #8003

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

v1.59.5 issue with Tracing on Langfuse v3.13.0 + prompt #7938

v1.59.5 issue with Tracing on Langfuse v3.13.0 + prompt #7938

qymab commented Jan 23, 2025

ishaan-jaff commented Jan 23, 2025

ishaan-jaff commented Jan 23, 2025

qymab commented Jan 23, 2025

ishaan-jaff commented Jan 24, 2025

qymab commented Jan 24, 2025

motin commented Jan 24, 2025 •

edited

Loading

motin commented Jan 24, 2025

motin commented Jan 24, 2025

ishaan-jaff commented Jan 24, 2025

ishaan-jaff commented Jan 24, 2025

ishaan-jaff commented Jan 24, 2025

motin commented Jan 24, 2025

motin commented Jan 24, 2025

krrishdholakia commented Jan 25, 2025

krrishdholakia commented Jan 25, 2025 •

edited

Loading

krrishdholakia commented Jan 25, 2025 •

edited

Loading

qymab commented Jan 26, 2025

motin commented Jan 27, 2025

v1.59.5 issue with Tracing on Langfuse v3.13.0 + prompt #7938

v1.59.5 issue with Tracing on Langfuse v3.13.0 + prompt #7938

Comments

qymab commented Jan 23, 2025

ishaan-jaff commented Jan 23, 2025

ishaan-jaff commented Jan 23, 2025

qymab commented Jan 23, 2025

ishaan-jaff commented Jan 24, 2025

qymab commented Jan 24, 2025

motin commented Jan 24, 2025 • edited Loading

motin commented Jan 24, 2025

motin commented Jan 24, 2025

ishaan-jaff commented Jan 24, 2025

ishaan-jaff commented Jan 24, 2025

ishaan-jaff commented Jan 24, 2025

motin commented Jan 24, 2025

motin commented Jan 24, 2025

krrishdholakia commented Jan 25, 2025

krrishdholakia commented Jan 25, 2025 • edited Loading

krrishdholakia commented Jan 25, 2025 • edited Loading

REGARDING YOUR ERROR

qymab commented Jan 26, 2025

motin commented Jan 27, 2025

motin commented Jan 24, 2025 •

edited

Loading

krrishdholakia commented Jan 25, 2025 •

edited

Loading

krrishdholakia commented Jan 25, 2025 •

edited

Loading