Is there a way to clear trace data in batch?

[renwei-work]

Hi,

I want to apply retention policy to our self-host langfuse data. Something like only persists 1 months traces and discard all traces generated 1 month ago.
In trace tab, I see we can delete in batch but it requires manual deletion and batch size is 50 at maximum.
Does langfuse self-host currently support this?
Or can I just go into postgre db observations table to simply delete data their? But I think there are some foreign keys applied.

[dosubot[bot]]

Hey @renwei-work 👋 I'm here to help you out with any bugs, questions, or contributions you have. Let's tackle this together!

Currently, Langfuse does not provide built-in support for automated batch deletion of trace data based on a retention policy. You would need to manually manage data retention by running cleanup tasks on your PostgreSQL database. You can purge the trace and observation tables based on the created_at timestamp ^[1].

Regarding your concern about foreign keys, the observations table has a foreign key constraint that references the projects table. Specifically, it has a constraint that ensures referential integrity with the projects table, and it is set to cascade on delete and update operations ^[2]. This means that if you delete a project, the related observations will also be deleted, but you should be cautious when manually deleting data to maintain data integrity.

For now, you might consider writing a custom script to automate the deletion process based on your retention policy.

To continue talking to Dosu, mention @dosu.

[jannikmaierhoefer]

Hi @renwei-work, this is answered in this post: https://github.com/orgs/langfuse/discussions/2268#discussioncomment-10201896 :)