Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] KNN doesn't release memory when close index #1012

Closed
ylwu-amzn opened this issue Jul 28, 2023 · 3 comments
Closed

[BUG] KNN doesn't release memory when close index #1012

ylwu-amzn opened this issue Jul 28, 2023 · 3 comments
Assignees
Labels
bug Something isn't working

Comments

@ylwu-amzn
Copy link

Run big KNN indexes, and see memory usage keep high as 95%. Close one big KNN index, but memory usage still keep high.
Have to delete the KNN index to release the memory. After deleting the index, can see memory usage goes down to 75%.

Seems like a bug that when close an index, KNN doesn't release the memory.

@ylwu-amzn ylwu-amzn added bug Something isn't working untriaged labels Jul 28, 2023
@jmazanec15
Copy link
Member

@ylwu-amzn Thanks for reporting this. Yes, the problem is that we do watch for the files to be deleted to free memory as opposed to freeing memory when the IndexReader gets closed.

I think a fix for this is to implement a DocValuesReader that frees memory when it is closed.

@vamshin vamshin moved this from Backlog to Backlog (Hot) in Vector Search RoadMap Sep 15, 2023
@vamshin vamshin moved this from Backlog (Hot) to 2.12.0 in Vector Search RoadMap Oct 5, 2023
@vamshin vamshin moved this from 2.12.0 to 2.13.0 in Vector Search RoadMap Nov 17, 2023
@vamshin vamshin moved this from 2.13.0 to 2.14.0 in Vector Search RoadMap Mar 27, 2024
@vamshin vamshin added the v2.15.0 label Apr 1, 2024
@vamshin vamshin moved this from 2.14.0 to 2.15.0 in Vector Search RoadMap Apr 1, 2024
@vamshin vamshin moved this from 2.15.0 to Backlog (Hot) in Vector Search RoadMap Jul 2, 2024
@vamshin vamshin moved this from Backlog (Hot) to 2.17.0 in Vector Search RoadMap Jul 2, 2024
@naveentatikonda naveentatikonda moved this from 2.17.0 to Backlog (Hot) in Vector Search RoadMap Sep 18, 2024
@jmazanec15
Copy link
Member

Addressed in 2.18 with #2182

@luyuncheng
Copy link
Collaborator

@jmazanec15 After #1946, when close the indices, it would release the memory.

@vamshin vamshin closed this as completed Oct 31, 2024
@github-project-automation github-project-automation bot moved this from Backlog (Hot) to ✅ Done in Vector Search RoadMap Oct 31, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Status: Done
Development

No branches or pull requests

6 participants