Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Django command to load batch work embedding into ES #4560

Open
albertisfu opened this issue Oct 11, 2024 · 0 comments
Open

Django command to load batch work embedding into ES #4560

albertisfu opened this issue Oct 11, 2024 · 0 comments

Comments

@albertisfu
Copy link
Contributor

Once #4559 is completed. We'd require creating this command that pulls up embeddings from S3 and indexes them into ES using a bulk update operation with a reasonable batch size that ES can handle properly.

  • As part of this task, we can must tweak the Opinion index mapping to support the embeddings storage.

  • Evaluate ES workload to determine if the cluster can handle this load beside the regular index and other bulk tasks running related to the ES cluster.

The output of this issue would be:

  • A PR that includes the Django command to pulls up embeddings from S3 and index them to ES.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant