Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ES Signal processor tweaks to index embeddings #4561

Open
albertisfu opened this issue Oct 11, 2024 · 0 comments
Open

ES Signal processor tweaks to index embeddings #4561

albertisfu opened this issue Oct 11, 2024 · 0 comments

Comments

@albertisfu
Copy link
Contributor

albertisfu commented Oct 11, 2024

This task consists of tweaking the ES Signal processor so that opinion text embeddings can be indexed or updated in Elasticsearch when new documents are added or updated after the initial batch processing.

This can be done after #4559 since it can reuse the Celery task used for the command.

  • This will require tweaking the ES signal processor specifically on the save/update methods.
  • So if an Opinion is saved and it contains content for the text field or if it's updated and the text field changed, we should include the Celery task to request embedding generation for this opinion text to the chain as the first task to execute.
  • The response of this task should be passed to the regular save/update ES task which will include other fields + the embeddings.
  • Include tests to confirm the integration works properly.

The output of this issue would be:

A PR that tweaks the ES Signal processor to request embeddings to the microservice and index them to ES.

Image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant