Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add binary Lucene vector updates for 2.19 #9102

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
+8 −2
Open

Add binary Lucene vector updates for 2.19 #9102

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions _search-plugins/knn/knn-index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,9 @@

Starting with k-NN plugin version 2.16, you can use `binary` vectors with the `faiss` engine to reduce the amount of required storage space. For more information, see [Binary vectors]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/knn-vector#binary-vectors).

Starting with k-NN plugin version 2.19 you can also use `binary` vectors with the `lucene` engine.
{: .note}

## SIMD optimization for the Faiss engine

Starting with version 2.13, the k-NN plugin supports [Single Instruction Multiple Data (SIMD)](https://en.wikipedia.org/wiki/Single_instruction,_multiple_data) processing if the underlying hardware supports SIMD instructions (AVX2 on x64 architecture and Neon on ARM64 architecture). SIMD is supported by default on Linux machines only for the Faiss engine. SIMD architecture helps boost overall performance by improving indexing throughput and reducing search latency. Starting with version 2.18, the k-NN plugin supports AVX512 SIMD instructions on x64 architecture.
Expand Down Expand Up @@ -155,9 +158,12 @@

### Supported Lucene methods

Method name | Requires training | Supported spaces | Description
Method name | Requires training | Supported spaces | Description
:--- | :--- |:--------------------------------------------------------------------------------| :---
`hnsw` | false | l2, cosinesimil, innerproduct (supported in OpenSearch 2.13 and later) | Hierarchical proximity graph approach to approximate k-NN search.
`hnsw` | false | l2, cosinesimil, innerproduct (supported in OpenSearch 2.13 and later), hamming | Hierarchical proximity graph approach to approximate k-NN search.

Check failure on line 163 in _search-plugins/knn/knn-index.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.Spelling] Error: cosinesimil. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.

Raw Output:
{"message": "[OpenSearch.Spelling] Error: cosinesimil. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_search-plugins/knn/knn-index.md", "range": {"start": {"line": 163, "column": 22}}}, "severity": "ERROR"}

Check failure on line 163 in _search-plugins/knn/knn-index.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.Spelling] Error: innerproduct. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.

Raw Output:
{"message": "[OpenSearch.Spelling] Error: innerproduct. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_search-plugins/knn/knn-index.md", "range": {"start": {"line": 163, "column": 35}}}, "severity": "ERROR"}

The `hamming` space type is supported for binary vectors in OpenSearch version 2.19 and later. For more information, see [Binary k-NN vectors]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/knn-vector#binary-vectors).
{: .note}

#### HNSW parameters

Expand Down
Loading