Update default engine to FAISS #2221

VijayanB · 2024-10-20T03:05:56Z

Description

Update default engine to FAISS.
Fixed tests that assumed default engine to be nmslib to either use nmslib explicitly or update test assertion to look for faiss specific artifacts like file extension instead of nmslib artifacts
removed tests that are just validating default (provided it is nmslib) doesn't support certain features

Related Issues

Check List

New functionality includes testing.
New functionality has been documented.
API changes companion pull request created.
Commits are signed per the DCO using --signoff.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

VijayanB · 2024-10-20T22:37:08Z

Doc issue: opensearch-project/documentation-website#8589

vibrantvarun

Looks good! Will wait for other engineers review

jmazanec15 · 2024-10-21T16:32:37Z

src/main/java/org/opensearch/knn/index/mapper/KNNVectorFieldMapperUtil.java

@@ -203,7 +203,7 @@ static KNNMethodContext createKNNMethodContextFromLegacy(
            ? topLevelSpaceType
            : KNNVectorFieldMapperUtil.getSpaceType(indexSettings);
        return new KNNMethodContext(
-            KNNEngine.NMSLIB,


Keep as NMSLIB - nmslib is only legacy support

maybe add comment to like I should have done haha

jmazanec15 · 2024-10-21T16:34:21Z

Can you update this too: https://github.com/opensearch-project/k-NN/blob/main/src/main/java/org/opensearch/knn/index/engine/EngineResolver.java#L51

navneet1v · 2024-10-21T17:19:05Z

src/main/java/org/opensearch/knn/index/engine/KNNEngine.java

@@ -29,7 +29,7 @@ public enum KNNEngine implements KNNLibrary {
    FAISS(FAISS_NAME, Faiss.INSTANCE),
    LUCENE(LUCENE_NAME, Lucene.INSTANCE);

-    public static final KNNEngine DEFAULT = NMSLIB;
+    public static final KNNEngine DEFAULT = FAISS;


How we are handling the backward compatibility? Also can we validate if we have proper BWC tests to test this default engine change.

I was just curious to know , why do we need to handle BWC in this case , So i created a Index , without specifying any engine so it picked NMSLIB , so now this information is already there in Cluster State and Serialised and passed to all Nodes, Now During query time , we do get it information from there and it does not depend on the new Default.

So i created a Index , without specifying any engine so it picked NMSLIB , so now this information is already there in Cluster State and Serialised and passed to all Nodes,

The engine value gets persisted in the cluster state if you define the key method while creating the index. But if you don't specify the key method while creating the index(also called as LegacyFieldMapping) and lets say create an index like this where my_vector1 doesn't have the engine defined:

PUT my-knn-index-1 { "settings": { "index": { "knn": true } }, "mappings": { "properties": { "my_vector1": { "type": "knn_vector", "dimension": 2 }, "my_vector2": { "type": "knn_vector", "dimension": 2, "method": { "name": "hnsw" } } } } }

then k-NN plugin infers the engine. Ref: https://github.com/opensearch-project/k-NN/blob/HEAD/src/main/java/org/opensearch/knn/index/mapper/KNNVectorFieldMapperUtil.java#L196-L219 for more details. If you try doing a getMapping on the index after creating an index like this you will never get the engine value in return for my_vector1 but you will get engine as nmslib for my_vector2. Same is true for cluster state.

shatejas · 2024-10-21T17:45:51Z

src/test/java/org/opensearch/knn/index/KNNIndexShardTests.java

@@ -114,7 +115,7 @@ public void testGetAllEngineFileContexts() throws IOException, ExecutionExceptio
        engineFileContexts = knnIndexShard.getAllEngineFileContexts(searcher.getIndexReader());
        assertEquals(1, engineFileContexts.size());
        List<String> paths = engineFileContexts.stream().map(KNNIndexShard.EngineFileContext::getIndexPath).collect(Collectors.toList());
-        assertTrue(paths.get(0).contains("hnsw") || paths.get(0).contains("hnswc"));
+        assertTrue(paths.get(0).contains(FAISS_HNSW_EXTENSION));


is the compound extension covered?

Updated test to use nmslib

shatejas · 2024-10-21T17:47:23Z

src/test/java/org/opensearch/knn/index/VectorDataTypeIT.java

@@ -263,29 +263,6 @@ public void testByteVectorDataTypeWithNmslibEngine() {
        assertTrue(ex.getMessage().contains("is not supported for vector data type"));
    }

-    @SneakyThrows
-    public void testByteVectorDataTypeWithLegacyFieldMapperKnnIndexSetting() {


Just checking if this case is covered when engine is explicitly NMSLIB

Yes, it is covered explicitly for nmslib engine

shatejas · 2024-10-21T17:48:25Z

src/test/java/org/opensearch/knn/index/mapper/KNNVectorFieldMapperTests.java

@@ -146,7 +146,7 @@ public void testTypeParser_build_fromKnnMethodContext() throws IOException {
        // Check that knnMethodContext takes precedent over both model and legacy
        ModelDao modelDao = mock(ModelDao.class);

-        SpaceType spaceType = SpaceType.COSINESIMIL;
+        SpaceType spaceType = SpaceType.DEFAULT;


Out of curiosity, why this change?

@vamshin is it okay default engine doesnt support cosine?

We could add this by taking a normalization and then setting to innerproduct. But this would take some effort.

ah good catch! I think still we need to make faiss as default based on all the advanced capabilities like nested fields, efficient filters, disk optimized search, in future GPUs. In long term makes sense to build the normalization to enable cosine.

@shatejas FAISS doesn't support cosine yet.

fail it

VijayanB · 2024-10-24T23:53:44Z

Tests are failing not related to this PR. This PR #2236 will fix those issue in main. Will rebase once the other PR is merged.

Since faiss supports more features than nmslib, and, we had seen data points that there are more number of vector search users are interesed in faiss, we will be updating default engine to be faiss. This will benefit users who preffered to use defaults while working with vector search. Signed-off-by: Vijayan Balasubramanian <[email protected]>

Signed-off-by: Vijayan Balasubramanian <[email protected]>

VijayanB · 2024-11-07T19:22:29Z

Test failure is not due to this change.

opensearch-trigger-bot · 2024-11-07T19:41:14Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.x 2.x
# Navigate to the new working tree
cd .worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-2221-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 7d3445631591296d00c37ea16351a07ca08ffbd3
# Push it to GitHub
git push --set-upstream origin backport/backport-2221-to-2.x
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-2221-to-2.x.

VijayanB · 2024-11-07T19:43:25Z

Failure is due to waiting for this PR to be merged #2248

* Update default engine to FAISS Since faiss supports more features than nmslib, and, we had seen data points that there are more number of vector search users are interesed in faiss, we will be updating default engine to be faiss. This will benefit users who preffered to use defaults while working with vector search. Signed-off-by: Vijayan Balasubramanian <[email protected]> * Update legacy mapping Signed-off-by: Vijayan Balasubramanian <[email protected]> * Create legacy mapping only up to V_2_17_2 Signed-off-by: Vijayan Balasubramanian <[email protected]> * Update test engine Signed-off-by: Vijayan Balasubramanian <[email protected]> * Update test method Signed-off-by: Vijayan Balasubramanian <[email protected]> --------- Signed-off-by: Vijayan Balasubramanian <[email protected]> (cherry picked from commit 7d34456)

This reverts commit 7d34456.

* Update default engine to FAISS Since faiss supports more features than nmslib, and, we had seen data points that there are more number of vector search users are interesed in faiss, we will be updating default engine to be faiss. This will benefit users who preffered to use defaults while working with vector search. Signed-off-by: Vijayan Balasubramanian <[email protected]> * Update legacy mapping Signed-off-by: Vijayan Balasubramanian <[email protected]> * Create legacy mapping only up to V_2_17_2 Signed-off-by: Vijayan Balasubramanian <[email protected]> * Update test engine Signed-off-by: Vijayan Balasubramanian <[email protected]> * Update test method Signed-off-by: Vijayan Balasubramanian <[email protected]> --------- Signed-off-by: Vijayan Balasubramanian <[email protected]> (cherry picked from commit 7d34456) Co-authored-by: Vijayan Balasubramanian <[email protected]>

VijayanB requested review from heemin32, navneet1v, vamshin, jmazanec15, naveentatikonda, junqiu-lei, martin-gaievski, ryanbogan, luyuncheng and shatejas as code owners October 20, 2024 03:05

VijayanB self-assigned this Oct 20, 2024

VijayanB marked this pull request as draft October 20, 2024 03:06

VijayanB force-pushed the update-default-engine branch 6 times, most recently from 97240f3 to daf89f6 Compare October 20, 2024 22:00

VijayanB marked this pull request as ready for review October 20, 2024 22:32

VijayanB mentioned this pull request Oct 20, 2024

[DOC] Update default value for engine in k-NN index opensearch-project/documentation-website#8589

Closed

4 tasks

vibrantvarun reviewed Oct 21, 2024

View reviewed changes

jmazanec15 reviewed Oct 21, 2024

View reviewed changes

navneet1v reviewed Oct 21, 2024

View reviewed changes

shatejas reviewed Oct 21, 2024

View reviewed changes

VijayanB force-pushed the update-default-engine branch from aa34e58 to b5c37b9 Compare October 22, 2024 22:32

VijayanB added the skip-changelog label Oct 22, 2024

VijayanB force-pushed the update-default-engine branch 2 times, most recently from 78527e6 to 1ae649e Compare October 23, 2024 06:18

jmazanec15 self-requested a review October 24, 2024 19:41

navneet1v self-requested a review October 24, 2024 19:41

VijayanB dismissed navneet1v’s stale review via 93f9c24 October 24, 2024 20:06

VijayanB force-pushed the update-default-engine branch 2 times, most recently from 93f9c24 to 0d1db53 Compare October 24, 2024 20:13

VijayanB force-pushed the update-default-engine branch from 3b56639 to 21f1ebd Compare October 25, 2024 18:26

VijayanB removed the backport 2.18 label Oct 25, 2024

VijayanB force-pushed the update-default-engine branch from 21f1ebd to 73bd246 Compare October 30, 2024 19:01

VijayanB added 4 commits November 6, 2024 12:54

Update legacy mapping

5ed1dbd

Signed-off-by: Vijayan Balasubramanian <[email protected]>

Create legacy mapping only up to V_2_17_2

4a285f3

Signed-off-by: Vijayan Balasubramanian <[email protected]>

Update test engine

25357c3

Signed-off-by: Vijayan Balasubramanian <[email protected]>

VijayanB force-pushed the update-default-engine branch from 73bd246 to c0a0ccc Compare November 6, 2024 21:03

Update test method

c79575a

Signed-off-by: Vijayan Balasubramanian <[email protected]>

VijayanB force-pushed the update-default-engine branch from c0a0ccc to c79575a Compare November 6, 2024 23:54

jmazanec15 approved these changes Nov 7, 2024

View reviewed changes

naveentatikonda approved these changes Nov 7, 2024

View reviewed changes

VijayanB merged commit 7d34456 into opensearch-project:main Nov 7, 2024
29 of 30 checks passed

VijayanB deleted the update-default-engine branch November 7, 2024 19:41

VijayanB added backport 2.x and removed backport 2.x labels Nov 7, 2024

opensearch-trigger-bot bot mentioned this pull request Nov 7, 2024

[Backport 2.x] Update default engine to FAISS #2258

Merged

navneet1v added a commit to navneet1v/k-NN that referenced this pull request Nov 7, 2024

Revert "Update default engine to FAISS (opensearch-project#2221)"

99ecd1a

This reverts commit 7d34456.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update default engine to FAISS #2221

Update default engine to FAISS #2221

VijayanB commented Oct 20, 2024 •

edited

Loading

VijayanB commented Oct 20, 2024

vibrantvarun left a comment

jmazanec15 Oct 21, 2024

jmazanec15 Oct 21, 2024

jmazanec15 commented Oct 21, 2024

navneet1v Oct 21, 2024 •

edited

Loading

Vikasht34 Oct 21, 2024

navneet1v Oct 22, 2024 •

edited

Loading

shatejas Oct 21, 2024

VijayanB Oct 23, 2024

shatejas Oct 21, 2024

VijayanB Oct 22, 2024

shatejas Oct 21, 2024

jmazanec15 Oct 21, 2024

jmazanec15 Oct 21, 2024

vamshin Oct 22, 2024 •

edited

Loading

VijayanB Oct 22, 2024

VijayanB commented Oct 24, 2024

VijayanB commented Nov 7, 2024

opensearch-trigger-bot bot commented Nov 7, 2024

VijayanB commented Nov 7, 2024 •

edited

Loading

Update default engine to FAISS #2221

Update default engine to FAISS #2221

Conversation

VijayanB commented Oct 20, 2024 • edited Loading

Description

Related Issues

Check List

VijayanB commented Oct 20, 2024

vibrantvarun left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jmazanec15 commented Oct 21, 2024

navneet1v Oct 21, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

navneet1v Oct 22, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vamshin Oct 22, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

VijayanB commented Oct 24, 2024

VijayanB commented Nov 7, 2024

opensearch-trigger-bot bot commented Nov 7, 2024

VijayanB commented Nov 7, 2024 • edited Loading

VijayanB commented Oct 20, 2024 •

edited

Loading

navneet1v Oct 21, 2024 •

edited

Loading

navneet1v Oct 22, 2024 •

edited

Loading

vamshin Oct 22, 2024 •

edited

Loading

VijayanB commented Nov 7, 2024 •

edited

Loading