metadata filter

[danpak6]

How can I query by metadata filter?
https://docs.pinecone.io/docs/metadata-filtering

Thank you!

[neon-sunset]

How can I query by metadata filter? https://docs.pinecone.io/docs/metadata-filtering

Right, it appears to be an implementation gap. I will push an update shortly. Thanks for raising the issue!

[neon-sunset]

@danpak6 the version that supports querying with metadata filtering has been published: https://www.nuget.org/packages/Pinecone.NET/1.2.0
Example:

using var index = await pinecone.GetIndex("my-test-index");

var filter = new MetadataMap
{
   ["genre"] = "horror",
   ["year"] = 1963
};

var scored = await index.Query("birds", topK: 5, filter);

or

var scored = await index.Query("birds", topK: 5, new()
{
   ["genre"] = "horror",
   ["year"] = 1963
});

Standard limitations / rules for working with Pinecone's metadata type apply.
Let me know how it works for you.

[danpak6]

Hi @neon-sunset,

Thank you for the quick response,

• you should remove the id if I want to search all the metadata in Pinecone
• you should support operators:
  The metadata filters can be combined with AND and OR:
  $eq - Equal to (number, string, boolean)
  $ne - Not equal to (number, string, boolean)
  $gt - Greater than (number)
  $gte - Greater than or equal to (number)
  $lt - Less than (number)
  $lte - Less than or equal to (number)
  $in - In array (string or number)
  $nin - Not in array (string or number)

      using var index = await pinecone.GetIndex(_indexName).ConfigureAwait(false);
      return await index.Query(filter: new()
      {
          ["id"] = "{'$eq': '0349574d-1920-440d-a69d-95f237d85a11'}'}"
          //["id"] = "0349574d-1920-440d-a69d-95f237d85a11"
      }, topK: 10);

FYI The Pinecone docs, support vectors and metadata filtering.

  index.query(
      vector=[0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1],
      filter={
          "genre": {"$eq": "documentary"},
          "year": 2019
      },
      top_k=1,
      include_metadata=True
  )
  
  # Returns:
  # {'matches': [{'id': 'B',
  #               'metadata': {'genre': 'documentary', 'year': 2019.0},
  #               'score': 0.0800000429,
  #               'values': []}],
  #  'namespace': ''}

[danpak6]

Pinecone allows to filter by metatdata only and you should just pass a json for metadata like the pinecone example.

{
    "genre": {"$eq": "documentary"},
    "year": 2019
}

  index.query(
      filter={
          "genre": {"$eq": "documentary"},
          "year": 2019
      },
      top_k=1,
      include_metadata=True
  )
  

[neon-sunset]

I don't think that's true. .Query method requests an array of ScoredVectors. If neither vector id nor previously unseen vector values are provided, there is nothing to score vectors in the index against, such an action just would not make sense.

Metadata delcaration format in C# is subject to strong type system which makes it somewhat clunkier to work with when compared to JS or Python, therefore an expression {"$and": [{"genre": "comedy"}, {"genre":"documentary"}]} would map onto

var filter = new MetadataMap
{
   ["$and"] = new MetadataValue[]
   {
      new MetadataMap { ["genre"] = "comedy" },
      new MetadataMap { ["genre"] = "documentary" }
   }
}

MongoDB-style conditionals just happen to hit the worst case scenario when it comes to syntax. If you look at how the MetadataMap is implemented, it is a Dictionary<string, MetadataValue> where MetadataValue can be either a null, a string, a double, a MetadataValue[] or another MetadataMap which allows to express all states the metadata nodes can take.

[danpak6]

I understand your point, and I wanted to provide an explanation of the Pinecone SDK's functionality.

[neon-sunset]

Thanks, I'm going to close the issue as done. Should you have any questions, feel free to open a new one or start a discussion (enabled now) 😄

Closed by #9

[danpak6]

I've been searching for a C# version for quite some time, and you're the first one to provide it. This is fantastic! In fact, I had to create an Azure Function just to be able to interact with Pinecone. Keep up the good work! 👍

[danpak6]

One more thing how can I filter all the metadata without the Id?

[neon-sunset]

One more thing how can I filter all the metadata without the Id?

I'm not sure, after brief double-checking of the official documentation there does not seem to be an apparent way to do so. It is similarity search after all. Theoretically speaking, you could apply a metadata filter, and then provide a dummy vector (i.e. new float[1536] or whatever is the width of your index) while setting topK to the desired count of matched vectors. I would expect them to have low or zero score but include the data you're interested in. Then you can just apply LINQ to manipulate the data as you wish.

[danpak6]

I am using this on my Azure Function using Pinecone Python SDK.

Query an index
The following example queries the index example-index with metadata filtering.

import pinecone

pinecone.init(api_key="YOUR_API_KEY", environment="us-west1-gcp")
index = pinecone.Index("example-index")

query_response = index.query(
    namespace="example-namespace",
    top_k=10,
    include_values=True,
    include_metadata=True,
    vector=[0.1, 0.2, 0.3, 0.4],
    filter={
        "genre": {"$in": ["comedy", "documentary", "drama"]}
    }
)

[danpak6]

Using namespaces vs. metadata filtering
https://support.pinecone.io/hc/en-us/articles/7884238411165-Using-namespaces-vs-metadata-filtering

[neon-sunset]

This would directly translate to C# code with this library:

using Pinecone;

using var pinecone = new PineconeClient("YOUR_API_KEY", "us-west1-gcp")
using var index = await pinecone.GetIndex("example-index");

var queryResponse = await index.Query(
   new[] { 0.1f, 0.2f, 0.3f, 0.4f },
   topK: 10,
   filter: new()
   {
      ["genre"] = new MetadataMap { ["$in"] = new MetadataValue[] { "comedy", "documentary", "drama" } }
   },
   indexNamespace: "example-namespace",
   includeMetadata: true);

the only "creative liberty" I took is includeValues being true by default, unlike in the official library.

[danpak6]

Your code example has birds as an Id

var scored = await index.Query("birds", topK: 5, filter);

[neon-sunset]

Pinecone allows two modes of querying:

• Query by a previously unseen vector (the overload that takes float[] values and optional SparseVector? sparseValues for sparse-dense vectors). The returned scored vectors present in your pinecone index will be scored against tge vector you have sent in the query;
• Query by an ID of a vector already present in your pinecone index (the overload that takes in string id)

The "birds" example assumes the index already has a vector with id "birds". This is just to showcase what methods the library has (and that they match the official API).

[danpak6]

If I have KB with metadata like tenantId A & B and I want to filter documents on tenant A, how can I do that without having to use namespacing?

[neon-sunset]

I'm not sure, but this is probably best answered either by Pinecone documentation (see the examples section) or community portal. I'm still learning vector DBs myself and wrote this library mostly because there was none in C#, which I love and wanted to use over Python🙂

[danpak6]

Got it, it will be hard to find or partition data in Pinecone without the metadata or if anybody is using it for semantic search and need to reference a document or page.

[neon-sunset]

Have you checked https://docs.pinecone.io/docs/semantic-text-search ? It does seem like name-spacing would be relatively safe/logical choice to achieve tenant separation. You could try tagging vectors with metadata value "tenant": "tenantName" such approach does not seem to be safe enough, and would run into scaling issues, same as with name-spacing. If I were to implement a knowledge base, I would probably (depending on estimated dataset size) choose to use per-tenant indexes instead - that way, for example, you can archive data by snapshotting it into collection and then terminating an index, etc.

[danpak6]

Here is a sample code that I am using Pinecone SDK and Azure Functions

It's a KB system and I have several companies uploading their documentation

• we have private and public docs by customer
• we have several folders (Get Started, User Guide, Troubleshooting, etc...)

I need to filter public documents as '00000000-0000-0000-0000-000000000000' and their private docs 'XXXXXX-XXXXXXX'
depending on where they are on the page the system needs to filter the folders.
If I don't filter the metadata it will return all documents in Pinecone including the private ones.

Below is the code.

def search_company_documents(index, text, company_id, folder_ids, top_k=50, include_metadata=True):
    company_ids = ['00000000-0000-0000-0000-000000000000', company_id]
    search_response = index.query(
        top_k=top_k,
        vector=vectorize_query(text),
        filter={'company_id': {"$in": company_ids}, 'folder_id': {'$in': folder_ids}},
        include_metadata=include_metadata)
           
    return search_response['matches']

[neon-sunset]

The above code is expressible in C#, feel free to refer to one of the previous examples on how to work with metadata. Converting this to a discussion because the feature support has been solved by 1.2.0.

[danpak6]

@neon-sunset As I mentioned earlier, the library is great and I hope it will continue to evolve. For now, I will continue to use the Pinecone SDK. Good luck!

[ra-one]

This would directly translate to C# code with this library:

using Pinecone;

using var pinecone = new PineconeClient("YOUR_API_KEY", "us-west1-gcp")
using var index = await pinecone.GetIndex("example-index");

var queryResponse = await index.Query(
   new[] { 0.1f, 0.2f, 0.3f, 0.4f },
   topK: 10,
   filter: new()
   {
      ["genre"] = new MetadataMap { ["$in"] = new MetadataValue[] { "comedy", "documentary", "drama" } }
   },
   indexNamespace: "example-namespace",
   includeMetadata: true);

the only "creative liberty" I took is includeValues being true by default, unlike in the official library.

@neon-sunset First of all, thank you for the all the help and new functionality.
I have question regarding filtering.
I am bit confused on how am I suppose to create filter using MetadataMap when I have a list that needs to go in filter.
Slight modification to above, what do I need to do if I want to send in ids as MetadataValue?

using Pinecone;

using var pinecone = new PineconeClient("YOUR_API_KEY", "us-west1-gcp")
using var index = await pinecone.GetIndex("example-index");

List<Guid> ids = Enumerable.Range(1, 10).Select(x => Guid.NewGuid()).ToList();

await index.Delete(
  filter: new()
  {
     ["BookID"] = new MetadataMap { ["$in"] = new MetadataValue[] { ?? } }
  },
);

[neon-sunset]

Hi!

There are two restrictions at hand you need to take care of:

• Because metadata works with primitive types, you need to format GUIDs into their string representation, just like how they are actually stored in Pinecone;
• string[] or List<Guid> cannot be assigned to MetadataValue directly despite its support for implicit casts from MetadataValue[] and List<MetadataValue>

In order to address these, the following snippet should work:

var bookIds = new Guid[] { guid1, guid2, ... }; // example GUIDs
var filter = new MetadataMap
{
    ["BookID"] = new MetadataMap
    {
        ["$in"] = bookIds
            .Select(id => (MetadataValue)id.ToString())
            .ToArray()
    }
};

await index.Delete(filter);

Let me know if it works for your use case.

[ra-one]

@neon-sunset Thank you for prompt reply, yes it is working as expected. :)