November 30, 2023
-
12
Min Read

Getting Started with Momento Vector Index in .NET

Momento lets you create powerful, complete vector indexes for your .NET applications with just 5 API calls.
Nate Anderson
Headshot of the blog author
by
Nate Anderson
,
,
by
Nate Anderson
by
green squirrel logo for momento
Nate Anderson
,
,
AI/ML

A vector index is a specialized data store that is optimized for storing and searching high-dimensional vectors. Instead of searching through row or column oriented data as seen in relational databases, vector indexes are designed to calculate the similarity between vectors. They are particularly useful for tasks like as text search, image search, recommendations, image and voice recognition, and more. Vector search comes into play in any application where data can be represented as a vector and when similarity searches are important. For a more detailed introduction to vector indexes, see Kirk Kirkonnell’s article about vector indexes. For more in-depth information on how they work and how to integrate them into machine learning workflows, see Michael Landis’ talk on vector searching.

Momento Vector Index provides a fast and easy way to get started with vector indexes. In this article, we’ll make a simple C# program that sets up a vector index, adds data to it, and searches that data.

Prerequisites

Momento API Key

To use Momento Vector Index you will need a Super User key for the AWS us-west-2 region.Go to the Momento console and follow the instructions to log in with your email address, Google account, or GitHub account.

In the console, select the API Keys menu option.

Once on the API key page, select the information that matches where your caches live:

  1. Cloud provider - AWS
  2. Region - us-west-2
  3. Key Type - Super User
  4. (Optional) Expiration date

The Momento .NET SDK

You’ll need to install the Momento SDK package to use it in your program. It is available on nuget, and can be added via the .NET CLI: dotnet add package Momento.Sdk

Writing your first Momento Vector Index program

Here is the basic Momento Vector Index example we’ll be going over.


using Momento.Sdk;
using Momento.Sdk.Auth;
using Momento.Sdk.Config;
using Momento.Sdk.Messages.Vector;
using Momento.Sdk.Requests.Vector;
using Momento.Sdk.Responses.Vector;

namespace VectorExample;

public static class Program
{
    public static async Task Main()
    {
        var config = VectorIndexConfigurations.Laptop.Latest();
        var authProvider = new EnvMomentoTokenProvider("MOMENTO_API_KEY");
        using IPreviewVectorIndexClient client = new PreviewVectorIndexClient(config, authProvider);

        // create a momento vector index
        const string indexName = "getting-started-index";
        var createIndexResponse =
            await client.CreateIndexAsync(indexName, numDimensions: 2,
                similarityMetric: SimilarityMetric.CosineSimilarity);
        switch (createIndexResponse)
        {
            case CreateIndexResponse.Success:
                Console.WriteLine($"Index with name {indexName} successfully created!");
                break;
            case CreateIndexResponse.AlreadyExists:
                Console.WriteLine($"Index with name {indexName} already exists");
                break;
            case CreateIndexResponse.Error error:
                Console.WriteLine($"Error while creating index {indexName}: {error.Message}");
                break;
        }

        // list all indexes
        var listIndexesResponse = await client.ListIndexesAsync();
        switch (listIndexesResponse)
        {
            case ListIndexesResponse.Success success:
                Console.WriteLine($"Found indexes: {string.Join(", ", success.Indexes.Select(i => i.Name))}");
                break;
            case ListIndexesResponse.Error error:
                Console.WriteLine($"Error while listing indexes: {error.Message}");
                break;
        }

        // upsert data into the index
        var items = new List
        {
            new("item_1", new List { 0.0f, 1.0f },
                new Dictionary { { "key1", "value1" } }),
            new("item_2", new List { 1.0f, 0.0f },
                new Dictionary { { "key2", 12345 }, { "key3", 678.9 } }),
            new("item_3", new List { -1.0f, 0.0f },
                new Dictionary { { "key1", new List { "value2", "value3" } } }),
            new("item_4", new List { 0.5f, 0.5f },
                new Dictionary { { "key4", true } })
        };
        var upsertResponse = await client.UpsertItemBatchAsync(indexName, items);
        switch (upsertResponse)
        {
            case UpsertItemBatchResponse.Success:
                Console.WriteLine("Successfully added items");
                break;
            case UpsertItemBatchResponse.Error error:
                Console.WriteLine($"Error while adding items to index {indexName}: {error.Message}");
                break;
        }

        // wait a short time to ensure the vectors are uploaded
        Thread.Sleep(2000);

        // search the index
        var searchResponse = await client.SearchAsync(indexName, new List { 1.0f, 0.0f }, topK: 4,
            metadataFields: MetadataFields.All);
        switch (searchResponse)
        {
            case SearchResponse.Success success:
                Console.WriteLine($"Search succeeded with {success.Hits.Count} matches:");
                foreach (var hit in success.Hits)
                {
                    Console.WriteLine($"ID: {hit.Id}, score: {hit.Score}, metadata:");
                    foreach (var (key, value) in hit.Metadata)
                    {
                        Console.WriteLine($"  {key}: {value}");
                    }
                }

                break;
            case SearchResponse.Error error:
                Console.WriteLine($"Error while searching on index {indexName}: {error.Message}");
                break;
        }

        // delete the index
        var deleteResponse = await client.DeleteIndexAsync(indexName);
        switch (deleteResponse)
        {
            case DeleteIndexResponse.Success:
                Console.WriteLine($"Index {indexName} deleted successfully!");
                break;
            case DeleteIndexResponse.Error error:
                Console.WriteLine($"Error while deleting index {indexName}: {error.Message}");
                break;
        }
    }
}

It creates an index, lists all indexes, uploads data, searches for that data, and finally deletes the index.

After installing the Momento dependency we can run the program and get the following output:

Index with name getting-started-index successfully created!

Found indexes: getting-started-index

Successfully added items

Search succeeded with 4 matches:

ID: item_2, score: 1, metadata:    

key3: 678.9  

key2: 12345

ID: item_4, score: 0.7071067690849304, metadata:

  key4: True

ID: item_1, score: 0, metadata:

   key1: value1

ID: item_3, score: -1, metadata:

  key1: value2, value3

Index getting-started-index deleted successfully!

Process finished with exit code 0.

In the next section, we'll explain how this output was produced.

Creating a client

The first thing we need to do is create a vector index client:


var config = VectorIndexConfigurations.Laptop.Latest();
var authProvider = new EnvMomentoTokenProvider("MOMENTO_API_KEY");
using IPreviewVectorIndexClient client = new PreviewVectorIndexClient(config, authProvider);

A vector index client, like all Momento clients, requires a configuration object and an auth provider. The auth provider loads and parses your Momento API key. It can either load from an environment variable or directly from a string.

The configuration contains settings for the underlying gRPC client, as well as any customer configurable features. To avoid needing to create one every time, we provide prebuilt configurations. VectorIndexConfigurations.Laptop.Latest() is the newest version of the Laptop configuration, meant for developer use. We version the configurations for backwards compatibility, so changes to a configuration won’t affect customer deployments. latest will always point to the newest version.Since the client is an IDisposable, we can use the using statement to automatically clean it up when it goes out of context.

CreateIndexAsync

Now that we have a client, we can create an index:


// create a momento vector index
const string indexName = "getting-started-index";
var createIndexResponse =
    await client.CreateIndexAsync(indexName, numDimensions: 2,
        similarityMetric: SimilarityMetric.CosineSimilarity);
switch (createIndexResponse)
{
    case CreateIndexResponse.Success:
        Console.WriteLine($"Index with name {indexName} successfully created!");
        break;
    case CreateIndexResponse.AlreadyExists:
        Console.WriteLine($"Index with name {indexName} already exists");
        break;
    case CreateIndexResponse.Error error:
        Console.WriteLine($"Error while creating index {indexName}: {error.Message}");
        break;
}

The CreateIndexAsync function takes three arguments: indexName - the name of the index, numDimensions - the number of dimensions in the index, and similarityMetric - the metric that will be used to compare vectors in a search.

For this example we are creating an index with 2 dimensions. Since a dimension represents a feature or attribute of a piece of complex data, a real-world index may have hundreds. We’re doing this to make it easier to visualize how the index compares vectors when searching. We’re also using the cosine similarity as our similarity metric. It compares the angles between vectors, normalized to between -1 and 1. It is the default choice when setting up a Momento vector index.

This function illustrates the error handling pattern that the Momento APIs use. A client method should never throw an exception. Instead, it returns a response object that represents different types of call results. Here it can be Success if the index is created, AlreadyExists if there was already an index by that name, or Error if the call failed. All Momento calls can return an error object containing details about the specific failure.

ListIndexesAsync

Now that we created an index, we can see it by listing all indexes:


var listIndexesResponse = await client.ListIndexesAsync();
switch (listIndexesResponse)
{
    case ListIndexesResponse.Success success:
        Console.WriteLine($"Found indexes: {string.Join(", ", success.Indexes.Select(i => i.Name))}");
        break;
    case ListIndexesResponse.Error error:
        Console.WriteLine($"Error while listing indexes: {error.Message}");
        break;
}

The ListIndexesAsync function takes no arguments and returns a Success object with a list of information about all indexes in your account in your region. This function is useful if your code doesn’t use a hard-coded index and needs to look one up, or if you need to keep track of your index count to make sure you aren’t creating too many.

UpsertItemBatchAsync

We can now add vectors to our new index:


var items = new List
{
    new("item_1", new List { 0.0f, 1.0f },
        new Dictionary { { "key1", "value1" } }),
    new("item_2", new List { 1.0f, 0.0f },
        new Dictionary { { "key2", 12345 }, { "key3", 678.9 } }),
    new("item_3", new List { -1.0f, 0.0f },
        new Dictionary { { "key1", new List { "value2", "value3" } } }),
    new("item_4", new List { 0.5f, 0.5f },
        new Dictionary { { "key4", true } })
};
var upsertResponse = await client.UpsertItemBatchAsync(indexName, items);
switch (upsertResponse)
{
    case UpsertItemBatchResponse.Success:
        Console.WriteLine("Successfully added items");
        break;
    case UpsertItemBatchResponse.Error error:
        Console.WriteLine($"Error while adding items to index {indexName}: {error.Message}");
        break;
}

The UpsertItemBatchAsync function takes in an index name and a list of Items representing vectors and inserts them into the index, replacing any existing vectors with matching IDs. An Item contains a unique ID, a vector matching the dimensionality of the index, and optional metadata. The metadata keys must be strings, but the values can be strings, ints, floats, booleans, or lists of strings.

We uploaded the vectors [-1.0, 0.0], [0.0, 1.0], [0.5, 0.5], and [1.0, 0.0]. Since they are two dimensional, we can visualize them:

You can see how we could compare a query vector to these by the difference in their angles.

SearchAsync

Now that our index contains data, we can search for it:

 
var searchResponse = await client.SearchAsync(indexName, new List { 1.0f, 0.0f }, topK: 4,
    metadataFields: MetadataFields.All);
switch (searchResponse)
{
    case SearchResponse.Success success:
        Console.WriteLine($"Search succeeded with {success.Hits.Count} matches:");
        foreach (var hit in success.Hits)
        {
            Console.WriteLine($"ID: {hit.Id}, score: {hit.Score}, metadata:");
            foreach (var (key, value) in hit.Metadata)
            {
                Console.WriteLine($"  {key}: {value}");
            }
        }

        break;
    case SearchResponse.Error error:
        Console.WriteLine($"Error while searching on index {indexName}: {error.Message}");
        break;
}

The SearchAsync function takes an index name, a query vector matching the dimensionality of the index, an optional topK argument representing the number of results to return, and an optional metadataFields argument representing which metadata you want returned. We’re using the MetadataFields.All sentinel value here, meaning that all metadata should be returned, but you could also supply a list of fields, e.g. new List<string> {"key1", "key3"} to specify that only metadata matching those field names should be returned. If metadataFields is not specified, no metadata is returned.

A Success search response contains a list of search hits. Each hit contains the ID of the vector, the score, i.e. the similarity of that vector to the search vector (from -1 to 1 for cosine similarity), and any requested metadata. Here is an example from our program’s output:

ID: item_2, score: 1, metadata:

  key3: 678.9

  key2: 12345

ID: item_4, score: 0.7071067690849304, metadata:

 key4: True

ID: item_1, score: 0, metadata:

 key1: value1

ID: item_3, score: -1, metadata:

 key1: value2, value3

Matches are returned ordered by score. We searched for the vector [1.0, 0.0], so item_2, which matches that vector exactly, has a score of 1.0. item_3, with a vector [-1.0, 0.0], is the exact opposite, and has a score of -1.0. item_1, which has an vector orthogonal to our search vector, has a score of 0.0. Higher dimensional vectors cannot be easily visualized, but the pattern is the same: the more closely a vector matches the search vector, the closer its score will be to 1.0. The more a vector matches the opposite of the search vector, the closer it will be to -1.0.

DeleteIndexAsync

Finally, we delete the index to clean up for after example:


var deleteResponse = await client.DeleteIndexAsync(indexName);
switch (deleteResponse)
{
    case DeleteIndexResponse.Success:
        Console.WriteLine($"Index {indexName} deleted successfully!");
        break;
    case DeleteIndexResponse.Error error:
        Console.WriteLine($"Error while deleting index {indexName}: {error.Message}");
        break;
}

The DeleteIndexAsync function takes an index name and deletes that index and all data in it. It returns success if the deletion is successful or there is no index by that name to delete.

Ready to start?

At Momento, we approach our services with simplicity in mind. We aim to give you the fastest, easiest developer experience on the market so you can focus on solving the problems that you actually care about. Momento Vector Index is a fully serverless vector index, and can be used to its fullest extent with only five API calls.

If you want to learn more about Momento Vector Index, you can read through the dev docs here, or you can read about integrating with LangChain on our blog here.

Nate Anderson
by
Nate Anderson
,
,
by
Nate Anderson
by
green squirrel logo for momento
by
Nate Anderson
,
,
Author
Nate Anderson

Nate is a software engineer at Momento. He started his career at Thetus Corporation before moving on to specialize in JVM backend and streaming applications at companies like Marin Software, Womply, and New Relic. At New Relic, Nate became a tech lead and discovered a passion for fostering engineers new in their careers. He was drawn to Momento because of the opportunity to work on large-scale distributed systems in a smaller company setting—while also getting to work with so many different programming languages and interact with customers. Between New Relic and Momento, he took a year off to travel, refresh, and spend more time on what he loves outside of work: weightlifting, fiddling with his 3D printer, and connecting with friends through games.

Author
Author
Open