December 1, 2023
-
12
Min Read

Getting started with Momento Vector Index in JavaScript

Momento lets you create powerful, complete vector indexes for your JavaScript applications with just 5 API calls.
Nate Anderson
Headshot of the blog author
by
Nate Anderson
,
,
by
Nate Anderson
by
green squirrel logo for momento
Nate Anderson
,
,
AI/ML

A vector index is a specialized data store that is optimized for storing and searching high-dimensional vectors. Instead of searching through row- or column-oriented data as seen in relational databases, vector indexes are designed to calculate the similarity between vectors. They are particularly useful for tasks like text search, image search, recommendations, image and voice recognition, and more. Vector search comes into play in any application where data can be represented as a vector and when similarity searches are important. For a more detailed introduction to vector indexes, see Kirk Kirkonnell’s article about vector indexes. For more in-depth information on how they work and how to integrate them into machine learning workflows, see Michael Landis’ talk on vector searching.

Momento Vector Index provides a fast and easy way to get started with vector indexes. In this article, we’ll make a simple TypeScript program that sets up a vector index, adds data to it, and searches that data.

Prerequisites

Momento API Key

To use Momento Vector Index you will need a Super User key for the AWS us-west-2 region.Go to the Momento console and follow the instructions to log in with your email address, Google account, or GitHub account.

In the console, select the API Keys menu option.

Once on the API key page, select the information that matches where your caches live:

  1. Cloud provider - AWS
  2. Region - us-west-2
  3. Key Type - Super User
  4. (Optional) Expiration date

The Momento JavaScript SDK

You’ll need to install the Momento SDK dependency to use it in your program.

For Node applications, use @gomomento/sdk. It is available on npm and can be installed with npm install @gomomento/sdk.

For web applications, use @gomomento/sdk-web. It is also on npm an can be installed with npm install @gomomento/sdk-web.

Write your first Momento Vector Index program


import {
  CreateVectorIndex,
  CredentialProvider,
  DeleteVectorIndex,
  ListVectorIndexes,
  PreviewVectorIndexClient,
  VectorIndexConfigurations,
  VectorIndexItem,
  VectorSearch,
  VectorUpsertItemBatch,
} from '@gomomento/sdk';
import {ALL_VECTOR_METADATA} from '@gomomento/sdk-core';

async function main() {
  const credentialProvider = CredentialProvider.fromEnvironmentVariable({
    environmentVariableName: 'MOMENTO_API_KEY',
  });
  const configuration = VectorIndexConfigurations.Laptop.latest();
  const client = new PreviewVectorIndexClient({credentialProvider, configuration});

  // create a momento vector index
  const indexName = 'getting-started-index';
  const createIndexResult = await client.createIndex(indexName, 2, VectorSimilarityMetric.COSINE_SIMILARITY);
  if (createIndexResult instanceof CreateVectorIndex.Success) {
    console.log(`Index with name ${indexName} successfully created!`);
  } else if (createIndexResult instanceof CreateVectorIndex.AlreadyExists) {
    console.log(`Index with name ${indexName} already exists`);
  } else if (createIndexResult instanceof CreateVectorIndex.Error) {
    console.log(`Error while creating index: ${createIndexResult.errorCode()}: ${createIndexResult.toString()}`);
  }

  // list all indexes
  const listIndexesResult = await client.listIndexes();
  if (listIndexesResult instanceof ListVectorIndexes.Success) {
    console.log(
      `Indexes:\n${listIndexesResult
        .getIndexes()
        .map(listIndexesResult => JSON.stringify(listIndexesResult.getName()))
        .join('\n')}`
    );
  } else if (listIndexesResult instanceof ListVectorIndexes.Error) {
    console.log(`Error while listing indexes: ${listIndexesResult.errorCode()}: ${listIndexesResult.toString()}`);
  }

  // upsert data into the index
  const items: Array = [
    {
      id: 'item_1',
      vector: [0.0, 1.0],
      metadata: {key1: 'value1'},
    },
    {
      id: 'item_2',
      vector: [1.0, 0.0],
      metadata: {key2: 12345, key3: 678.9},
    },
    {
      id: 'item_3',
      vector: [-1.0, 0.0],
      metadata: {key1: ['value2', 'value3']},
    },
    {
      id: 'item_4',
      vector: [0.5, 0.5],
      metadata: {key4: true},
    },
  ];
  const upsertResult = await client.upsertItemBatch(indexName, items);
  if (upsertResult instanceof VectorUpsertItemBatch.Success) {
    console.log('Successfully added items');
  } else if (upsertResult instanceof VectorUpsertItemBatch.Error) {
    console.log(`Error while adding items to index: ${upsertResult.errorCode()}: ${upsertResult.toString()}`);
  }

  // wait a short time to ensure the vectors are uploaded
  await new Promise(resolve => setTimeout(resolve, 1000));

  // search the index
  const searchResult = await client.search(indexName, [1.0, 0.0], {topK: 4, metadataFields: ALL_VECTOR_METADATA});
  if (searchResult instanceof VectorSearch.Success) {
    console.log(`Search succeeded with ${searchResult.hits().length} matches`);
    console.log(searchResult.hits());
  } else if (searchResult instanceof VectorSearch.Error) {
    console.log(`Error while searching index ${indexName}: ${searchResult.errorCode()}: ${searchResult.toString()}`);
  }

  // delete the index
  const deleteResponse = await client.deleteIndex(indexName);
  if (deleteResponse instanceof DeleteVectorIndex.Success) {
    console.log(`Index ${indexName} deleted successfully!`);
  } else if (deleteResponse instanceof DeleteVectorIndex.Error) {
    console.log(`Failed to delete index ${indexName}: ${deleteResponse.errorCode()}: ${deleteResponse.toString()}`);
  }
}

main().catch(e => {
  throw e;
});

Here is the basic Momento Vector Index example we’ll be going over. It is built using the  @gomomento/sdk dependency. A web application would use the web dependency, and should to use a more limited API key loaded from a string.

It creates an index, lists all indexes, uploads data, searches for that data, and finally deletes the index.

After installing the Momento dependency we can run the program and get the following output:

Index with name getting-started-index successfully created!

Indexes:

"getting-started-index"

Successfully added items

Search succeeded with 4 matches

[

{ id: 'item_2', score: 1, metadata: { key2: 12345, key3: 678.9 } },

{ id: 'item_4', score: 0.7071067690849304, metadata: { key4: true } },

{ id: 'item_1', score: 0, metadata: { key1: 'value1' } },

{ id: 'item_3', score: -1, metadata: { key1: [Array] } }

]

Index getting-started-index deleted successfully!

In the next section, we'll explain how this output was produced.

Creating a client

The first thing we need to do is create a vector index client:


const credentialProvider = CredentialProvider.fromEnvironmentVariable({
  environmentVariableName: 'MOMENTO_API_KEY',
});
const configuration = VectorIndexConfigurations.Laptop.latest();
const client = new PreviewVectorIndexClient({credentialProvider, configuration});

A vector index client, like all Momento clients, requires a configuration object and an auth provider. The auth provider loads and parses your Momento API key. It can either load from an environment variable or directly from a string.

The configuration contains settings for the underlying gRPC client, as well as any customer configurable features. To avoid needing to create one every time, we provide prebuilt configurations. VectorIndexConfigurations.Laptop.latest() is the newest version of the Laptop configuration, built for local development. We version the configurations for backwards compatibility, so changes to a configuration won’t affect customer deployments. latest will always point to the newest version.

createIndex

Now that we have a client, we can create an index:


const indexName = 'getting-started-index';
const createIndexResult = await client.createIndex(indexName, 2, VectorSimilarityMetric.COSINE_SIMILARITY);
if (createIndexResult instanceof CreateVectorIndex.Success) {
  console.log(`Index with name ${indexName} successfully created!`);
} else if (createIndexResult instanceof CreateVectorIndex.AlreadyExists) {
  console.log(`Index with name ${indexName} already exists`);
} else if (createIndexResult instanceof CreateVectorIndex.Error) {
  console.log(`Error while creating index: ${createIndexResult.errorCode()}: ${createIndexResult.toString()}`);
}

The createIndex function takes three arguments: indexName - the name of the index, numDimensions - the number of dimensions in the index, and similarityMetric - the metric that will be used to compare vectors in a search.

For this example we are creating an index with 2 dimensions. Since a dimension represents a feature or attribute of a piece of complex data, a real-world index may have hundreds. We’re doing this to make it easier to visualize how the index compares vectors when searching. We’re also using the cosine similarity as our similarity metric. It compares the angles between vectors, normalized to between -1 and 1. It is the default choice when setting up a Momento vector index.

This function illustrates the error handling pattern that the Momento APIs use. A client method should never throw an exception. Instead, it returns a response object that represents different types of call results. Here it can be Success if the index is created, AlreadyExists if there was already an index by that name, or Error if the call failed. All Momento calls can return an error object containing details about the specific failure.

listIndexes

Now that we created an index, we can see it by listing all indexes:


const listIndexesResult = await client.listIndexes();
if (listIndexesResult instanceof ListVectorIndexes.Success) {
  console.log(
    `Indexes:\n${listIndexesResult
      .getIndexes()
      .map(listIndexesResult => JSON.stringify(listIndexesResult.getName()))
      .join('\n')}`
  );
} else if (listIndexesResult instanceof ListVectorIndexes.Error) {
  console.log(`Error while listing indexes: ${listIndexesResult.errorCode()}: ${listIndexesResult.toString()}`);
}

The listIndexes function takes no arguments and returns a Success object with a list of information about all indexes in your account in your region. This function is useful if your code doesn’t use a hard-coded index and needs to look one up, or if you need to keep track of your index count to make sure you aren’t creating too many.

upsertItemBatch

We can now add vectors to our new index:


const items: Array = [
  {
    id: 'item_1',
    vector: [0.0, 1.0],
    metadata: {key1: 'value1'},
  },
  {
    id: 'item_2',
    vector: [1.0, 0.0],
    metadata: {key2: 12345, key3: 678.9},
  },
  {
    id: 'item_3',
    vector: [-1.0, 0.0],
    metadata: {key1: ['value2', 'value3']},
  },
  {
    id: 'item_4',
    vector: [0.5, 0.5],
    metadata: {key4: true},
  },
];
const upsertResult = await client.upsertItemBatch(indexName, items);
if (upsertResult instanceof VectorUpsertItemBatch.Success) {
  console.log('Successfully added items');
} else if (upsertResult instanceof VectorUpsertItemBatch.Error) {
  console.log(`Error while adding items to index: ${upsertResult.errorCode()}: ${upsertResult.toString()}`);
}

The upsertItemBatch function takes in an index name and a list of Items representing vectors and inserts them into the index, replacing any existing vectors with matching IDs. An Item contains a unique ID, a vector matching the dimensionality of the index, and optional metadata. The metadata keys must be strings, but the values can be strings, ints, floats, booleans, or lists of strings.

We uploaded the vectors [-1.0, 0.0], [0.0, 1.0], [0.5, 0.5], and [1.0, 0.0]. Since they are two dimensional, we can visualize them:

You can see how we could compare a query vector to these by the difference in their angles.

search

Now that our index contains data, we can search for it:


const searchResult = await client.search(indexName, [1.0, 0.0], {topK: 4, metadataFields: ALL_VECTOR_METADATA});
if (searchResult instanceof VectorSearch.Success) {
  console.log(`Search succeeded with ${searchResult.hits().length} matches`);
  console.log(searchResult.hits());
} else if (searchResult instanceof VectorSearch.Error) {
  console.log(`Error while searching index ${indexName}: ${searchResult.errorCode()}: ${searchResult.toString()}`);
}

The search function takes an index name, a query vector matching the dimensionality of the index, an optional topK argument representing the number of results to return, and an optional metadataFields argument representing which metadata you want returned. We’re using the ALL_VECTOR_METADATA sentinel value here, meaning that all metadata should be returned, but you could also supply a list of fields, e.g. metadataFields: ['key1', 'key3'] to specify that only metadata matching those field names should be returned. If metadataFields is not specified, no metadata is returned.

A Success search response contains a list of search hits. Each hit contains the ID of the vector, the score, i.e. the similarity of that vector to the search vector (from -1 to 1 for cosine similarity), and any requested metadata. Here is an example from our program’s output:

[

{ id: 'item_2', score: 1, metadata: { key2: 12345, key3: 678.9 } },

{ id: 'item_4', score: 0.7071067690849304, metadata: { key4: true } },

{ id: 'item_1', score: 0, metadata: { key1: 'value1' } },

{ id: 'item_3', score: -1, metadata: { key1: [Array] } }

]

Matches are returned ordered by score. We searched for the vector [1.0, 0.0], so item_2, which matches that vector exactly, has a score of 1.0. item_3, with a vector [-1.0, 0.0], is the exact opposite, and has a score of -1.0. item_1, which has an vector orthogonal to our search vector, has a score of 0.0. Higher dimensional vectors cannot be easily visualized, but the pattern is the same: the more closely a vector matches the search vector, the closer its score will be to 1.0. The more a vector matches the opposite of the search vector, the closer it will be to -1.0.

deleteIndex

Finally, we delete the index to clean up for after example:


const deleteResponse = await client.deleteIndex(indexName);
if (deleteResponse instanceof DeleteVectorIndex.Success) {
  console.log(`Index ${indexName} deleted successfully!`);
} else if (deleteResponse instanceof DeleteVectorIndex.Error) {
  console.log(`Failed to delete index ${indexName}: ${deleteResponse.errorCode()}: ${deleteResponse.toString()}`);
}

The deleteIndex function takes an index name and deletes that index and all data in it. It returns success if the deletion is successful or there is no index by that name to delete.

Ready to start?

At Momento, we approach our services with simplicity in mind. We aim to give you the fastest, easiest developer experience on the market so you can focus on solving the problems that you actually care about. Momento Vector Index is a fully serverless vector index, and can be used to its fullest extent with only five API calls.

If you want to learn more about Momento Vector Index, you can read through the dev docs here, or you can read about integrating with LangChain on our blog here.

Nate Anderson
by
Nate Anderson
,
,
by
Nate Anderson
by
green squirrel logo for momento
by
Nate Anderson
,
,
Author
Nate Anderson

Nate is a software engineer at Momento. He started his career at Thetus Corporation before moving on to specialize in JVM backend and streaming applications at companies like Marin Software, Womply, and New Relic. At New Relic, Nate became a tech lead and discovered a passion for fostering engineers new in their careers. He was drawn to Momento because of the opportunity to work on large-scale distributed systems in a smaller company setting—while also getting to work with so many different programming languages and interact with customers. Between New Relic and Momento, he took a year off to travel, refresh, and spend more time on what he loves outside of work: weightlifting, fiddling with his 3D printer, and connecting with friends through games.

Author
Author
Open