With the exponential growth of the information we have, building a search system is a vital tool for the modern day engineer. Building a search capability into your website can be intimidating and expensive, requiring you to set up dedicated infrastructure just to index your data and search it.
In this series we are going to teach you how to build a quick but powerful search system without breaking the bank. Starting with basic search and continuing to add and improve on it as the series continues. The secret tools we are going to use to power this search are a large language model and a vector index.
A vector index is a powerful tool for storing and managing data in the form of vectors. The direction of each vector encodes essential attributes and features of the data. These data stores play a crucial role in enabling advanced functionalities such as reverse image search and empowering AI chatbots.
While the concept of vector indexes might initially appear complex and intimidating, Momento has simplified the process. Surprisingly, with just a few lines of code, vector indexes can seamlessly replace many of the search capabilities you rely on. Let's delve deeper into this exciting technology.
Architecture
For our test app we are going to be building a simple movie catalog service that allows you to search for movies by their titles and descriptions(I got the dataset from here). You'll end up with an architecture that looks like this:

Creating a vector from text
For this tutorial we are going to be using Sentence Transformers all-MiniLM-L6-v2 model which is a general purpose model for generating embeddings(vectors) from some input text. We are going to write a quick utility function get_embedding to make this request for us.
Ingesting your data into the index
Next up, let’s scan our movies.csv data file and for each item we get the embedding and insert it into our Momento Vector Index. The code to do that will look like this
Search Data
Now that we have our data loaded into our index we can search it easily with a plain text query. For example, here I'm going to search for movies with "dogs".
When we run this we get the following output:
You can see our top 2 results were both movies about dogs Babe and Balto 🐕.
Conclusion and Next steps
That's it! Now you have added a basic search capability to your stack in just a few minutes!
Stay tuned for the upcoming blogs in this series where we will learn how to:
- Measure the quality of our search
- Tune how we pass input data to the embedding function for more accurate results
- Add additional data to be indexed for movies
- Use multiple indexes and assign different weights to certain fields
- Inject business logic into your search ex: recommend more scary movies at night!
- Bring your own model
- Adding in caching to help with scaling.