Semantic response caching with Valkey in Momento Cache

Different wording, repeated work

Imagine a retailer whose public support assistant answers questions across a large policy catalog, with rules that vary by region and product category. It retrieves policy text and generates a cited explanation for each request. Within that broader workload, some questions recur with only small changes in wording.

One shopper asks, “How long do I have to return headphones?” The assistant explains the 30-day return window, citing the official policy. Then, a second shopper asks, “What’s the return window for headphones?” The same explanation would answer both questions, but the assistant blindly repeats retrieval and generation to produce a similar response.

A semantic response cache avoids that repeated work. Valkey stores previous answers along with metadata about the question. The Valkey Search module uses this metadata to find responses to similar questions. The application can now fetch a set of possible responses and return one that fits.

The benefit is fewer inference calls and shorter waits. Each reused answer avoids generation costs and returns without waiting for a new model response. Across recurring questions, those savings can reduce both the bill and the average response time.

Why Momento Cache for Valkey fits this pattern

Momento Cache is the leading provider of Valkey, an open-source fork of Redis built for real-time workloads. With Momento Cache and Valkey, your team can enjoy the same high-performance infrastructure that powers global leaders like Snap, FOX, Coinbase, and Capcom.

Each semantic cache record contains the original question, its complete answer and citation, and an embedding: a numeric vector representing the question for similarity search. Additional fields such as region, language, product category, condition, and policy version define where the answer can be reused.

Valkey holds those records, and its Search module indexes their vectors and metadata. Combining vector queries with structured filters gives the semantic cache three useful capabilities:

• Keep answers and search data together. Store the response, citation, question vector, and scope fields under one key. A matching record contains everything the application needs to assess and serve the answer.
• Search with the right context. Combine vector similarity with filters for additional fields like region. Search can find the response to a paraphrased question, while still excluding records from a different policy scope.
• Share each cached answer across workers. A response stored in Valkey by one application worker becomes available to others, allowing later requests to benefit regardless of which instance handles them.

Valkey supplies the shared answer store and filtered vector search. The application defines the embedding used for queries, and ultimately decides which proposed responses to accept.

How semantic caching works

A suitable semantic match skips retrieval and generation.

Let’s follow the second shopper’s question about returning headphones to illustrate how semantic caching works in practice:

1. Check eligibility and scope. The application identifies the request as a general return-policy question about headphones. It extracts the US region, English language, and active policy version from application context.

2. Find a candidate answer. An embedding service converts the question to a vector. The application passes this vector and the metadata filters to Valkey with the command FT.SEARCH, which returns a set of response candidates containing an answer, policy citation, metadata, and vector distance.

3. Check whether the answer fits. The application checks the vector distance and confirms that details affecting the answer match. This catches small text differences like “opened” vs “unopened”. If the relevant details match, the application returns the cached explanation.

4. Generate an answer on a miss. If no candidate fits, the application retrieves current policy and generates a fresh response.

5. Store answers that qualify for reuse. After responding, the application stores the new response in Valkey if it’s suitable for caching. For example, an answer containing shopper-specific information does not belong in this shared cache.

Each answer added to Valkey gives later questions another opportunity to skip retrieval and generation. When the next shopper asks about the same return policy in different words, the application can deliver the explanation sooner and avoid another model call.

Lower costs and faster answers

A semantic cache saves time and money by avoiding retrieval and generation when it serves a reusable answer. A 25% hit rate means one in four eligible questions takes that shorter path.

Lower costs. For every $1,000 you spend on generation, a 25% hit rate could save $250! Of course, caching infrastructure adds some cost, but this is usually far less than repeatedly generating duplicate responses.

Faster answers. Retrieval and inference are two of the slowest types of calls, often taking multiple seconds. Meanwhile, Momento Cache returns hits from Valkey in just milliseconds.

The actual semantic cache hit rate depends on the query patterns in your application. The opportunity grows as more questions can reuse an answer: fewer paid model calls for the business, and less waiting for users.

Turn repeated questions into faster answers

When users ask the same thing in different ways, semantic caching lets one answer serve many requests. Valkey Search on Momento Cache combines a shared answer store with filtered vector matching, helping your application reuse relevant responses, reduce generation costs, and shorten the wait for users.

Bring semantic caching to your application: get started with Momento Cache and put your recurring questions to work.