Agent memory sounds like a cut-and-dried vector search problem. You embed the task, find the nearest memories, and give them back to the model. Done.
Unfortunately it’s not that simple. Memories need to be similar, yes, but they also need to be recent. You don’t want memories from a year ago influencing your agent. And outcome is important too. If one approach worked and another failed, you probably want the successful one steering behavior.
I ran into this while building a small demo that puts Valkey Search inside an agent’s inference loop. Every task the agent finishes gets written as a memory. It persists the task text, the approach that worked, whether it succeeded, when it happened, and a vector of the task. Next time a similar task shows up, the agent recalls what worked instead of rediscovering it.
I ended up learning how valkey-search handles this the hard way, because my first FT.SEARCH call did not work the way I expected:
FT.SEARCH idx:memory "(@outcome:{success} @created_at:[1782900000 +inf])=>[KNN 3 @vector $vec]" PARAMS 2 vec <query-vector> DIALECT 2Left of the => is an ordinary boolean filter of a tag and a numeric range over two fields I indexed alongside the embedding. Right of it is the vector search.
I thought Valkey was going to find the nearest vectors and apply the tag and timestamp filters afterward. But it doesn’t work like that (in a good way). The filter criteria are actually inputs to the query planner. Depending on how many memories they match, Valkey chooses a different algorithm to search the vector index. In other words, adding a filter changes how the search runs, which meant I needed to rethink my initial memory schema. Turns out my retrieval policy was also an index-design decision.
Nobody post-filters anymore
If you’ve used Pinecone, Qdrant, Weaviate, or Milvus, then this should be familiar. All of them decide at query time whether to walk the graph with your filter applied or abandon the graph and brute-force the matching subset instead. Pinecone calls it single-stage filtering. Qdrant calls it query planning. Weaviate calls it a flat search cutoff. Milvus doesn’t really call it anything, it just does it 😂. valkey-search is the same idea, and on the FT.SEARCH vector path it doesn’t implement post-filtering at all.
If you’re a pgvector user, however, filtering happens after the index scan. With the default hnsw.ef_search of 40, a filter that keeps roughly 10% of those candidates might leave you with only 4 results. Iterative index scans were added in 0.8.0 to make that a little better, and they’re off by default.
valkey-search calls it a query planner. It estimates how many keys your filter matches and picks one of two algorithms to perform the search.
If the estimate is small compared to the index, it pre-filters results by walking the qualified key set, computing each distance directly, and keeping a top-k heap. The HNSW graph is never traversed. So valkey-search is essentially doing a brute-force scan over a tiny set.
If the estimate is large, it performs an inline filter. Your predicate is handed to hnswlib as an isIdAllowed functor and evaluated during traversal of the base layer. Non-matching nodes are still visited and expanded, they just don’t get added to the result set.
The cutoff point for one algorithm vs the other is 0.001. So if the number of estimated matching keys is at most .1% of the number of vectors in the index, it will go the brute-force route. Otherwise it uses the inline filter.
For reference, the cutoff point for Milvus is around 7%, which makes valkey-search about 70x stricter. You end up on the inline path more often than you’d think.
Make the filter disappear
On my demo index with a few hundred memories, the threshold works out to well under 1 key, so every recall that matches anything takes the inline path. You’d never notice either way at that scale.
But what would happen with the same schema in production with 1,000,000 memories? The threshold is 1,000 keys. The planner considers the selectiveness of the filter. If @outcome:{success} matches 70% of the index and your @created_at window is also broad, you’re squarely in the inline path. And the planner is right to put you there. If you scope recall to a single tenant with 400 memories, you drop under the threshold, and the query becomes an exact scan. Perfect, fast recall because 400 distance computations is nothing.
Let’s make it more difficult. A filter matching 1% of a large index sits 10x above the cutoff point, so it takes the inline path, where HNSW traverses a significant number of candidates for every one it’s allowed to keep. That results in a lot of extra latency you didn’t account for. And you can’t change that by tuning it, because search.prefiltering-threshold-ratio is immutable unless search.debug-mode is on, and it isn’t in the public configurables table at all.
You can check which path you’re actually on, by the way. valkey-search counts both, named search_prefiltering_requests_count and search_inline_filtering_requests_count. They’re module fields, so you need INFO SEARCH and not plain INFO. You can run your recall query a hundred times to see which one moves.
So your best bet is the index itself. Make the filter act like a namespace. Put a hash tag in the index name, prefix the keys to match, and each tenant’s recall hits one shard against a small index where the filter is mostly irrelevant. Valkey enforces it in both directions, too. A tagged index name requires every prefix to carry the same tag, and an untagged one requires that none of them do. Be careful with this though, because it’s not easy to undo if you change your mind since there’s no FT.ALTER here.
Forgetting is expensive
Removing a vector from an HNSW index calls markDelete and returns. The node isn’t removed from the graph, it’s still visited, it still routes other searches through itself, it just fails the deleted check. search.hnsw-allow-replace-deleted would let the next insert reuse that slot, but it’s false by default and more of a non-production flag. In production the space is stranded until you drop the index.
Luckily, updates are a different story. Modifying an indexed vector routes to an in-place update under the existing label, so the node keeps its slot and just gets its links rewired. Writing a hash field that isn’t the vector doesn’t mess with the graph, and rewriting the vector with identical bytes short-circuits before it gets there.
So updates are cheap, but deletion is where it gets expensive. That includes DEL, eviction, and expiry.
That’s a little scary, because expiry is the recency policy. valkey-search subscribes to generic, expired, and evicted keyspace notifications, so a TTL’d memory really does leave the vector index when the key goes away. Which is what you want, but it’s also what makes graph nodes stranded.
Which means you have to change how you key your memories. Creating a new key every run feels like a safe and reasonable default, and it’s what my demo does with its memory:<id> per completed task. But add a TTL to keep things fresh, and every expiring key leaves a node behind. That’s an expensive default at scale. Instead, use a stable key derived from a task fingerprint, and update it in place as the agent learns more about that kind of task. This means the task costs only one node instead of one per attempt.
There are two recency controls here to consider. The @created_at range decides what you’re willing to believe on any given query, and it doesn’t delete anything. The TTL decides what you’re willing to pay to store. Range should be your first lever and the TTL should be the slower fallback.
Decide before you have data
This demo surprised me twice: the filter I wrote as a query detail decides which algorithm runs, and the TTL I (almost) added to keep memories fresh led to stranding the index.
Everything here is discoverable, at least. The planner code is a short function to read and understand. The filtering comes from hnswlib. Even the deleted node behavior is a comment in the source, and the counters are available in INFO SEARCH. You don’t have to take my word for any of it (but you should 😜).
What you can’t read your way out of is when you have to make decisions. How the index is scoped and how a memory is keyed are day-one calls, made before you have a single memory to check them against. Outside of that dev-only flag, a rebuild is the only way to reclaim space that deletion stranded. Get those wrong and you’re reindexing.
The demo is on GitHub if you want somewhere to start. docker compose up -d gets you Valkey with the search module and an agent that writes its own memories. Run a few tasks through it, then look at INFO SEARCH and find out which path your queries are actually taking.
Happy coding!
