We’re opening a limited preview of the new Cluster and Flex configurations for Momento Cache. This release conveniently packages the technology and operating experience that power some of the largest Valkey clusters in the world.

Momento Cache is built for fast-moving teams who want optimal Valkey performance without the hassle of babysitting infrastructure. Cluster capacity and Flex capacity bring single-tenant, fully-managed resources to this self-service infrastructure platform.

Cluster provides direct control over instance type, shards, replicas, and availability zones. Flex automatically optimizes resources within specified bounds. Both were designed for workloads that need stronger isolation, more control, and a higher performance ceiling than Momento Cache’s Serverless configuration.

Avoiding the trap of operational creep

In the AI era, it’s trivial to stand up basic infrastructure at near-zero cost. We’ve all been there: it’s fast, it’s easy, it mostly works. It’s the right solution when you need to ship.

Then, growth hits. Traffic changes shape. Memory fills unevenly. A shard needs to move. A primary fails. Clients stampede. A zero-day patch lands at midnight. Suddenly, operational creep has consumed the time and the token budget that you wanted to spend on building.

And the problems only multiply as you add more features, products, and services. Soon your entire team is stuck fighting against infrastructure as latency, cost, and complexity steadily creep up and to the right.

Fast, reliable, efficient - pick all three

The Momento platform powers critical features for millions of users around the world at companies like Capcom, Coinbase, Paramount, and Snap. Now, Momento Cache puts the full power and flexibility of Valkey at your fingertips, packaging up hard-won production lessons into a streamlined service.

The new Cluster and Flex configurations help you to tailor a Valkey deployment to fit your specific needs. Momento seamlessly operates the lifecycle behind that system: provisioning, health, failover, rolling topology changes, version upgrades, and security patches.

Whether you’re pushing one thousand or one million requests per second, Momento delivers unrivaled performance and resource utilization. As you grow, your infrastructure grows alongside you. For companies with a mature platform org, Momento Cache can even be deployed in a BYOC configuration, as part of your internal developer platform.

Transparent pricing

Momento Cache pricing is designed to be simple, predictable, and cost-effective. Pricing in us-east-1:

• Flex starts at $13 per GB-month of physical Valkey storage
• Cluster applies a 25% surcharge to the list price for every deployed instance
• Data transfer includes 200 GB of ingress + egress each month, then $0.05/GB

Momento Cache supports rapid autoscaling within a specified capacity range, making it easy to reduce the cost of idle resources.

Designing for scale

Momento Cache employs a two-tiered architecture that significantly improves reliability and efficiency at scale. Each valkey cluster sits behind a gateway that handles the hard traffic problems like hot keys and connection storms before they hit the data layer.

The gateway exposes a RESP endpoint, so Momento Cache is compatible with all standard Valkey and Redis clients. It masks the underlying cluster, presenting a single stable endpoint across any topology changes.

block
  app("Redis or Valkey\nclient"):3
  space
  gateway("gateway"):3
  space
  pool("Valkey cluster"):3

  app --> gateway
  gateway --> pool

The gateway is optimized to quickly process TLS, auth, rate limits, and other traffic management concerns at high concurrency and high throughput. It multiplexes client traffic across a pool of warm connections to the Valkey cluster. This reduces connection latency, and enables advanced capabilities like request coalescing.

The result is an efficient system with deliberate separation of responsibilities. The gateway absorbs traffic concerns, while Valkey nodes focus their resources on handling data.

Get started with Momento Cache

Try out Momento Cache in a few short steps with the Momento CLI. While the service is still in preview, you’ll also need to request access via the web console.

First, create an API key in the Momento console, copy the endpoint for your region, and configure the default CLI profile:

# paste the api key and endpoint when prompted
momento configure

Then, create a Capacity Pool. Be sure to provide valid zone IDs for your region:

momento preview pool create \
  --name example-pool \
  --capacity-gib 32..128 \
  --replicas-per-shard 1..2 \
  --zones use1-az1,use1-az2

momento preview pool describe --name example-pool

Once the Capacity Pool status is active, you can create a Database:

momento preview database create \
  --name example-db \
  --pool-name example-pool

Back in the console, open the pool’s Databases tab and copy the regional RESP endpoint. Connect any Valkey or Redis client in standalone mode to this endpoint over TLS on port 6379. The Database name is the username, and your Momento API key or token is the password. Here, we’ll demonstrate with the official valkey cli:

valkey-cli -h <resp-endpoint> -p 6379 --tls \
  --user example-db --pass <momento-api-key>

> SET example-key "ready"
OK
> GET example-key
"ready"

Congratulations! You now have a high-performance Valkey cluster ready to go.

Next, check out the docs to learn more about Momento Cache’s capabilities. Or, if you want to push your cache to the limit, load up cachecannon on a c7g.xlarge instance in the same region and zone as your Database!

Up next

This launch begins the next chapter for Momento Cache. Stay tuned as we port more features from our enterprise services into Momento Cache, including VPC peering and S3 integration!

We’re looking for feedback from teams operating demanding Valkey workloads as we refine the product. If you’re building a fast-growing product, operating a large Valkey or Redis cluster, designing an internal caching platform, or helping teams adopt Valkey, we would love to hear what you need next and where we can help out.

We’re grateful to the engineers and partners who turned years of demanding operating experience into a service anyone can start using today. Special thanks to Dylan Abraham and Jason LaPier for leading the development effort!

But doubling the number of indexes I was using made me nervous. I didn’t know what impact that would have on performance. I’m using Valkey because I needed ultra-low latency, I didn’t want to shoot myself in the foot trying to be clever.

And yes, going from one to two indexes is probably not a big deal. But indexes have a way of piling up. You create one to filter products by category, then add price to it a month later, then the search team wants a vector field, then the payment team indexes their keys, then fulfillment indexes theirs. Before you know it, there are a dozen FT.CREATE statements in the repo with no single owner.

My instinct tells me that performance scales inversely with the count. Every index in valkey-search subscribes to a key prefix. When you write a matching key, that index gets an entry in a mutation queue, and your client stays blocked until every entry is processed. That’s what gives you read-after-write consistency. If you have a dozen indexes, you also have a dozen entries, and a dozen times the wait. Right?

So I built a benchmark to see how much additional indexes cost in valkey-search. And I discovered I was very wrong.

The setup

The server was an r7i.4xlarge with 8 physical cores, running Valkey 9.1.1 and valkey-search 1.2.1. valkey-search sized its own writer pool to 8 threads. I turned persistence off, because I didn’t need a background save forking mid-run throwing a wrench into my latency numbers. Load came from a separate c7i.8xlarge in the same placement group, over 384 connections.

Every write is the same 4.6KB product hash that includes a category, a price, a SKU, a title, a description, and a 1024-dimension embedding. The load generator sends on a schedule and times each request from when it was supposed to go out, not when it managed to (more on this later). Every point runs for 25 seconds, three times, and I capture the median.

Do idle indexes cost anything?

As I mentioned earlier, indexes subscribe to a key prefix. If a key is upserted that doesn’t match on an index, does that slow things down? In other words, my test here determines if the existence of indexes slows down the performance of others.

I ran the same write workload against product: keys in two setups: one index on product:, and another one with the same index plus fifty more indexes on unrelated prefixes.

Nice! I couldn’t measure a difference between them at any rate I tried.

This is because the prefix subscriptions live in a trie. When you write product:88213, valkey-search walks the trie and only notifies the indexes whose prefix matches. An index subscribed to orders: doesn’t hear about it. FT.INFO idx:other0 reports a field called mutation_queue_size, and across the entire run it never left zero, which validates the promise of the trie architecture.

So that hodgepodge of indexes in your repo isn’t what’s slowing your writes down. Only indexes whose prefix matches the keys you write are in the path at all.

The impact of matching indexes

So what’s the performance impact of having multiple matching indexes? I ran the same workload again, this time adding multiple identical indexes directly on the product: prefix.

Every matching index gets an indexing job, and the write isn’t done until all of them are. To figure this out, I took Valkey’s search_ingest_hash_keys counter and divided by the writes that completed in the same window. I took this number and multiplied it by the write rate to calculate the indexing jobs/sec column.

The expensive part, naturally, is switching indexing on in the first place. Going from zero indexes to one cost me 58% of my write rate. The second index cost another 41%. Doubling from two to four only cost 16%. Each additional index cost less than the previous one.

Adding indexes pulls more total indexing work per second out of the same server, 33,636 jobs a second at one index and 67,272 at four. So one or two indexes clearly weren’t saturating the writer pool, because it eventually went on to do twice the work.

Indexing jobs fan out across a pool sized to your physical core count, and the blocked-client handles collapse into a single block on your connection. The write waits for the slowest single job, which is why four indexes don’t cost four times what one does.

Unfortunately I don’t know what the limiting factor is. My hunch is that it’s the per-index bookkeeping that happens on the main thread before a job reaches the pool, but I didn’t measure that, so 🤷.

NOTE - Adding matching indexes won’t appear to add latency until it’s too late. At 2,000 writes/sec, one index and four indexes were within a millisecond of each other. You won’t catch it watching p99 on a healthy system, because what you’re spending is headroom. You find out it’s gone when you need it.

Vector fields hit different

The benchmarks above use TAG and NUMERIC fields, just the normal filter field types. So I went back to the first benchmark, added a single 1024-dimension HNSW vector field to the one-index setup, and re-ran it to find some staggering results.

32,000 writes per second became 2,828. 🤯

That’s an 11x difference in throughput because of a single field. My napkin math says that’s 2-3 milliseconds of CPU per vector insert spread across eight writer threads. To make matters worse, the performance cliffs. I increased the rate on my benchmark by ~40%, and the wheels fell off.

The performance hit goes from five milliseconds to two seconds. The queue went from basically empty to 376 entries deep, and it stayed there for every rate I tried above that. You’re either under the line and fine, or over it and everything is late. If you’re capacity planning, be sure to check whether an index on your hot prefix has a vector field.

Do vector fields affect other matching indexes?

Back to the search feature I was building that required two indexes. A single index can’t do both jobs because of stop words. The text pipeline is configured for the entire index, and it splits words on punctuation before dropping anything in the stop word list. So IT-500 becomes it and 500, it is a stop word, so just 500 is added to the index. Turning on NOSTOPWORDS means your description field will index every the, is, and, and it (plus a lot more) in the catalog. So we need two indexes to have it turned on for one and off for the other.

But that made me wonder what the second index costs when the first one has a vector field. We saw how much of a hit it made to throughput in our earlier benchmarks.

About a three percent difference. The exact match index processed 141,408 text fields during the benchmark run. Both mutations hit the pool together, the write waits for the slower one (the vector). So if you’re already paying the HNSW tax, the lexical index has essentially no additional latency.

My takeaways

So it turns out I was asking the wrong question when I started this experiment. I thought the number of indexes I had was going to slow performance down to a crawl. But it doesn’t. The real question is what fields are inside the indexes that match your keys? Which is a relief, when I think about it.

I also learned a couple of things about benchmarking while I was busy answering the wrong question. 😅

Every sustained writes/sec number in this post could have been bigger. At the top rung of my ladder, the no-index setup completed 128,000 writes a second (but my tables show 80,000). At that run rate, it was making every request wait 1.9 seconds in a queue first. A server running at full utilization drains as fast as it fills, so throughput looks perfect, but at a cost to latency. So the number I used in every table is the highest rate where p99 stayed within reason.

It took me three runs to believe what I saw in that four-row table early in this post. The first run stepped the rate by 1.4x per rung, which put two indexes and four indexes on the same 16,000 rung. Which at first made me think indexes three and four had no performance implications. But in reality, they had both fallen apart somewhere between rungs and the ladder couldn’t show me where. So I re-ran it with 1.19x steps and got a cleaner separation. Then I noticed that ladder started at 14,000, which is already well up the curve, so it never measured what latency looks like when the server is idle. My rule for picking a sustainable rate is relative to that idle number, which meant the second run was grading itself on a curve. The third run started at 2,000 and is the one in the table. A rate ladder can’t resolve a difference smaller than its own step, and it can’t tell you where the knee is if it never saw the flat part before it.

Try it yourself

I have the benchmark, scripts, and results available in GitHub. If you want to check my numbers (or disagree with them!), please do and let me know what you find.

If you want to run the same tests on your own Valkey cluster, you can run these three commands:

FT.INFO <index> # shows mutation_queue_size (write backlog) for the specified index
INFO search # shows search_writer_queue_size for the whole pool
CONFIG SET search.info-developer-visible yes # unlocks per-field-type counters like search_ingest_field_vector

It’s cheap to experiment with your existing clusters because these mutations are reversible. FT.DROPINDEX is instant and your data was never in the index to begin with. Add a shape, measure it, then discard it.

Happy coding!

Unfortunately it’s not that simple. Memories need to be similar, yes, but they also need to be recent. You don’t want memories from a year ago influencing your agent. And outcome is important too. If one approach worked and another failed, you probably want the successful one steering behavior.

I ran into this while building a small demo that puts Valkey Search inside an agent’s inference loop. Every task the agent finishes gets written as a memory. It persists the task text, the approach that worked, whether it succeeded, when it happened, and a vector of the task. Next time a similar task shows up, the agent recalls what worked instead of rediscovering it.

I ended up learning how valkey-search handles this the hard way, because my first FT.SEARCH call did not work the way I expected:

FT.SEARCH idx:memory "(@outcome:{success} @created_at:[1782900000 +inf])=>[KNN 3 @vector $vec]" PARAMS 2 vec <query-vector> DIALECT 2

Left of the => is an ordinary boolean filter of a tag and a numeric range over two fields I indexed alongside the embedding. Right of it is the vector search.

I thought Valkey was going to find the nearest vectors and apply the tag and timestamp filters afterward. But it doesn’t work like that (in a good way). The filter criteria are actually inputs to the query planner. Depending on how many memories they match, Valkey chooses a different algorithm to search the vector index. In other words, adding a filter changes how the search runs, which meant I needed to rethink my initial memory schema. Turns out my retrieval policy was also an index-design decision.

Nobody post-filters anymore

If you’ve used Pinecone, Qdrant, Weaviate, or Milvus, then this should be familiar. All of them decide at query time whether to walk the graph with your filter applied or abandon the graph and brute-force the matching subset instead. Pinecone calls it single-stage filtering. Qdrant calls it query planning. Weaviate calls it a flat search cutoff. Milvus doesn’t really call it anything, it just does it 😂. valkey-search is the same idea, and on the FT.SEARCH vector path it doesn’t implement post-filtering at all.

If you’re a pgvector user, however, filtering happens after the index scan. With the default hnsw.ef_search of 40, a filter that keeps roughly 10% of those candidates might leave you with only 4 results. Iterative index scans were added in 0.8.0 to make that a little better, and they’re off by default.

valkey-search calls it a query planner. It estimates how many keys your filter matches and picks one of two algorithms to perform the search.

If the estimate is small compared to the index, it pre-filters results by walking the qualified key set, computing each distance directly, and keeping a top-k heap. The HNSW graph is never traversed. So valkey-search is essentially doing a brute-force scan over a tiny set.

If the estimate is large, it performs an inline filter. Your predicate is handed to hnswlib as an isIdAllowed functor and evaluated during traversal of the base layer. Non-matching nodes are still visited and expanded, they just don’t get added to the result set.

The cutoff point for one algorithm vs the other is 0.001. So if the number of estimated matching keys is at most .1% of the number of vectors in the index, it will go the brute-force route. Otherwise it uses the inline filter.

For reference, the cutoff point for Milvus is around 7%, which makes valkey-search about 70x stricter. You end up on the inline path more often than you’d think.

Make the filter disappear

On my demo index with a few hundred memories, the threshold works out to well under 1 key, so every recall that matches anything takes the inline path. You’d never notice either way at that scale.

But what would happen with the same schema in production with 1,000,000 memories? The threshold is 1,000 keys. The planner considers the selectiveness of the filter. If @outcome:{success} matches 70% of the index and your @created_at window is also broad, you’re squarely in the inline path. And the planner is right to put you there. If you scope recall to a single tenant with 400 memories, you drop under the threshold, and the query becomes an exact scan. Perfect, fast recall because 400 distance computations is nothing.

Let’s make it more difficult. A filter matching 1% of a large index sits 10x above the cutoff point, so it takes the inline path, where HNSW traverses a significant number of candidates for every one it’s allowed to keep. That results in a lot of extra latency you didn’t account for. And you can’t change that by tuning it, because search.prefiltering-threshold-ratio is immutable unless search.debug-mode is on, and it isn’t in the public configurables table at all.

You can check which path you’re actually on, by the way. valkey-search counts both, named search_prefiltering_requests_count and search_inline_filtering_requests_count. They’re module fields, so you need INFO SEARCH and not plain INFO. You can run your recall query a hundred times to see which one moves.

So your best bet is the index itself. Make the filter act like a namespace. Put a hash tag in the index name, prefix the keys to match, and each tenant’s recall hits one shard against a small index where the filter is mostly irrelevant. Valkey enforces it in both directions, too. A tagged index name requires every prefix to carry the same tag, and an untagged one requires that none of them do. Be careful with this though, because it’s not easy to undo if you change your mind since there’s no FT.ALTER here.

Forgetting is expensive

Removing a vector from an HNSW index calls markDelete and returns. The node isn’t removed from the graph, it’s still visited, it still routes other searches through itself, it just fails the deleted check. search.hnsw-allow-replace-deleted would let the next insert reuse that slot, but it’s false by default and more of a non-production flag. In production the space is stranded until you drop the index.

Luckily, updates are a different story. Modifying an indexed vector routes to an in-place update under the existing label, so the node keeps its slot and just gets its links rewired. Writing a hash field that isn’t the vector doesn’t mess with the graph, and rewriting the vector with identical bytes short-circuits before it gets there.

So updates are cheap, but deletion is where it gets expensive. That includes DEL, eviction, and expiry.

That’s a little scary, because expiry is the recency policy. valkey-search subscribes to generic, expired, and evicted keyspace notifications, so a TTL’d memory really does leave the vector index when the key goes away. Which is what you want, but it’s also what makes graph nodes stranded.

Which means you have to change how you key your memories. Creating a new key every run feels like a safe and reasonable default, and it’s what my demo does with its memory:<id> per completed task. But add a TTL to keep things fresh, and every expiring key leaves a node behind. That’s an expensive default at scale. Instead, use a stable key derived from a task fingerprint, and update it in place as the agent learns more about that kind of task. This means the task costs only one node instead of one per attempt.

There are two recency controls here to consider. The @created_at range decides what you’re willing to believe on any given query, and it doesn’t delete anything. The TTL decides what you’re willing to pay to store. Range should be your first lever and the TTL should be the slower fallback.

Decide before you have data

This demo surprised me twice: the filter I wrote as a query detail decides which algorithm runs, and the TTL I (almost) added to keep memories fresh led to stranding the index.

Everything here is discoverable, at least. The planner code is a short function to read and understand. The filtering comes from hnswlib. Even the deleted node behavior is a comment in the source, and the counters are available in INFO SEARCH. You don’t have to take my word for any of it (but you should 😜).

What you can’t read your way out of is when you have to make decisions. How the index is scoped and how a memory is keyed are day-one calls, made before you have a single memory to check them against. Outside of that dev-only flag, a rebuild is the only way to reclaim space that deletion stranded. Get those wrong and you’re reindexing.

The demo is on GitHub if you want somewhere to start. docker compose up -d gets you Valkey with the search module and an agent that writes its own memories. Run a few tasks through it, then look at INFO SEARCH and find out which path your queries are actually taking.

Happy coding!

Last year, the first edition of Buffer-Free Video)^AI left me in disbelief that a scrappy ten-week sprint had pulled 150 streaming professionals into one room. This year the question in my head was different. Not “can we do it again?” but “can we keep the signal-to-noise ratio that made the first one special?”

I believe we did, and the numbers back it up.

The Room We Wanted

Here is the stat I am proudest of: we had 117 attendees show up out of 124 registrations. That is a 94.4% show-rate. If you have ever organized a tech conference, you know the typical number hovers around 60%. People register, life happens, half the badges never get picked up.

That did not happen here, and it tells you something about who was in the room. Definitely not people collecting free T-shirts, stickers, Mo plushies or vintage green Elemental swag (thanks Kiran!). Of the seven no-shows, six were from sponsoring organizations who had other folks covering the event.

Those 117 people represented 41 unique organizations, from the largest broadcasters, tech platforms, video infrastructure providers… the full spectrum of who actually builds and runs media workloads at scale. When you get that kind of density of real operators in one place, the hallway conversations become as valuable as the sessions. Actually, maybe even more.

We Pre-Screened the Pitches Out

Here is the thing we are strict about. BuffConf is not a place to sell. Every proposed session got human-screened, and anything that read like a vendor pitch got sent back. We prioritized end-user and practitioner voices, the people who have actually been paged at 3am when things flinched.

The result was 22 technical sessions with 29 speakers from 17 companies. And critically, 40% of our speakers were non-vendors. We are talking real production stories from operators like Paramount, Netflix, FOX, CBS Interactive, Meta and Red Bull Media House. Not “here is our roadmap.” More like “here is what we tried, what failed and what finally held up.”

It would be easy to fill an agenda with vendors who have marketing budgets and room for another video nerd conference (all the more when the traditional bigger one is seemingly canceled silently this year… of course you know what I’m talking about!). It is much harder, and also much more valuable, to get the operators to stand up and tell the unvarnished truth about their infrastructure. When they do, everyone in the room learns something they can use in their own work.

What The Presentations Covered

Three main themes ran through the two days, and a handful of sessions captured each one.

Production workflows at scale. I opened with “104 Matches. Five Billion Viewers. Zero Margin for Error,” walking through the FIFA 2026 World Cup numbers and what global live scale actually demands, with no excuses. Spencer Shanson from Paramount then told the story of scaling a FAST platform from a handful of channels to thousands. That kind of journey, from a few streams to an entire catalog of linear channels, is exactly the operational reality most of the room is living on a daily basis, and hearing it from someone who has actually done it is worth more than any architecture diagram. And when “production at scale” means “at Netflix’s scale”, who better than Sunjeet Singh and William Schor could explain how Netflix handles over one billion RPS on 120,000 nodes?

Practical AI, and honest boundaries around it. This was the part I found most refreshing. Nobody stood up and told us AI was going to solve everything. It’s still a journey, and our keynote speaker, Manish Rao, showed us all the progress that AWS Elemental Inference has made since NAB. It’s staggering how three months can mean a world of difference at the current speed of AI innovation. Edwin Rivera from CBS Interactive showed how CBS Sports built an AI-powered “For You Page,” blending editorial curation with embedding-based recommendations to actually move engagement. Content discovery in sports is one of the richest problem spaces in our industry right now. And Jeffrey Kember from NVIDIA walked through what accelerated computing is doing for encoding, quality, and real-time media workflows. A reminder that we are genuinely at the beginning of AI for media, not the end. Across the AI sessions, we also saw token efficiency work borrowed straight from video compression concepts, automated QC loops, and an edge anti-piracy simulation harness. Real implementations, with the scars to prove it, paired with hands-on framework training so engineering teams could take something home and build.

The client-side reality. Our panel on Optimizing Playback SDKs and Device Capabilities brought together Jonathan Colwell from FOX, Joshua Lamb from Red Bull Media House, and David Van der Voort “DVD” from Paramount. Three very different takes on the same daily chaos: supporting every device under the sun, with every usage pattern. That fragmentation is a grind everyone in playback knows intimately, and getting three operators to compare notes in the open was exactly the kind of session you cannot get anywhere else.

We kept the format balanced through all of it. Deep technical talks, real networking time, and interactive workshops where people could get their hands dirty. Room to think, argue, and go deeper.

The Q&A Is Still the Best Part

Same as last year, the Q&A sessions were electric. A technical crowd asked technical questions, and every speaker rose to meet them. Those exchanges are where a prepared talk turns into something better, where you get past the slides into the messy details that only come out when a peer in the audience pushes on exactly the right point.

That is the whole reason we do this in person. You cannot replicate that online.

Thank You

A conference only works because of the people who believe in it before it exists.

To our speakers, from Paramount, Netflix, CBS Interactive, FOX, Red Bull Media House, NVIDIA, Meta, AWS, Visionular, Bitmovin and every company that sent someone to share hard-won lessons: thank you for choosing substance over spin. You are the reason people fly in.

To JP Saibene and Nicolas Gonzalez from Uruguay, to Ali Begen from Tűrkiye and to everyone who traveled a long way to be in Seattle: thank you for bringing the curiosity and the generosity that makes this community what it is.

To the Momento crew who pulled this together, and in particular Hannah, Mike, Kelsey, Maéline and everyone who spent months making a two-day event look effortless: thank you, I’m incredibly grateful and lucky to be working with you all.

And to the people who showed up: this community was the event. Your questions, your stories, your willingness to say “actually, that broke for us too” is what makes this community worth gathering. The streaming landscape keeps moving fast, especially where AI meets live video. We were all here to discuss the future of video, one honest conversation at a time.

See you next year.

That should feel stranger than it does. When we typically think about inference, speed is a whole thing. Time to first token (TTFT), tail latency, and shaving milliseconds off the round trip so a human can stay in the flow are all you see in articles these days. Yet there I was running one of the most token-hungry things from my laptop, and I didn’t really care how long it took.

Meryem Arik, CEO of Doubleword, was able to articulate really well how I felt about it in her recent talk Inference for Async Agents in Production. She posed the simple question “who’s waiting?” that made it all click for me.

When nobody’s waiting, the cost of inference looks entirely different.

Schedule inference like nobody’s watching

In a synchronous system like chat, a human is directly in the flow. Make them wait without any feedback and they start clicking around wondering if anything is happening. In an async system, an agent might fan out, run for thirty minutes, and ping you when it’s done. Nobody is watching the screen. In many instances, it doesn’t really matter whether it comes back in 35 minutes or 45, because the user has gone off to do something else.

To me, this reads like a UX detail, but it drives almost every other decision in your stack.

“Who’s waiting?” is the right place to start. More specifically, how long is the work allowed to wait and what is blocked behind it? Even without a person staring at the screen, a call on an agent’s critical path still has a deadline. Async workflows just give us more room to decide what that deadline actually is.

A human waiting is the most expensive clock in the system. “Fast enough for a person” forces things like how you provision, how much you batch, which chips you buy, and how much idle capacity you keep on standby. Saying “I want an answer now” makes everything expensive. But if you take the human out of the loop, that clock loosens its grip, along with the price tag it was setting for everything else.

We reinvented batch processing

We’ve done this a time or two in the past with things like interactive database queries versus overnight analytics jobs. We prioritize the web request a user is staring at versus the batch job that runs at 2 a.m. The tech industry has been separating “someone is waiting” from “nobody is waiting” since before I was born. 😅

Inference is speed-running what databases and job queues learned decades ago, but in a good way. When a young field’s hardest problem is something the infrastructure world already recognizes, you can jump straight into borrowing patterns that are proven to work.

Meryem frames the core tension as the old trade-off triangle: latency, quality, throughput. You get to pick two of them.

Chat apps pick latency and quality. Async agents pick quality and throughput, and they let latency slide. A long-running agent mostly needs the smartest model it can get so it stays coherent across a two-hour piece of work while it manages subagents and juggles tasks. Delivering tokens at conversational speed doesn’t buy it anything.

Batching is really a scheduling problem

Batch-size math can be tricky. In one example from Meryem’s talk, a model running at batch size one and tuned for maximum interactivity costs roughly $4.65 per million tokens. Push enough concurrent work through the same model and tune for throughput instead, and that cost comes out closer to nine cents per million tokens.

In that example, the same weights and quality produce a 50x difference in cost. That makes a substantial difference to your unit economics.

But there’s a catch. Bigger batches produce their best economics only when you have enough compatible work waiting to keep them full. An empty slot in a batch is silicon you’re paying for that isn’t producing tokens. The hard part is keeping batches full while demand stays bursty and wildly unpredictable.

I’ve made this argument from the other direction before, that GPUs are the most expensive resource in tech and we use them badly. Batch size is the same thing. High utilization is paramount, and you win or lose it in the scheduler.

The queue is the product

I really liked how Meryem put it: most providers lose money on real-time serverless endpoints. It comes back to the “now” tax. To promise a human fast responses against lumpy traffic, you have to provision for the peak and then eat the idle troughs in between. That idle compute is money on fire.

If the work has room to wait, the batch queue becomes a powerful tool. You can push work into the low periods and backfill the troughs that are already being paid for. Spot instances become fair game for soaking up spare compute when the work can tolerate interruption or be checkpointed. SLA-aware routing lets you jump an urgent request ahead of the rest. You can even reorder the queue so requests that hit the same mixture-of-experts weights run back to back and skip reloading those experts every time.

Once you stop committing to “now”, the scheduler/router/whatever is holding the queue becomes the center of the system. It’s where the value collects, and it’s where everything is heading. As hardware and models commoditize, the advantage moves up into routing, placement, and orchestration, and whoever owns the “what runs next” decision owns the economics.

Start labeling the wait

I mark everything real-time out of habit. Every call gets treated as if a human is holding their breath on the other end, even when the “human” is an agent off doing six other things. That results in an unnecessary “now” tax on work that doesn’t need it.

Doubleword has categorized latency into real-time, async (about a minute), and batch (24 hours) buckets. The 50× cost gap we mentioned earlier comes down to which bucket you pull from, and many of us (myself included) reach straight for the expensive one by reflex.

One of the most boring-yet-useful things you can do immediately after reading this article is going through every call your agent makes one at a time and ask whether a human is waiting on each one. You might be surprised how few are. The background call fetching context isn’t waiting. The sub-agent grinding through step four of eleven isn’t either. The nightly summarization job definitely isn’t.

In the near future, we’ll have routers smart enough to infer a request’s latency tier from its context, dependencies, and deadline, so we don’t have to label everything by hand. Once we figure out how to do that efficiently, system costs drop dramatically because the scheduler can optimize around how long each piece of work is actually allowed to wait. Until then, start by giving every model call a realistic latency budget instead of letting the entire workload take on “now” by default.

Everything we’re talking about has been around in the infrastructure world for years. The same instinct drives shared caching at Momento: preserve expensive state, reuse work you have already paid for, and keep scarce resources productive instead of idle. Inference gives those old ideas a very large new price tag.

If you’re wrestling with any of this, we’d love to compare notes.

Happy coding!

If you came here looking for a post from me on consistency models, I have to disappoint you. Today, I want to talk about the value of consistency in life. In life, effort is additive, but consistency is multiplicative.

Over the last few years, I have had the honor of watching the Valkey project blossom from an idea into an inspirational, community-driven effort with consistent improvements in each release. These improvements range from memory efficiency to availability at scale to substantial performance gains. These small improvements compound.

This compounding effect and the power of a driven community is perhaps best illustrated by the journey of the I/O-threading architecture and its impact on large objects from 1 MB to 64 MB in Valkey. Larger items are becoming increasingly important for inference KV caches, 4K video streaming, and other enticing use cases. At the very least, the impact of large objects should not be overlooked, as they can impact everyone else’s latency; this post measures how fast the large objects themselves go.

Buckle up, because it’s about to get spicy.

A brief history of I/O threads

Valkey 7.2 inherited an I/O-thread model from a software stack doing its best to stay single-threaded. The main thread and I/O threads worked in coordinated phases, separated by synchronization barriers, and the 7.2 config file is candid about the result: “Usually threading reads doesn’t help much.”

Valkey 8.0 delivered a fundamental rearchitecture of I/O threads, tripling throughput to over a million requests per second. This idea was not new. In August 2023, seven months before the fork, Dan Touitou filed redis#12489, laying out exactly this design in detail, benchmarks included.

Redis let #12489 sit. The issue is still open in the tracker today, unassigned and without a milestone.

Within days of the fork, the Valkey community copied the proposal verbatim into issue #22, greeted it as “a true gem,” and shipped it. The new architecture enabled continuously running I/O threads connected by queues, so reads, parses, and writes proceed on separate cores while the main thread executes commands. Valkey 8.1 delivered TLS handshake offload to I/O threads in #1338.

Redis shipped a strikingly similar asynchronous I/O-threading model in Redis 8 in May 2025, a year after the fork and 21 months after the design landed in its own tracker. Around the same time, we put Valkey 8.1 and Redis 8.0 head to head on small objects. Valkey 8.1 outran Redis 8.0 by 37% on writes and 16% on reads.

Valkey 9.0 brought reply copy avoidance, changing the performance of large items entirely. Before this change, the main thread copied the entire object into a connection reply buffer before moving to the next command. While a large item is being copied, the entire pipeline stalls. Small objects are not serviced until the copy finishes.

Valkey 9.0 instead passes a reference to the I/O threads and keeps the object alive with a reference count. The I/O worker for that connection hands the object’s memory directly to writev(). This shortens the handoff to the I/O threads and gets the main thread back to handling requests.

Valkey 9.1 then redesigned communication between the main thread and I/O threads around lock-free queues, credited in the release notes with an 8-17% throughput gain.

Based on these changes, we expected 8.0 to lift reads and writes for smaller items but still struggle to fill the network link for larger items. We expected 9.0 to bring GETs to line rate and 9.1 to improve writes.

This is what open-source competition buys everyone, including teams that never leave Redis. A performance design that sat for seven months as an unassigned issue became table stakes for both projects within two years of being filed.

Show me the numbers

We swept values from 1 MB to 64 MB across four Valkey releases on two nodes with 200 Gbps of bandwidth between them. The full setup is below.

Valkey 7.2: reads capped at 35 Gbps, writes at 55

An 8 MB value delivers 21 Gbps on reads and 23 Gbps on writes, with p99 latencies of 174 and 164 ms. Turning on io-threads-do-reads moved only the 1 MB read cell.

Valkey 8.0 and 8.1: writes take off, large reads stay put

The threading rebuild lifts our 8 MB write from 23 to 137 Gbps, six times faster, and 1 MB reads reach 183 Gbps. Larger reads settle at 30-33 Gbps whether the value is 8 MB or 64 MB. That flat floor points to a serial, per-byte bottleneck. Valkey 8.0.9 and 8.1.8 measured the same at every size, so one line carries both.

Valkey 9.0: large GETs jump to line rate

Every size from 1 MB to 64 MB reads at 190-201 Gbps. The 8 MB p99 falls from 112 to 24 ms, and a 64 MB read drops from just under a second to 230 ms. Writes do not move because the ingest path was never copy-bound. That ceiling waits for 9.1.

Valkey 9.1: more headroom for writes

With reads pinned at the network limit, the gain surfaces on writes. The 8 MB write rises from 134 to 166 Gbps, a 24% gain, while values from 12 MB to 64 MB gain 11-19%.

The short run held

To make sure the 15-second runs were not catching a lucky window, I reran every 8 MB GET and SET cell for 15 minutes. Valkey 9.1 is a good example: GET held 200.8 Gbps and SET held 165.5 Gbps, right on top of the original 201 and 166 Gbps results. Nothing sagged as the runs went on.

Detailed results

Reads (GET)

GET-only, 32 connections, 100% hit rate.

GET bandwidth (Gbps)

Note: Valkey 9.1 reads also hold line rate, so the table stops at 9.0.

GET p99 latency (ms)

Writes (SET)

SET-only, 32 connections.

SET bandwidth (Gbps)

SET p99 latency (ms)

The setup

Machines. Two c8gn.16xlarge instances with Graviton4 and 200 Gbps networking in the same availability zone and cluster placement group, running Amazon Linux 2023.

Server. We tested the official valkey/valkey Docker image at versions 7.2.13, 8.1.8, 9.0.4, and 9.1.0. We also measured 8.0.9. It matched 8.1.8 within run-to-run noise at every size, so the tables show 8.1 as the 8.x column. Valkey 7.2 ran with the same flags as the newer versions. A control with io-threads-do-reads yes changed only the 1 MB GET cell, from 32 to 41 Gbps. Each version ran in a fresh container with host networking:

docker run --network host --cpuset-cpus 8-23 \
  --ulimit nofile=32768:65536 valkey/valkey:<version> \
  --save '' --appendonly no --io-threads 16 \
  --protected-mode no --maxmemory 60gb

Persistence was off. The 16 threads in Valkey’s io-threads count were one main thread plus 15 I/O workers. The process was pinned to cores 8-23 so it never fought the kernel for the cores doing network interrupt work.

Interrupts. irqbalance was off on both machines. The ENA NIC was configured with four combined queues and their IRQs pinned to cores 0-3. Without this step, results wander from run to run as the kernel shuffles interrupts onto whatever cores the server or client threads happen to be using. If you benchmark at these speeds, pin your IRQs first and thank yourself later.

# Run on both machines. ens50 is the ENA interface name on these instances.
sudo systemctl stop irqbalance
sudo ethtool -L ens50 combined 4
i=0
for irq in $(grep ens50 /proc/interrupts | awk '{print $1}' | tr -d ':'); do
  echo $((i%4)) | sudo tee /proc/irq/$irq/smp_affinity_list > /dev/null
  i=$((i+1))
done

Client. We used valkey-lab, built on cachecannon, with 16 worker threads pinned to cores 4-19, 32 connections, and pipeline depth 1. Reads and writes were measured in separate passes. Read passes prefilled the keyspace and ran at a 100% hit rate. Each short-run cell was a 15-second measurement after a five-second warmup, over a 500-key keyspace with 16-byte keys.

We kept concurrency at 32 connections because AWS caps a single TCP flow at roughly 9.5 Gbps. We verified 9.53 Gbps with iperf3. Saturating a 200 Gbps NIC requires spreading the load across flows.

One invocation per cell, with -s swept across the value sizes:

# GET pass: prefill the 500-key keyspace, then measure at a 100% hit rate.
valkey-lab -h <server> -c 32 -P 1 -t 16 --cpu-list 4-19 \
  -n 500 -r 100:0 --prefill --warmup 5s -d 15s \
  -s <value_bytes> --key-size 16

# SET pass.
valkey-lab -h <server> -c 32 -P 1 -t 16 --cpu-list 4-19 \
  -n 500 -r 0:100 --warmup 5s -d 15s \
  -s <value_bytes> --key-size 16

Scope of the result. This is a single-node, no-TLS, persistence-disabled test with 32 closed-loop connections, pipeline depth 1, and a 100% hit rate. Most table cells are one 15-second measurement after warmup. The results describe this rig and workload. Broader production claims need repeated runs and additional configurations.

We benchmarked whole releases rather than individual changes, so we treat the pull requests named in this post as leading explanations rather than proof of causality.

What this means if you run Valkey

Valkey 9.x delivers roughly 6x the 64 MB read bandwidth of 8.x while cutting p99 from just under a second to about 230 ms on this single-node, no-TLS test. Our earlier mixed-workload test also found far less collateral latency for small requests when a large read arrived.

The practical takeaway is narrower than “Valkey is always faster.” Valkey 8.1 is essentially flat on this workload, Valkey 9.0 changes the large-GET path, and Valkey 9.1 improves large SETs on this rig. If large values matter to your workload, test 9.x with your object-size distribution, concurrency, TLS, and persistence settings rather than extrapolating from a small-object benchmark, or from this one.

Valkey has consistently improved performance, memory efficiency, and availability at scale. Each version brings about a new set of improvements driven by issues faced by real users in production. A vibrant community where nobody is incentivized to withhold features for the sake of revenue and everyone is incentivized to chase continuous improvement is what makes Valkey truly special. Effort is additive. Consistency is multiplicative.

Serving megabyte-sized objects at wire speed is what I’ve been spending a lot of time on lately. If you are wrangling larger objects, let’s talk. Join me on Valkey Slack.

On a single L4 running a Qwen3-4B coding agent, the server holds 12 concurrent sessions at p99 under 2.6 seconds. Add two more and p99 jumps to 39 seconds, a 15× increase in a single step. The cliff is the moment the KV cache fills.

An agentic coding workload on commodity hardware

Our setup is a single g6.4xlarge EC2 instance with one NVIDIA L4 GPU (24 GB VRAM), running Qwen3-4B with FP8 weights on vLLM 0.20.2, automatic prefix caching (APC) enabled.

The workload models a lightweight coding agent mid-task. Each session opens with a 10,000-token shared system prompt (repository context plus agent instructions), followed by a 12-turn conversation where each turn appends about 1,000 tokens of unique context (tool call inputs, code snippets, responses). At turn 12, total context reaches roughly 22,800 tokens per session. Output is capped at 75 tokens per turn. The workload is heavily input-dominated, as agentic workloads tend to be.

System prompt:          10,000 tokens  (shared across sessions → APC cached)
Per-session turns:      ~12,800 tokens across 12 turns  (unique per session)
Output per turn:        75 tokens
—————————————————
Total at turn 12:       ~22,800 tokens

APC caches the system prompt once and amortizes its KV cost across all sessions (the blocks still occupy GPU memory, but only one copy exists). Within a session, APC also caches the growing turn history. Turn n+1 extends the exact prefix from turn n, so each turn only prefills the new ~1,000 tokens, as long as the prior turns’ KV blocks survive in cache. But the per-session turn history is unique (different code, different tool outputs) and cannot be shared across sessions. When concurrency pressure forces eviction of a session’s blocks, the next request in that session has to re-prefill the full accumulated history (up to ~12,800 tokens at turn 12). That re-prefill cost drives the TTFT cliff.

We swept concurrency from 1 to 48 sessions, measuring TTFT at each level, across three KV cache precisions, fp16, fp8, and TurboQuant 4-bit. We also ran best-case (all context cached) and worst-case (all context re-prefilled) bounds to bracket where realistic performance should land.

The concurrency cliff

The chart below shows TTFT percentiles (p50, p95, p99) and throughput across the full concurrency sweep with fp8 KV cache. The Y axis is logarithmic, and even on a log scale the cliff is sharp.

TTFT p50, p95, and p99, with throughput on the right axis, versus concurrent sessions. fp8 KV cache, Qwen3-4B FP8 on a g6.4xlarge (L4). The shaded band marks the 12 to 16 session collapse, and the dashed line marks the KV cache filling at 14c.

Below 12 concurrent sessions, p99 TTFT stays under 2.6 seconds and throughput climbs to a peak of 1.36 req/s. At 14 sessions, p99 jumps to 38.9 seconds in a single step, a 15× increase, and throughput drops 23 percent at the same time.

At 12 sessions, everything fits in KV cache. At 14, filling it forces eviction of another session’s blocks. That evicted session has to re-prefill its full 12,800 tokens of unique context at its next turn, and the cascade collapses latency.

The cliff sits between 12 and 14 sessions, and throughput never recovers past the peak at 12c. For a 2-second p99 SLA, the safe operating point is 6 concurrent sessions.

After the cliff, p50 lies to you

Above the cliff, p50 and p99 live in different regimes. At 20c, p50 is 2.2 seconds, which looks manageable, while p99 is 44.8 seconds, which is not. The distribution is bimodal, because APC creates two populations of requests. Lucky requests hit warm cache entries for their session context, prefill only the latest turn, and finish fast. Unlucky requests arrive after their session’s blocks were evicted, re-prefill the full 12,800 tokens, and take 30 to 50 seconds under load.

The p50 reflects the lucky cohort and the p99 reflects the unlucky one. A single TTFT average is meaningless past the cliff. You have to look at the tail to see the failure.

A few points from the sweep show the whole shape, flat through 12 sessions then the cliff at 14 and the widening p50/p99 gap past it.

KV cache precision moves the knee

Running the same workload with 16-bit KV cache (vLLM defaults to the model’s dtype, bf16 for Qwen3, when --kv-cache-dtype is not set) halves the token capacity, and the knee shifts left in proportion.

The fp16 to fp8 shift is confirmed, with fp16 knees at about 8 and fp8 at about 14, a 1.75× shift for a 2× capacity increase. The slight compression below 2× is expected, since KV management overhead and block table fragmentation consume some of the headroom regardless of precision.

TTFT p99 for fp8 KV against fp16 KV, same workload and GPU. The markers sit at the observed knees, 8 sessions for fp16 and 14 for fp8. The tq4 knee near 23 is a capacity-based estimate, pending the experiment.

Quantization buys concurrency headroom directly. Halving KV precision from fp16 to fp8 nearly doubles how many concurrent sessions fit before the cliff. TurboQuant 4-bit (about 3.8× fewer bytes per element than fp16, partially offset by the lower gpu_memory_utilization it needs for autotuning scratch space) predicts a knee at about 23c, roughly 3× more concurrent sessions than fp16.

The accuracy tradeoff may be small. KV cache quantization at 4-bit typically reports low single-digit perplexity impact, though the exact effect depends on model and task. The knee shifts from about 8 sessions (fp16) to about 14 (fp8) to an estimated 23 (tq4), roughly 3× more sessions before eviction onset, from the same GPU.

Best case, worst case, realistic

To separate the latency budget that is fundamental (prefill compute) from the part that is avoidable (cache misses), we ran two controlled bounds alongside the realistic workload.

In the best case (miss_rate=0.0), every request hits the same cached content. APC holds the full 12,800-token session context, so only about 200 unique tokens need prefilling, which is perfect KV utilization.

The worst case (miss_rate=1.0) gives every request a unique prefix that breaks APC for the user context. The 10K system prompt still hits the cache, but all ~12,800 tokens of per-session turn history are re-prefilled on every request. Every miss lands at peak session depth (turn 12), forcing the maximum re-prefill cost each time.

TTFT p50 for best (fully cached), realistic (natural APC), and worst (always re-prefilled), on a log scale. The band between best and worst is the envelope any real workload lands in.

What the bounds tell us

At a single concurrent session, with zero contention, the three bounds separate cleanly.

On an absolute scale, realistic (388 ms) is much closer to best (57 ms) than to worst (2,500 ms). But realistic is still 7× slower than best. That gap is the cost of prefilling about 12,800 tokens of per-session unique context on each request. APC removes the system prompt cost, but the per-session turn history still has to be computed.

The gap between realistic and worst is about miss depth. In the realistic workload, cache misses happen at any turn. A session evicted at turn 3 re-prefills about 3,000 tokens, while eviction at turn 12 costs about 12,800. The worst case forces every miss to peak session depth, paying the maximum re-prefill on every request. Real traffic produces a distribution of miss depths, which is why realistic latency stays close to best.

The best-case result is the surprising one. With perfectly cached session context, the L4 handles more than 48 concurrent sessions within a 2-second p99 SLA. The 6 session realistic ceiling is the cost of per-session context uniqueness, the turn histories that cannot be shared. It is not a GPU compute limit.

The worst case grows linearly at about 2.1 seconds per additional concurrent session, reaching 102 seconds at 48c. Throughput saturates at 0.32 req/s from 6 sessions onward. The GPU is fully consumed re-prefilling 12,800 tokens per request, and extra concurrency just lengthens the queue.

Why the knee is where it is

The L4 has 24 GB of VRAM, but far less than that is available for KV cache. The memory that actually holds KV cache is roughly half of raw VRAM.

Where the memory goes

vLLM’s gpu_memory_utilization was set to 0.9 for the fp8 and fp16 experiments, reserving about 21.6 GB. After model weights, CUDA graph capture, activation tensors, and block table overhead, about 13 GB remains for KV cache. The TurboQuant experiment used 0.8 (it needs about 2 GB of extra scratch for torch.inductor autotuning at startup), leaving about 10.6 GB.

KV cache per token

Qwen3-4B uses GQA with 36 layers, 8 KV heads, and head_dim 128. The per-token KV cache size depends on precision.

2 (K+V) × 36 layers × 8 KV heads × 128 head_dim × bytes_per_element

FP16/BF16: ... × 2 bytes = 147,456 bytes/token  → ~89K tokens in ~13 GB
FP8:       ... × 1 byte  =  73,728 bytes/token  → ~178K tokens in ~13 GB
TQ4:       ~0.53 B effective (4-bit + quantization metadata)
           =  ~38,700 bytes/token  → ~275K tokens in ~10.6 GB

The capacity arithmetic (with APC)

With APC, the 10,000-token system prompt is stored once and shared. Only the per-session unique context (about 12,800 tokens at peak depth) needs its own blocks.

FP8 KV
Token budget:     ~178K
Shared prefix:     10K (1×)
Available:        ~168K
Per-session:      ~12.8K
Max sessions:   168K / 12.8K ≈ 13      Observed knee: ~14c

FP16 KV
Token budget:      ~89K
Shared prefix:     10K (1×)
Available:         ~79K
Per-session:      ~12.8K
Max sessions:    79K / 12.8K ≈ 6       Observed knee: ~8c

TQ4 KV (estimated)
Token budget:     ~275K (0.8 util)
Shared prefix:     10K (1×)
Available:        ~265K
Per-session:      ~12.8K
Max sessions:   265K / 12.8K ≈ 21      Predicted knee: ~23c

The arithmetic predicts the knees within 1 to 2 sessions of the observed values. The slight overshoot (observed 14 sessions against predicted 13) is because sessions are not all at peak depth at once. Earlier turns have smaller contexts, which buys a few extra sessions before capacity runs out.

In this setup, the concurrency cliff is a memory limit. The binding constraint is how many sessions’ KV caches fit in VRAM at once. The best-case bound supports this. With perfect caching, the same GPU handles more than 48 sessions within 2-second p99. Compute, scheduling, and continuous batching also contribute, but memory capacity sets the ceiling.

How to find the knee for your workload

The knee location depends on three variables. Available KV cache memory is total VRAM minus model weights, CUDA graphs, activations, and fragmentation, typically about half of raw VRAM, and vLLM reports the exact number at startup. Per-session unique context is the total session tokens at peak depth, minus any shared prefix cached by APC. KV precision is the bytes per element, where halving it from fp16 to fp8 to 4-bit roughly doubles token capacity at each step and shifts the knee right.

The estimate is max concurrent sessions ≈ (token capacity − shared prefix) / per-session unique context.

For this setup (Qwen3-4B, L4, 22.8K-token agentic sessions with a 10K shared prefix), the arithmetic predicts about 13 sessions (fp8) and 6 (fp16). The observed knees are about 14 and 8. The arithmetic gives a first-order estimate, and a concurrency sweep gives the precise number. The gap between estimate and observation comes from session depth staggering, block fragmentation, and APC reuse patterns.

Different workloads shift each variable. A single-turn QA workload with 2K tokens per session has a much higher knee. A code review agent with 50K-token inputs has a much lower one. A GPU with more VRAM (A100, H100) raises the budget. The method is the same. Estimate the budget, divide by per-session cost, then verify with a sweep.

What this means for deployment

Know your KV budget before you set your concurrency limit. Below the knee you get the best throughput with stable latency and effective caching. Above it you get worse throughput, worse latency, and a wasted APC investment.

KV quantization is a direct concurrency multiplier. On this L4, switching from fp16 to fp8 KV cache moves the 2-second p99 SLA ceiling from about 4 to 6 (50 percent more sessions) and the eviction knee from about 8 to 14 (75 percent more sessions). The gain is a direct consequence of halving the bytes per KV element. Quantization buys memory, and memory buys concurrency.

Monitor tail latency, not averages. After the cliff, p50 looks manageable while p99 is catastrophic. The bimodal distribution means some users get sub-second responses while others wait 40-plus seconds, and an average-based dashboard hides it until users complain.

If some of your sessions are latency-tolerant background work, run them off the interactive path. They do not need to compete for cache memory, and keeping them off it frees KV budget for the sessions that need low TTFT.

Several caveats temper the numbers. These experiments use synthetic token content, not real code. The workload has a fixed 12-turn structure, while real agent sessions vary widely in depth. Poisson arrivals do not capture bursty agentic traffic, where agents send follow-up requests immediately. p99 at high concurrency is noisy, since with about 200 requests per run it is only the second-worst request. Chunked prefill, not enabled here, could smooth the knee transition. The numbers are specific to a single L4 with Qwen3-4B. Larger models, multi-GPU setups, and different context lengths shift the absolute numbers while the pattern holds.

KV cache behaves like a systems problem, and the concurrency knee is where that meets a specific GPU, a specific model, and a specific workload shape. The math is simple. The discipline is running it before production tells you the hard way.

But there’s some trouble with what standard KV cache benchmarks run on.

LMCache ships with a long-document benchmark for measuring KV cache offloading performance. Run it without a corpus file and it generates documents like this:

warmup_prompts = [
  str(i) + " " + " ".join(["hi"] * args.document_length)
  for i in range(args.num_documents)
]

A 10,000-token document comes out looking a little underwhelming.

0 hi hi hi hi hi hi hi hi hi hi hi hi hi hi hi …

Technically speaking, it is the 10K token count you were looking for, but it doesn’t represent a real 10K token workload.

KV cache systems do not run on token count alone. Compression ratios, activation patterns, transfer sizes, and cache behavior all depend on the shape of the input. Two documents of the same length can be two entirely different workloads.

Unfortunately, much of the current KV cache ecosystem is benchmarked on synthetic inputs that look nothing like the workloads people run.

The benchmark is not representative

The default benchmark document contains a numeric identifier and the token “hi” repeated thousands of times.

Transformers do not produce identical activations for repeated tokens. Positional encoding, attention mixing, and residual connections keep every position distinct. To an LLM, distinct and varied are not the same thing. Repeating a single token produces far more regular activation patterns than diverse text does.

Compression improves, transfer sizes shrink, and cache behavior becomes easier to predict with non-varied workloads. The benchmark is measuring something, but it is not measuring a realistic production workload.

Benchmarking KV cache offloading with “hi hi hi” is like benchmarking a database with SELECT 1. The numbers come back fast, but they do not tell you much about real workloads.

The difference shows up immediately

We compared the default benchmark document against a realistic medical document using Qwen3-8B-FP8. Both ran about 10,000 tokens. The synthetic one carried two unique tokens, a token diversity of 0.02 percent. The medical one carried 1,329, or 13.3 percent. The token count is the same. Everything else is different.

Token diversity affects activation patterns. Activation patterns affect tensor value distributions. Tensor distributions affect compression ratios and transfer sizes. A benchmark built from highly repetitive inputs can diverge sharply from one built on realistic text.

Compression behaves differently too

We first ran into this while building a Valkey-backed KV cache connector. We followed the LMCache tutorials and used dummy weights, and compression ratios looked excellent. Then we switched to Qwen3-8B-FP8 with real trained weights, and the results were night and day.

Real model weights produce tensor values that behave more like high-entropy floating-point data. General-purpose compression still helps, but the gains are smaller than they appear with dummy weights. Repetitive inputs create more structured activation patterns that compress more effectively, making the benchmark results appear better than they actually are.

So we built a more realistic corpus

To see how KV cache systems behave under representative inputs, we built a corpus of 30 long-form documents across medical and legal domains. We wanted documents that resemble the structure, formatting, vocabulary, and variability that real systems process.

Medical documents averaged 14.2 percent token diversity. Legal documents averaged 9.4 percent. Both are hundreds of times more diverse than the synthetic baseline at 0.02 percent.

The corpus holds 300,000 tokens and was generated with Qwen3-8B-FP8. The corpus and generation scripts are open source for those interested.

Benchmarking the workload you actually have

While token count is easy to generate, it can also be the least informative. Match the diversity, structure, and vocabulary of the text your system serves, and the compression ratios and transfer sizes start to represent values you can trust.

The input comes first. Before ranking cache connectors, compression schemes, or storage backends, run them on inputs that look like your traffic. Compare them on “hi hi hi” and you are ranking them on a workload nobody runs.

Before you evaluate your next KV cache offloading system, be sure to ask “Are the benchmark documents representative of the workload I actually run?”

But it’s still the same type of reuse we’ve always had. Shared prefixes are reusable. Shared content that is not a prefix is not. That rule sets the biggest limit on what today’s inference infrastructure can reuse.

Most of the recent work in KV caching has focused on finding prefixes more efficiently. Hash chains, radix trees, and automatic discovery all improve the mechanics of reuse. But they don’t change what can be reused. The workloads where it succeeds, and where it falls short, show why.

When prefix caching is enough

Reusing the KV cache only pays off when requests share computation, so the question is how much of a real workload today’s infrastructure can reuse.

For many agentic workloads, the answer is often “enough”. Stable system prompts, tool definitions, and conversation scaffolding create long shared prefixes, and inference engines are good at finding and reusing them.

Plenty of workloads share more than just prefixes, though. RAG pipelines are where it breaks down. The system prompt stays fixed while the retrieved documents change from request to request. Two requests might carry the same five documents in a different order. The meaning is almost identical. The token sequence is not, and the cache matches on the token sequence. Same content, different positions, no reuse.

Why prefix caching remains prefix-bound

vLLM’s Automatic Prefix Caching uses content hashing to remove the need for explicit prefix tracking. The KV cache is divided into fixed-size blocks, and each block gets a SHA-256 hash. The hash for block N folds in the hash of every preceding block along with the content of the current block. Chained together, every block fingerprints not only its own content but the entire token history before it.

When a request arrives, vLLM hashes each block-sized chunk of input and checks whether a matching block already exists. Matching blocks reuse previously computed KV cache. Missing blocks trigger fresh computation. Lookup is effectively constant-time, eviction is straightforward, and fixed-size blocks map cleanly onto GPU memory.

But because each block hash depends on every block before it, one divergence changes every hash that follows. Take two requests that share the first 3,000 tokens and split at token 3,001. The block holding token 3,001 hashes differently, and so does every block after it. Reuse stops at the point of divergence. The system discovers shared prefixes on its own, and it can’t discover shared content that appears once requests have diverged.

Reuse happens only at block boundaries. If two requests share 1,000 tokens and a block holds 16, vLLM reuses 62 whole blocks, or 992 tokens, and recomputes the remaining 8. For long prefixes that waste is negligible. For short or irregular shared segments it’s more substantial.

There is no matching inside a block, either. Two blocks that share 15 of their 16 tokens still hash to completely different values, so reuse is all-or-nothing at the block level. These are reasonable tradeoffs that keep the implementation simple and fast, but they still leave you with the same limitation: reuse follows exact prefix structure.

A different cache structure might help. SGLang takes that route. Instead of hashing fixed-size blocks, it keeps cached state in a radix tree indexed by token sequences. When a request arrives, the runtime walks the tree and finds the longest matching cached prefix, and matches can fall on arbitrary token boundaries rather than fixed block ones. That helps workloads with variable-length turns, branching conversations, and irregular prefix lengths.

The radix tree still searches for the longest shared prefix, though. Once two requests diverge, the content they share later in the sequence stays out of reach. SGLang improves how prefixes are discovered, but it does not extend reuse past prefixes.

Beyond prefixes

Prefix caching tells us that KV reuse clearly works. The more open question now is how much reuse survives divergence. Today’s runtimes are tuned for shared prefixes. The next generation goes after shared segments, cache repair, and the reuse that prefix matching cannot reach.

So much of the current research now focuses on cache repair and segment-level reuse. The goal has shifted from proving that KV reuse is valuable to recovering the work that today’s prefix-based systems still leave behind.

Prefill is compute-heavy, with an order of magnitude more arithmetic intensity than decode. Decode is sensitive to memory bandwidth and to latency. Prefill wants high-FLOPS accelerators. Decode wants large, fast memory. Put both on the same GPU and you tune it for one profile while it absorbs the other, and under load prefill interferes with decode. Neither phase gets the hardware it would choose.

The cache becomes the connection

Disaggregation separates these two phases. Prefill nodes do prefill, decode nodes do decode, each on hardware suited to its bottleneck.

Separation removes the interference, but it opens a gap. Inside one accelerator, prefill flows straight into decode and the intermediate state never leaves the chip. Pull the two onto different machines and that state, the KV cache, has to be handed across. Prefill produces it, decode consumes it, and disaggregation turns it into the object that connects them.

On a single node, the KV cache is largely an implementation detail the inference engine manages for you. But once prefill and decode are separate systems, the cache is the connection between them, and every request depends on getting it from one to the other.

What the split asks of the cache

Once the cache has to travel between machines, it takes on the requirements of any object moving through a distributed system.

The cache has to move from prefill nodes to decode nodes, which makes transfer latency, serialization format, and network bandwidth first-order concerns. It has to land in the right place, so deciding which decode node receives which cache turns routing into a scheduling problem. It has to expire, which raises the question of who evicts an entry and when, work the engine handled in a colocated system and that now needs coordination. And it has to live somewhere across GPU memory, host memory, NVMe, and remote storage, each tier with its own latency and capacity tradeoffs.

These are distributed systems problems that present themselves whenever you separate storage and compute. What was colocated becomes independently addressable, connected by a transfer layer. The techniques for solving them, placement, routing, eviction, and tiered storage, are well understood. What is new is applying them to the KV cache inside inference serving.

From implementation detail to system design

The major inference stacks are already built around this split. NVIDIA Dynamo describes disaggregated inference as a prefill engine that computes the prefill phase and generates KV cache, hands that cache to a decode engine, and lets the decode engine run the decode phase. AWS is building the split into its infrastructure, with Trainium for compute-heavy prefill, and the Cerebras partnership likely points the same way, since wafer-scale SRAM suits memory-bound decode.

In each of these, the KV cache is the object the tiers hand between them. On a single node it was an optimization you could run, skip, or tune, and the architecture did not care. Disaggregation moves those same problems up a level, from implementation details the engine used to hide to system design concerns the architecture has to own. The cache now has to be transferred, routed, stored, and expired across machines, and the whole system is built around getting it from prefill to decode.

What began as transient state inside a single request becomes the handoff between two systems. Once we have that handoff, cache management becomes a first-class primitive of the architecture.

The benchmarks do not rescue it.

We’ve seen teams leave it at that. KV cache is necessary inside a single forward pass, so you keep it for the life of the request and move on. Keeping it alive beyond that, reused across requests, starts to sound like a serving-layer luxury. You picture the memory it would pin, the eviction logic, the extra moving parts in a stack that is already hard enough to operate. Set that against a few percent of latency and the trade does not look worth making.

Understandably so. Run the numbers on a single request and long-lived KV caching underwhelms. We ran them, and at first it was a very unflattering story. But a single request is the wrong unit to judge this on, and once you measure at production load the economics turn. 

The single-request savings are bounded

In one of our Qwen3-30B-A3B runs, a 1K input / 512 output request came in around 135 ms TTFT and about 2.5 seconds of total request latency. TTFT carries scheduling and queueing overhead on top of raw prefill compute, so call the prefill portion roughly 100 ms. Erase it completely, the most a perfect cache hit can do, and you save about 4 percent of the request. Treat that as the ceiling, not the everyday case.

As context grows, so does prefill’s share of the total request latency. At 16K input / 512 output, TTFT was about 769 ms of 3,200 ms total, which puts it near 24 percent. That is a real step up from the 1K case. The input/output ratio affects prefill more than the context length does. KV cache earns the most when a request carries a large input and returns a small output, because prefill is then a bigger share of the bill. In scenarios where you have short input and long output, decode takes over while the cache has little room to help.

On its own, a 4 or 24 percent share looks modest. Latency is measured at a target throughput, GPU capacity is scarce, and as throughput climbs, utilization, queueing, and pipeline stalls increase the cost of redoing prefill. So in production, prefill becomes a capacity and tail-latency problem once thousands of requests compete for the same accelerators.

So the skepticism is fair, for the single request. If the only question is whether one cache hit meaningfully cuts one request’s latency, the answer is often no, and it depends on the input/output ratio and how much of the request prefill actually owns. At that level, KV caching is not an automatic win.

Large-input, small-output workloads are getting more common, not less. Agentic workloads are multi-turn by nature. Context grows as chat history, tool-call results, and retrieval chunks pile up, so each new turn carries a larger input against a small output. Exactly the type of workload where prefill dominates and where reusing the cache has the most to give.

For application teams, reusing the cache shows up as lower TTFT, tighter p95, and lower cost per request. For the platform teams running the GPUs, it shows up as higher utilization and more capacity per dollar. 

Expensive to hold, hard to reuse

That said, the KV cache is not free to keep. Hold it in GPU memory and it competes with active inference for the scarcest space you have. Move it to host memory and it is cheaper but still bounded, with DRAM prices trending the wrong way. Push it out to remote memory or storage and you take on transfer latency, placement problems, and more operational surface. KV caching is a bet. You are spending scarce memory on the wager that future requests reuse the work you are holding.

But that’s only half the problem. Even when you are willing to pay for the memory, the reuse you get back is limited. The production-friendly option today is prefix caching: if a later request begins with the exact same prefix, the engine reuses the KV cache already computed for it. The rule is strict, exact prefix match or nothing. Plenty of real workloads share meaning without sharing a prefix. Reordered retrieval chunks, varying tool results, and shifting user context carry the same semantic content in different positions, and none of it counts as a hit, so hit rates suffer. 

It might seem like KV caching has a lot going against it. The single-node latency win is bounded. The memory cost is high. The reuse model is narrow. Evaluate it as an isolated optimization on one node and the honest question is whether the complexity earns its place. In isolation, often it does not. But isolation is the wrong frame because inference is not the system it was when those objections were formed. Each of them was measured against a single node running prefill and decode together, holding a cache that was large and expensive to keep. Two things have shifted since. The first is structural, in where prefill and decode run. The second is economic, in what the cache costs to hold and to move. Each one undercuts a different piece of the case against. 

Inference is becoming a distributed systems problem

Prefill and decode are not the same kind of work. Prefill is compute-heavy. Decode is sensitive to memory bandwidth and to latency. Put both on the same accelerator and you force a compromise on one to serve the other. Split them, and you create a clean boundary between two workloads that want different things. The KV cache, however, has to cross that boundary.

When prefill and decode live on different hardware, the KV cache becomes a first-class distributed systems primitive, something you transfer, place, and manage a lifecycle for. NVIDIA Dynamo and the disaggregated stacks coming out of AWS and Cerebras are building the split into the infrastructure itself, which is what forces developers to think about how the KV cache moves, where it lives, and how long it stays alive.

The economics are starting to move

The second shift is economic. The KV cache itself is getting more efficient to store and to move. A surprising amount of recent model progress is really KV cache innovation, and the last 18 months have been striking. 

DeepSeek-V2 and V3 introduced Multi-head Latent Attention (MLA). MLA compresses keys and values into a shared low-rank latent vector before anything gets cached. For V3, that takes the per-token cache from roughly 16,384 scalar dimensions under standard multi-head attention down to 576, a 512-dimensional latent plus 64 dimensions for decoupled RoPE. Against an MHA baseline that is about a 28x reduction. Against the GQA baseline most modern models already use, it is smaller, roughly 4 to 8x depending on group size, and MLA gets there while holding MHA-level quality, which GQA gives up.

Qwen 3.5 goes a different way with a Gated DeltaNet hybrid. It replaces 75 percent of its attention layers with Gated DeltaNet linear attention, layers that hold a fixed-size state matrix, 128 by 128 per head, and update it incrementally with each token. The state does not grow with sequence length. Only the remaining 25 percent, full softmax attention with GQA, still needs a traditional KV cache. At long contexts, where the KV cache usually dominates memory, this removes most of the growth. The payoff scales with context: substantial at 256K tokens, modest at 1K, where a fixed-size state costs about what a small KV cache would anyway.

TurboQuant and PolarQuant, from Google at ICLR 2026, take yet another angle. Instead of changing the attention mechanism, they quantize the KV cache itself to 3 or 4 bits per coordinate with no measurable accuracy loss on standard benchmarks. PolarQuant rotates vectors with a random orthogonal matrix so the coordinates follow a known distribution, then applies an optimal Lloyd-Max scalar quantizer, and QJL adds a 1-bit residual correction. At 4 bits the paper reports up to 8x faster attention on an H100. At 3 bits, roughly 6x memory reduction.

The exact numbers depend on baselines and configurations, but the direction is obvious. MLA shrinks the cache dramatically. Hybrid architectures such as Qwen’s Gated DeltaNet reduce cache growth across much of the network. Quantization approaches such as TurboQuant reduce memory requirements further without changing the model architecture. Different tradeoffs, same trend: the object is getting smaller. 

Memory cost is the usual objection to KV caching, but this recent work almost makes it moot. Shrink the cache by 6x to an order of magnitude and the economics look very different. More entries fit in the same budget. Transfers from remote memory, SSD, or another node get faster. There’s a misconception that the cache has to become trivially small. But it only has to get small enough that the economics cross over for the workloads people run in production.

The storage hierarchy is changing as well. The previous thought was if the KV cache is not in GPU memory, it is too slow to matter. That is getting harder to say. Fast interconnects and local NVMe continue to improve. Now the question is whether moving or loading the cache can free the decode GPU from repeated prefill work and keep it pointed at the latency-sensitive part of the pipeline. If a storage-backed cache lowers prefill pressure and keeps accelerator capacity on decode, it can pay off quickly.

The workload is the main success driver. For a given model and context length, the system weighs the time to recompute prefill against the time to move the cache over the network, the time to read it from SSD, the cost of reserving the memory or storage, and the odds the cache gets reused at all. When transfer or load time comes in well under recompute time and reuse is likely enough, the cache earns its place. When it does not, the cache is a cost with no return.

Prefix caching under load

We saw this behavior in an experiment we recently ran. We used Qwen3-1.7B on an L40S, a 10K-token prompt, and the number of shared prefix tokens varied from 0 to 10K across several concurrency levels. As the shared prefix grows, the vLLM prefix cache hit ratio climbs from 0 to 1. Throughput and request latency were monitored at each concurrency level.

Prefix cache hit ratio grows linearly with shared prefix length, from 0 to 10K tokens against the fixed 10K-token prompt.

Throughput rises as the shared prefix grows and is sharpest at high concurrency, where skipping repeated prefill lets the system sustain more requests per second.

Mean request latency falls as reuse increases, with the high-concurrency settings improving most. 

At low concurrency, a higher hit ratio helps, but only a little. At high concurrency, the same increase results in a much larger system effect. Requests per second climb sharply as more of the prompt comes from cache, and mean latency decreases along with it, dropping fastest at the higher concurrency levels. That is the behavior one would expect if KV caching is a systems optimization rather than a single-request latency trick.

The experiment is a small model on a single node, so it does not prove the disaggregated-architecture argument on its own. But it does verify that prefix cache hits remove prefill work from the serving path, and the system-level benefit grows with concurrency. The disaggregation thesis is that this gets stronger when prefill and decode run on separate hardware and the KV cache moves between them as a first-class object.

Prefix caching already works for the right workloads

Prefix caching generally has a narrow sharing mode. For agentic workflows it fits more naturally than you might expect, though the reason it fits changes by category of context.

System prompts are the easy case. They are stable across requests, they sit at the front of the prompt, and are a textbook prefix hit. An agent making a series of tool calls against the same backend reuses the same 2K to 8K token system prompt on every request. A multi-turn conversation with a fixed system prompt reuses the whole instruction block. A code-generation agent with stable repository context reuses the project description and file summaries. For this category, cross-request caching is straightforward.

Other kinds of context ask for more care. Chat history grows and shifts from turn to turn. Tool-call exemplars get reordered or swapped. Retrieval chunks change with every RAG query. These often share material across requests without sharing an exact prefix, so the effectiveness of the cache comes down to how much of the context is positionally stable (which prefix caching handles), versus variable (which needs something like CacheBlend to unlock). 

Research for a better solution

For messier patterns, like RAG with retrieval chunks that vary by query or tool results that differ between calls, two requests can share a great deal of material without sharing the exact same prefix, and classic prefix caching returns a miss in those cases even when most of the computation could have been reused.

CacheBlend is one of the research directions in this area. It is exploring the idea of cache repair, which takes a semantically similar cached entry to what the current request needs, and selectively recomputes only the parts that differ. If repair is cheap enough, individual caches become reusable across more requests and hit rates rise without spending more memory.

This is still in open research, it’s not solved yet. No major inference framework ships chunk-level KV reuse today. The selective recomputation carries its own latency, quality preservation depends on the workload, and the methodology for measuring these tradeoffs is still maturing. But the direction is promising. More flexible cached entries raise the effective hit rate inside the same memory budget.

Prefix caching answers “does caching work?” for a growing number of workloads. The open question is how much of the rest can be brought into the cacheable regime, and cache repair is where we are working that out.

From per-request state to systems primitive

If you evaluate KV caching as an isolated optimization on a single node, it doesn’t make much sense. The memory cost is high and prefix-based reuse is limited.

But the architecture underneath is changing. Disaggregated prefill and decode create the right interface. Better attention mechanisms shrink the object you have to store and move. Faster networks and SSDs reduce transfer costs. Cache repair could push reuse beyond strict prefixes. Scarce GPU capacity makes repeated prefill work harder to justify.

Together, these shifts are turning the KV cache from a temporary intermediate state into an inference systems primitive.

Don’t have five minutes? This infographic summarizes the key takeaways from this blog.

 The FIFA World Cup 2026 is not just a tournament for Momento. It is a live fire test of what real-time data at global scale actually means.

In stadiums and on sofas, hundreds of millions of fans will see goals, cards, and heartbreak. Behind the scenes, three of Momento’s largest customers will be doing something just as intense: pushing a real-time data platform to the limit across live origination, content protection, and AI-powered personalization.

Beneath these vastly different workloads lies the same fundamental principle that decisions must be made instantly, at global scale, with zero excuses.

What Do We Mean by “Momento is a Real-Time Data Platform”?

“Data platform” is one of those phrases that can mean anything from a gigantic static data warehouse to a firehose of events in flight. When we say Momento is a real-time data platform, we mean something specific as we combine:

• A sub-millisecond in-memory data engine: This is the RAM-cache based data plane that serves reads and writes in less than a millisecond, even under massive load.
• An intelligent control plane: This layer automatically handles sharding, scaling, partitioning, and hot-key management, so app teams do not have to be distributed systems experts

In practice, customers can treat Momento like a simple API to store and retrieve state in real time: segments and manifests, concurrency counters, user events, AI embeddings, and more. They describe their data model and policy. Momento makes it fast, durable, and observable.

The FIFA World Cup is a perfect way to show what that actually looks like when the stakes are highest. Think of it as a hat-trick of real-time use cases.

1/ A Live Origin That Just Does Not Flinch

Who: A large UK broadcaster holding FIFA rights Problem: Their live origin service (AWS Elemental MediaStore) was deprecated before the World Cup. They needed the same low latency, failover behavior and observability, at World Cup scale, without rewriting their encoder, packager, video player or CDN configurations.

How Momento helps:Momento Media Storage is their new live origin. It gives them:

• Predictable, low latency: Consistent latency for reads and writes at the live edge.
• Granular TTL control: TTL settings on manifests/segments to preserve automatic failover capabilities.
• Per-Service Limits and Metrics: Observability and metrics are built in so high traffic events don’t impact the rest of their 24/7 channels.

For viewers, nothing “looks” different. For their ops teams, the origin is now actively developed, supported, faster, and ready for tens of millions of concurrent fans.

🔗 Deep dive on How a major UK broadcaster moved its World Cup live origin to Momento ↗

2/ Content Protection Through Concurrency Tracking

Who: A major US broadcaster holding FIFA rights Problem: Pirates leverage stolen accounts and run illegal restreaming operations on top of the broadcaster’s own CDN. Traditional DRM and short-lived tokens verify a device can decrypt, but are completely blind to whether an account is behaving like a bot farm. Content rights holders are concerned about piracy and are asking broadcasters to implement server-side control over account behavior to shut down the illegal streams at the source.

**How Momento helps:**The broadcaster runs a server-side concurrency service backed by Momento. For each stream, a lightweight verification loop executes three steps:

• Receive the heartbeat: The player calls a Momento-powered service with identifiers for the subscriber’s account, their device, the content they want to access, their geography, etc.
• Enforce stream limits: Verify limits on concurrent streams per account and per event.
• Deliver instant decisions: Make allow/deny decisions in single-digit milliseconds, inline with playback.

If one account suddenly powers hundreds of devices on the same game, the system can automatically shut it down, protecting revenue, CDN bills, and QoE for legitimate fans.

🔗 Details on the architecture and anti-leeching approach: Stop CDN Leeching with Concurrency Tracking ↗

3/ AI-Powered Personalized Feeds for a Sports App

Who: A US-based popular Sports content Super App Problem: Just ahead of the World Cup, this content provider wanted to relaunch their app, with a brand new User Experience. Beyond the traditional “click to watch” from their editorial content, they needed a real-time, personalized feed that mixes editorial pieces, highlights, YouTube clips, and content from popular social networks, tuned to each fan’s behavior as it happens. Think of it as a TikTok “For You Page”, for Sports.

**How Momento helps:**Momento serves as the real-time event collection and embedding layer, making real-time machine learning models visible to the app users at massive scale:

• Dynamic content ingestion: New content from newsrooms, creators, social networks, and athletes is ingested and turned into AI embeddings.
• Real-time signal streaming: User signals including emoji reactions, comments, watch time, and scroll depth stream into Momento in real time.
• Sub-millisecond personalization: Recommendation services query Momento’s sub-millisecond data plane to match fresh content to each fan’s evolving interests.

The result: A feed that feels instantly relevant and keeps improving as fans interact with their personalized feed, at the scale of millions of concurrent users.

🔗 Video explainer on real-time embeddings with Momento: Momento AI embeddings & event collection explainer ↗

Let’s Watch Football, Not Infrastructure

Three very different workloads, all powered by the same underlying platform: a low latency data plane with an intelligent control layer that simplifies real-time data usage for application teams.

When the first match kicks off, the fans will be watching football, not infrastructure. But inside control rooms, NOCs, and product teams, our customers will know. They will see cleaner dashboards, stronger protections, and faster feedback loops. They will see a real-time data platform doing exactly what it was built to do.

We are excited to be part of their World Cup story. And once the final whistle blows, these capabilities will not disappear. They will become the new baseline for what fans expect from live sports, everywhere.