Reactive infrastructure describes this common pattern of connecting incoming observations to changes in the viewer experience. Specific capabilities like feed personalization, quality of experience monitoring, and anti-piracy detection each touch different parts of a live media system, but all of them transform some type of client activity into timely decisions.

The shared challenge begins with ingesting dynamic CDN-scale traffic without a CDN. Televisions, browsers, and mobile devices send observations directly to a collection endpoint. Audience surges become ingestion surges, and every accepted event creates work beyond that endpoint. And collecting the data is only the beginning! That raw data must become useful and be delivered for use elsewhere within seconds.

A four-stage pattern organizes that work: ingestion → processing → storage → decisioning. The stages adapt to each use case through the signals they collect, how they group data, the stores they use, and the decisions they drive. Their distinct responsibilities help identify the most important design choices at each stage.

How the four stages close the loop

Reactive architecture forms a closed loop from an observation on the client, through the data pipeline, and back to a change in the client experience. That change then produces new observations, and the cycle begins again.

The four stages of this architecture are:

1. Ingestion accepts client events and shapes the incoming traffic.
2. Processing organizes event fragments into an orderly, validated stream.
3. Storage makes that data available for analysis in forms suited to its consumers.
4. Decisioning connects the resulting information to product behavior.

A common four-stage pattern connects client activity to decisions that affect the media experience.

The cloud components for each stage focus on those specific responsibilities. An internet-facing gateway receives beacons from clients. Valkey (the community-backed fork of Redis) provides a shared collecting pool behind it, with lightweight functions and connectors preparing data for downstream stores. Decision consumers use relevant information to influence the experience through business logic, an edge function, or an operator action.

Note that the diagram’s Gateway or CDN path represents the outbound, or egress, direction that delivers the experience to viewers. This complements the inbound, or ingress, path through which client observations enter the architecture.

Data flows through this architecture from ingestion towards decisioning, but the design process should actually work backwards from the decision it drives. What signals does that decision require? How quickly must it affect the experience? How much data loss is tolerable? Those answers shape the choices at every stage.

Ingestion shields downstream services from client traffic

The ingestion stage is the interface between your infrastructure and millions of viewers. Its gateway must handle connections from devices across the open internet, absorb sharp changes in traffic, and prevent that volatility from overwhelming internal services.

HTTP beacons are the delivery vehicle for signals that eventually drive decisions about what the user experiences. Beacon size and frequency directly affect load on the gateway. Choose the signals needed for the decision to keep each payload small. Set the beacon period according to how quickly the decision needs new observations, so the system receives useful updates without collecting more than it needs.

The gateway controls how that traffic safely reaches the backend. It holds incoming events briefly and forwards them together, converting a firehose of individual requests into more manageable micro-batches. By reshaping the flow of data, the gateway protects downstream infrastructure and reduces the need for defensive overprovisioning.

The brief delay for micro-batching is a negligible component of end-to-end latency when the buffering period is small relative to the decision’s latency budget, but it slightly increases the opportunity for data loss. Buffered events have just one copy, so if a gateway node fails before forwarding them, they will be lost. Ultimately, batching policy trades a slight increase in exposure to data loss for significant gains in overall reliability and lower infrastructure costs.

Momento Valkey Router contributes connection pooling, authentication, throttling, and shard-aware routing at this boundary. These mechanisms support the gateway’s shielding role alongside the architecture’s beacon batching. Reusing backend connections lets the collecting tier serve a much larger device population without maintaining a separate upstream connection for every client.

Processing organizes events and routes them to consumers

Batches arrive through different gateway nodes, leaving fragments of related activity in separate buffers. Processing brings those fragments into common, replicated collecting pool powered by Valkey and prepares them for downstream use. Lightweight functions and asynchronous connectors handle transformation and routing.

A stream identity is defined by some property of a beacon, like a session id or a geographic region. The stream key directs related events towards the appropriate shard, where the fragments are assembled into an orderly stream. This organizing phase presents downstream consumers with a coherent sequence of related activity, ready for storage and analysis.

Keep preparation lightweight at this stage. Validate records before they reach live consumers, make small adaptations to their shape, and route them to suitable sinks. Expensive joins, exploratory queries, and complex analysis belong with the downstream systems chosen for that work. Keep a raw archive independent of live validation so malformed records remain available for inspection or replay.

Momento’s lightweight WASM functions perform these short validation and transformation operations in batches. Its asynchronous sink connector drains Valkey and flushes complete files to destinations such as S3, separating that transfer work from client requests.

Storage serves each consumer

Once events are organized, the next question is how they will be used. Storage serves those consumers’ access patterns, rather than asking one database to handle every question. Query shape, required response time, and retention can differ substantially even when consumers use the same underlying event flow.

QoE monitoring illustrates this perfectly. Automated alarms that should trigger within seconds use rolling counters in Valkey. Meanwhile, an operator investigating an incident needs an analytical store such as ClickHouse to perform ad hoc queries in less than a minute. On the side, events are funneled to an S3 archive in Parquet for retrospective analysis, replay, and auditing. A bounded counter lookup, an exploratory query, and a historical scan each place wildly different demands on storage.

Personalization changes the query shape again. Similarity retrieval against user and content representations calls for vector-oriented storage. The architecture keeps the same storage responsibility while choosing a different representation and retrieval mechanism. The particular stores follow the consumer’s requirements.

Valkey serves both collection and fast operational storage roles. Those roles can share a tier, while performing different work. The asynchronous connector introduced in processing gives archival storage its own transfer path, while live consumers use views prepared for their needs.

The time to read a counter or run a query is only one part of the time from client activity to an applied outcome. The beacon period, buffering, preparation, and decision timing also contribute. Selecting a fast store cannot compensate for a pipeline that delivers the needed information too late.

Decisioning applies results without blocking the experience

Decisioning connects prepared information to actions in the media system. A decision consumer, such as a policy engine or operator, evaluates that information and chooses an action. Product business logic or an edge function applies the result to rankings, delivery paths, session policies, or other application behaviors.

Keep the analytical workload off the live serving critical path. Requiring a complex query for each viewer request ties the product’s response time and availability to that query system. Compute decisions outside the viewer-request path and make the results available to the request-handling logic. That logic applies the result without waiting for a fresh analytical query on each request. Choose how often to update the decision according to the outcome it must affect.

Anti-piracy detection illustrates the timing relationship. Policy analysis runs between segment requests, with a result available for subsequent enforcement. That provides decision time without adding the same analytical work to each segment response. Existing access authorization remains part of the delivery system.

Personalization offers another reason for deliberate timing. Continuously updated interest data need not produce a different ranking on every refresh. Keep the published ranking stable while the underlying analysis continues. In both cases, the decision update frequency reflects what the experience needs.

Once the product applies a decision, subsequent client observations reveal its effects. Those observations return through ingestion, starting the cycle again.

Adapt the architecture to the decision

The four stages in this reactive architecture work together to convert CDN-scale observations into user-facing changes within seconds.

Gateway batching reduces request overhead before internal services receive the data. A collection pool assembles related activity into orderly streams sent to an appropriate datastore for each downstream consumer. Analytics engines then turn that information into decisions, which are applied through application business logic to impact the user experience.

Feed personalization, QoE monitoring, anti-piracy detection, and other use cases can all leverage the same architecture with slight adaptation. Remember to start with the outcome, then work backwards through the architecture to apply decisions and propagate constraints. The loop becomes reactive when the data changes what the user experiences.

Building systems that handle CDN-scale traffic without a CDN is challenging, even when you’re an expert. Momento engineers understand how systems fail in mission-critical situations, and designed the Momento platform to handle millions of requests per second. It has powered live experiences during the Football World Cup, the Super Bowl, and other global media events across leading providers like FOX, Paramount, and Warner Bros. Discovery.

Chat with us if you’d like to explore making your product more reactive!

A beacon carries device telemetry with selected signals for downstream analysis. The event data it carries might support quality-of-experience monitoring, anti-piracy detection or personalization. Selecting those signals matters to ingestion as well as analysis. Every added field increases the payload, and those extra bytes multiply across the beacon traffic that the system must receive and hold.

Beacon ingestion therefore needs a protective boundary between device traffic and internal processing. In this architecture, Momento Valkey Router receives beacons through a gateway tier, manages device connections and batches data for forwarding. Shared, open-source Valkey collects related events and buffers them until downstream services can process them. Together, the gateway and Valkey turn an unpredictable beacon firehose into organized data that internal services can consume at a manageable rate.

Valkey Router and shared Valkey collection form a protective ingestion boundary between device traffic and downstream services. Kafka, ClickHouse and S3 illustrate downstream choices.

Protect internal services from device traffic

The gateway separates the audience’s connections from the connections that internal infrastructure must support. Without that boundary, a growing device population can increase database connection pressure as well as beacon volume. These are distinct demands. An internal service might have enough processing capacity for the events while struggling to manage the connections delivering them.

Valkey Router handles device connections and reuses a pool of connections to Valkey. Pooling prevents each device connection from becoming a separate Valkey connection. The gateway can receive traffic from many devices while controlling the number of database connections. Momento supplies this gateway layer alongside Valkey’s open-source storage foundation, placing device-facing capabilities ahead of shared event collection.

The gateway also controls access to that collection. Authentication and access checks determine whether incoming traffic is allowed to reach internal infrastructure. HTTPS and TLS termination protect device-to-gateway communication, and custom certificates let the endpoint use the certificates required by the deployment. These capabilities put connection and access handling at the boundary where device traffic enters the system.

Throttling limits the forwarding rate when incoming traffic would otherwise overwhelm the next stage. The gateway controls ingestion before excess traffic reaches the database and consumers. Connection pooling and throttling address different parts of the same protection. One controls database connections, while the other controls the rate of data reaching internal services.

Share forwarding overhead across small beacons

Managing connections still leaves a large number of small payloads to forward. Sending every beacon separately repeats communication and request-handling overhead. When that repeated overhead consumes a substantial part of ingestion capacity, the system spends effort moving small pieces of telemetry instead of collecting their event data efficiently.

Valkey Router batches compatible beacons locally before forwarding them. Several beacons share the overhead of a batch, reducing the repeated forwarding cost per beacon. Their event data remains available for collection. Batching changes how data is transferred without requiring the application to remove the observations it needs for analysis.

Batching introduces a short wait at the gateway because data must accumulate before it can be sent together. The gateway bounds that wait and the amount of data it gathers. A count trigger sends a batch when enough beacons have accumulated. A timer sends the accumulated data when the waiting interval ends. During busy periods, the count trigger can fill batches quickly. During quieter periods, the timer lets small batches progress instead of leaving beacons waiting for traffic that may not arrive soon.

A size bound can also limit how much data a batch contains. This matters because beacon sizes can vary with the selected signals. A batch with an acceptable beacon count could still contain too many bytes if those beacons have larger payloads. Count, time and size bounds serve complementary purposes. They limit local accumulation while balancing forwarding efficiency against the delay introduced by waiting.

Each gateway assembles its own batches from the traffic it receives. This keeps batching close to device ingestion, where small payloads first accumulate. The local batch holds data briefly to improve transfer efficiency. Shared Valkey collection serves the longer-lived need to hold events for downstream processing. Efficient batches from separate gateways must also reach the right collection point, because each gateway may receive only part of a partition’s activity.

Bring related events to the same collection point

A gateway tier distributes ingestion across multiple instances, but related beacons can reach different gateways. A consumer processing an application-defined partition needs its related events together, even when they come from different devices. Leaving those events fragmented across gateway-local collections would push the task of gathering them onto every downstream consumer.

The architecture collates related events by deriving a grouping key deterministically from beacon properties. Every gateway applies the same grouping rule. Beacons with the same grouping property produce the same grouping key, giving their related activity a consistent logical address. The gateways use that address to route event data to the same Valkey shard. The shared collection brings related events together in a coherent data stream for downstream consumption.

Consider two devices sending beacons assigned to partition S42. One beacon reaches one Valkey Router instance and the other reaches another. Both carry the partition identifier, and both gateways derive the grouping key partition:S42 from it. Each gateway uses that logical address to route the related event data to the selected Valkey shard. The partition’s events meet in shared collection even though different devices and gateways handled the beacons.

The selected shard is a portion of shared Valkey collection that holds multiple keyed groups. The grouping key keeps the partition’s events associated alongside data for other partitions. A consumer can process the collected partition activity without gathering fragments from individual gateways. This separates the gateway that receives a beacon from the collection where its related events belong, allowing the ingestion tier to distribute device traffic while preserving the relationships downstream analysis needs.

Each gateway buffers events locally for efficient forwarding in batches. Matching partition keys bring related events from different gateways to the same Valkey collection point. The arrows follow the three marked S42 events; other partitions and shard placements are illustrative.

Give events room to wait for consumers

Organized events can still arrive faster than downstream services process them. A traffic burst increases ingestion immediately, while a consumer may continue at its previous processing rate. Consumer slowdowns can create the same imbalance even when incoming traffic remains steady. Shared buffering gives the architecture somewhere to hold that difference.

Valkey holds collected event data while consumers process it. During a burst, the buffer grows as ingestion outpaces consumption. When traffic subsides or consumer capacity increases, consumers can catch up and the backlog can shrink. The shared buffer lets the gateway continue receiving traffic through a temporary mismatch in rates, within the capacity available to hold the data.

That capacity is finite. The amount of data waiting depends on both incoming volume and how long it waits for downstream processing. Larger beacon payloads or heavier traffic increase the data entering the buffer. Slower consumers extend the waiting duration. Either change can increase the space required, and both can occur during the same live event.

Headroom accommodates those departures from normal operation. It provides space for traffic spikes, periods of slower consumption and the interval before autoscaling adds capacity. Autoscaling takes time to respond to increased demand. Existing capacity must hold the accumulating data during that response interval, so scaling complements headroom rather than replacing it.

This buffer serves a different purpose from gateway batching. The gateway waits briefly to forward small beacons efficiently. Valkey holds shared, collected events while downstream processing catches up. Keeping those responsibilities distinct explains why efficient forwarding alone cannot absorb a sustained difference between ingestion and consumption.

Hand collected events to downstream services

At the downstream boundary, internal services receive event data that has been collected, grouped and buffered. The ingestion architecture has already handled device connections and brought related events into shared collection. Consumers can use that data for the processing their applications require.

Destinations can include Kafka for event distribution, ClickHouse for analytics and S3 for object storage. These services have different downstream roles, but each can begin from the organized event data available at the handoff. The ingestion boundary’s contribution is to prepare that data while accommodating the device traffic that produces it.

Make live-event traffic manageable

The gateway and shared Valkey buffer protect internal processing through cooperating responsibilities. Valkey Router handles device connections and controls ingestion. Batching shares forwarding overhead across small beacons. Deterministic grouping keys bring related events from different gateways into coherent data streams. Shared buffering gives those events room to wait through temporary differences between ingestion and consumption.

Together, these mechanisms let internal services process organized event data while the boundary handles unpredictable device traffic. Their combined effect depends on sufficient capacity for the traffic and its waiting duration. The architecture makes that relationship explicit, connecting efficient ingestion with the headroom needed to support real-time processing during live events.

If you are designing this boundary, bring your expected beacon traffic, payload sizes, grouping requirements and downstream processing needs to Momento. We can help you explore how Valkey Router and shared Valkey collection fit your workload and topology.

A shared Valkey tier gives those results a place that every compatible worker can reach. Momento’s RDMA and AWS EFA modules provide a direct path between inference GPUs and KV blocks in Valkey RAM. Its NVMe module adds local disk capacity, and its object storage connector asynchronously flushes data to inexpensive long term storage. These capabilities let you retain reusable state beyond the lifetime and memory budget of an individual worker.

Consider an assistant answering questions about an operations manual. Each prompt begins with the same system instructions and manual, followed by a different question. The objective is to reuse that shared prefix wherever the next question runs.

Keep the useful state

KV cache contains the attention keys and values computed for prior tokens. Prefill processes the input and establishes this state. Decode uses it to generate the response and extends it as new tokens arrive. Keeping those tensors avoids recreating them during generation.

Prefix caching extends reuse across requests. If a later prompt has the same token prefix and compatible model state, the engine can reuse completed cached blocks. It still processes the new question and generates an answer. Matching subject matter is insufficient. Changing the instructions or moving the manual later in the prompt can change the reusable prefix.

Remote reuse also differs from prefill and decode disaggregation. Disaggregation runs the two phases on separate worker pools and requires a state handoff between them. A shared cache can support that design, but it also benefits workers that run both phases together. The manual example uses such workers.

Put Valkey between inference and storage

Deploy Valkey on storage nodes separate from the inference workers. GPU memory holds state needed for active inference. Valkey RAM holds reusable blocks shared across the fleet. Local NVMe on the Valkey nodes and an object bucket extend the retained working set.

Separating storage from compute lets you add cache capacity without buying another inference GPU. It also gives new or replaced workers access to previously computed prefixes. Valkey supplies an open source storage foundation with a module extension mechanism. Momento’s modules add the transfer and storage capabilities needed for this architecture.

The following reference design stages lower tier blocks through Valkey RAM. Placement, promotion, and coordination are architectural choices for this design.

Direct GPU transfers use Valkey RAM. Lower tier blocks return through RAM in this reference design. Object writes run asynchronously, separately from immediate reuse.

The runtime integration identifies reusable blocks, coordinates availability, and arranges for tensors to become usable GPU cache state. Lookup and bulk transfer have separate roles. Finding a block does not mean that the GPU is ready to consume it.

Suppose worker A has processed the manual and exported eligible completed blocks. Worker B receives another question and checks its local cache first. For a local miss, the integration looks for compatible prefix blocks in Valkey. A RAM hit uses Momento’s RDMA or EFA module to move the blocks into worker B’s GPU memory. The runtime then processes the unmatched prompt suffix and continues generation.

If no useful blocks are available, worker B computes the prefix. Its new blocks can be admitted to shared storage for later requests. Keep retrieval bounded so a slow lookup does not turn a recoverable miss into a long wait. Once RAM reuse works, the next step is retaining more useful prefixes than RAM alone can hold.

Extend the working set with NVMe and object storage

Give each tier a distinct role. Valkey RAM serves shared prefixes used frequently. Momento’s NVMe module provides more capacity on the storage node for blocks worth keeping outside RAM. Object storage provides a larger retention tier for contexts that may return after hours, sessions, or worker replacements.

In this reference design, requested NVMe blocks are staged into Valkey RAM and then transferred to the GPU through the same fast path. The disk is local to Valkey, so inference workers access it through the storage service. They do not each need a disk copy of every manual.

Momento’s object connector flushes data asynchronously. A block already available in RAM can be reused while its object copy is being written. That decouples immediate reuse from the upload, but it does not make every cache write immediately durable. If the only usable copy disappears before the flush completes, the prefix may need recomputation.

For an object hit, the proposed return path reads retained blocks into Valkey RAM before loading the GPU. Recoverable lookup metadata must accompany retained state so the system can identify compatible objects after a restart. Object storage extends the reuse window, while retrieval still has to fit the request’s latency budget. Choose a storage class that supports the access times you need.

Admission and retention policies keep this hierarchy useful. Prefer blocks likely to be reused over a stream of one-time prompts. Keep copies available until transfers using them finish. Coordinate expiry across lookup metadata and lower tiers so deleted or incompatible state cannot appear as a usable hit. Evaluate the tradeoffs between capacity and coordination when defining your architecture.

Workers still perform attention using state loaded into their GPU cache. The remote tiers preserve reusable blocks between uses. If reading and installing an older prefix costs more than recomputing it, a bounded fallback is the better serving decision. That comparison makes the network part of the cache design.

Match the network to the workload

Measure effective block transfer bandwidth under serving load, including lookup, lower tier reads, and GPU installation. Network line rate alone cannot tell you whether a hit will beat prefill. Concurrent inference traffic and cache fills also compete for bandwidth.

RDMA reduces overhead by moving data between registered memory regions without the usual socket data path. With compatible hardware and software, GPU direct transfer avoids staging the payload through inference host memory. Momento supplies the Valkey side of that direct path.

On AWS, use Momento’s EFA module with compatible GPU and storage endpoints. Confirm the instance pairing and GPU direct support before provisioning. EFA device traffic stays within an Availability Zone and VPC. Place the object bucket in the same Region and access it through normal object APIs.

Connect the serving stack

Momento Cache is the leading provider of Valkey, an open-source fork of Redis built for real-time workloads. With Momento Cache and Valkey, your team can enjoy the same high-performance infrastructure that powers global leaders like Snap, FOX, Coinbase, and Capcom.

Momento provides direct integrations with SGLang, vLLM, and LMCache. Choose the path that fits your serving stack. For the manual example, start with the direct vLLM integration if vLLM already runs your workers. If LMCache already manages offloading, use Momento’s LMCache integration. The direct SGLang integration offers the corresponding entry point for SGLang deployments.

Both workers need the same model revision and adapters, matching tokenized prefixes, and compatible tensor representation and parallel layout. Give a new model or cache format its own namespace. Scope sharing to authorized tenants or trust groups, and protect storage access independently of prefix matching. A content hash alone is not an authorization boundary. These requirements belong in the integration and deployment configuration so application code can stay focused on requests.

How to measure the impact of a shared cache

A useful first deployment needs two compatible inference workers, one separate Valkey node with RAM and NVMe, and an object bucket. Obtain the Momento modules and runtime integration for the chosen topology. Start by proving shared RAM reuse before introducing disk and object retrieval.

Send a long manual prefix to worker A, then send a different question with that same prefix to worker B. Keep local prefix caching enabled in your baseline. Confirm that B lacks a local copy and obtains compatible blocks from shared storage. Compare its time to first token with recomputing the prefix under the same serving conditions.

Next, grow the working set beyond the RAM tier to exercise NVMe. Test object retrieval using blocks whose flush has completed and whose faster copies have been removed through supported controls. Restart an inference worker to check that reuse survives worker turnover. Test an unavailable remote tier and verify bounded retrieval followed by recomputation.

Measure tail latency and transfer time as concurrency rises. Track hits and retrieved bytes by tier, GPU memory pressure, and the object flush backlog. Include cache memory, disk, object requests, and network costs when judging the result. A hit rate without its retrieval cost can make an expensive cache look successful.

Accelerate your KV caching with Valkey and Momento

Valkey is a practical solution when repeated prefixes outlive one worker’s cache and loading their state beats recomputation. Momento’s modules connect that shared RAM tier to inference GPUs and extend it onto disk and object storage. Looking for advice or help to trying it out? Get in touch with Momento and one of our solution engineers will be happy to take a look!

I wanted a reproducible benchmark to assess the progress through the builds. Here is the same benchmark, same tool (sorted-set-benchmark, pipelined ZADD NX, members m:{i}, integer scores), same Graviton4 box (r8g.4xlarge), same protocol: ten repetitions per version with a flush in between, official docker images with --save '' --appendonly no, one server at a time, single benchmark client on the same host. Six Redis versions, five Valkey versions.

The numbers

Memory figures are the server’s used_memory (decimal GB, as the tool reports it). Every version’s ten repetitions landed on the same value at the tool’s reporting precision, roughly 10 MB granularity, with mem_fragmentation_ratio at 1.01x throughout, so RSS tracks these numbers closely. Throughput spreads across the ten runs were tight; the two headline configurations don’t overlap (Valkey 9.1: 643 to 652k/s; Redis 8.10: 628 to 639k/s).

Three things stand out. First, every version of both engines that predates this optimization work sits at the same 4.83 GB for this workload, which is what you’d expect given the shared codebase history of the ZSET implementation. Second, Redis ships a comparable reduction: memory drops from 4.83 GB to 3.73 GB, 23%, exactly at the 8.6 boundary (8.4.7 has the old layout, 8.6.7 the new one), with median inserts rising from 536k/s to 613k/s. Third, Valkey 9.1.2 remains the smallest and fastest configuration I tested: about 10% less memory than Redis 8.10.2 and a small but consistent throughput edge.

The same idea, traveling

I found it really interesting that Redis’s newest number lands within one percent of Valkey 8.1’s: 3.73 GB versus 3.77, about 75 bytes per member either way. Redis in late 2026 is, byte for byte, roughly Valkey of April 2025. That’s not a coincidence, and you can trace exactly why through public pull requests and release notes.

Valkey PR #1427 (merged January 2025, shipped in Valkey 8.1 that April) restructured the sorted set so the hash lookup structure points directly at the skiplist node, instead of keeping its own copy of the mapping. In this benchmark that’s the 4.83 to 3.77 GB step. Valkey PR #2508 (merged November 2025, shipped in 9.1) then embedded the member string inside the skiplist node itself, removing a pointer and a separate allocation per element. That’s the 3.77 to 3.34 GB step.

Redis PR #14701 (merged January 2026, a year and ten days after Valkey’s) copied this work to Redis, and the PR description says where it came from, verbatim: “This is based on: valkey-io/valkey/pull/1427.” Credit to the engineer for saying so plainly; not everyone does. The adaptation kept Redis’s dict (in no_value mode) instead of adopting Valkey’s new hashtable, grafted on the node embedding, and shipped in Redis 8.6. My measurements agree with their release notes: the drop lands exactly at 8.6.

So why is Redis still 10% behind? Because copying is a trailing indicator. Redis took the design but not the hashtable it was built on, and by the time the port shipped, Valkey had already taken the next step. That’s the whole gap: the piece of the design they didn’t adopt, plus the release they haven’t caught up to yet. I mean this without a drop of snark toward the engineer who did the work: Valkey is BSD licensed precisely so anyone, including Redis, can build on it, and Redis users are genuinely better off for it. If the ideas flowed the other way, I’d report that too. But I benchmark what shipped, and what shipped says one project is setting the pace on this data structure and the other is matching it, about a year behind.

The operator’s takeaway

Same caveats as always: one large skiplist-encoded ZSET, short members, integer scores, allocator-level accounting, a single pipelined client. Your data shape will land somewhere else, so run the tool against your own workload.

If you run Valkey, this is simple: 9.1 posted the lowest memory and the highest insert rate of everything on this table, and the upgrade is the whole optimization. I’ll keep running this benchmark as both projects ship.

If you run Redis and are worried that Valkey is more efficient, you have two options. You can wait for Redis to eventually copy the latest Valkey PR, or you can switch to Valkey. What do you plan to do?

Momento Cache: the fastest Valkey in the cloud

You always get the latest Valkey through Momento’s managed services, without managing the infrastructure yourself. Momento smoothly handles upgrades, failover, optimization, and more with the same technology that powers some of the world’s largest caches. Join companies like Snap, Coinbase, Paramount, and Capcom: try Momento Cache now!

First question: what did the show floor say that IBC Daily didn’t?

“The official story this year had three headlines: latency cut by three, AI everywhere, and archives waiting to be monetized. So what was overplayed, and what should we have been talking about instead?”

Overplayed #1: “AI everywhere”

Walk the halls and count. About 90% of the AI I ran into was captioning, reframing, generated metadata, and synthetic hosts or actors. Great demos, and genuinely useful stuff, but not transformational stuff.

Very little of it survives contact with a real live event. Put it in front of a World Cup match, where the high-stakes money decisions are still left to humans, and most of it quietly steps aside.

Overplayed #2: “Latency cut by three”

Sure, the glass-to-glass number is real. But in a demo environment, at a trade show.

In production, a different set of problems eats that gain alive: ad insertion, content protection, the last mile, and everyone’s favorite, the sync problem where the goal alert hits your phone before the goal hits your screen. Sub-second end-to-end streaming is achievable in 2026, but nobody is paying for it yet at premium live scale. We’ve already established that streaming can beat broadcast latency (~5 to 6 sec glass to glass). Does that cut it for most live content?

The blind spot: the plumbing

What nobody said loudly enough is that the plumbing decides whether a big live event holds or falls over. Concurrency spikes, the token bomb at kickoff, auth, failover, cache hit ratio, origin offload. Real-time observability, meaning actual real time, not dashboards running three or five minutes behind while your audience is already on social media telling everyone the stream is down.

One line sums up the conversation this industry should have been having in 2026, the one I kept from the FIFA World Cup: 104 matches. 5 billion viewers. 0 margin for error.

Second question: September 2027, same hall. What sounds ridiculous, and what takes over?

The phrase that will age worst: “AI-powered”

As a selling point, it’s already on borrowed time. By 2027, AI will be assumed by default, the same way cloud became. Nobody sells “cloud” anymore. You sell what you do with it. “AI-powered” is heading to the same retirement home.

The topic that will own the show: actually two, and they’re linked

The media landscape is going through a major consolidation

Platforms are consolidating into mega-conglomerates. Not only the big mergers that have made the big news throughout 2026: Fubo + Disney, WBD + Paramount… but also a trend of legacy broadcasters signing distribution deals that push their content onto the same large ubiquitous platforms. At IBC, I had discussions on how TF1 getting distributed by Netflix, or France Télévisions with Prime Video are durably changing the French streaming landscape. The result is simple math for both vendors and content owners: the number of real opportunities to differentiate is collapsing. Fewer buyers, bigger bets, less room for error on every one of them.

And how do I get my token consumption under control?

Make or buy gets harder in 2027, because everyone is now using AI to build their product and try to stand out in a shrinking market. But AI-driven development is a one-way door. You create technical debt, and you need more and more tokens just to maintain a codebase the AI wrote. You can’t hand 100,000 lines of generated code to the last senior engineers you haven’t laid off yet and ask them to debug it overnight.

Put those two together and you get the real 2027 question: with fewer customers to win and a growing token bill to feed, what do you build yourself, and what do you trust someone else to run at scale?

That’s the conversation I want to have at IBC in Amsterdam next year. The plumbing isn’t the most glamorous part of the stack. But it’s just the part that decides whether you’re still standing at the final whistle.

Preparing for your next major live event? See how Momento handles the plumbing behind real-time media at scale.

Imagine a retailer whose public support assistant answers questions across a large policy catalog, with rules that vary by region and product category. It retrieves policy text and generates a cited explanation for each request. Within that broader workload, some questions recur with only small changes in wording.

One shopper asks, “How long do I have to return headphones?” The assistant explains the 30-day return window, citing the official policy. Then, a second shopper asks, “What’s the return window for headphones?” The same explanation would answer both questions, but the assistant blindly repeats retrieval and generation to produce a similar response.

A semantic response cache avoids that repeated work. Valkey stores previous answers along with metadata about the question. The Valkey Search module uses this metadata to find responses to similar questions. The application can now fetch a set of possible responses and return one that fits.

The benefit is fewer inference calls and shorter waits. Each reused answer avoids generation costs and returns without waiting for a new model response. Across recurring questions, those savings can reduce both the bill and the average response time.

Why Momento Cache for Valkey fits this pattern

Momento Cache is the leading provider of Valkey, an open-source fork of Redis built for real-time workloads. With Momento Cache and Valkey, your team can enjoy the same high-performance infrastructure that powers global leaders like Snap, FOX, Coinbase, and Capcom.

Each semantic cache record contains the original question, its complete answer and citation, and an embedding: a numeric vector representing the question for similarity search. Additional fields such as region, language, product category, condition, and policy version define where the answer can be reused.

Valkey holds those records, and its Search module indexes their vectors and metadata. Combining vector queries with structured filters gives the semantic cache three useful capabilities:

• Keep answers and search data together. Store the response, citation, question vector, and scope fields under one key. A matching record contains everything the application needs to assess and serve the answer.
• Search with the right context. Combine vector similarity with filters for additional fields like region. Search can find the response to a paraphrased question, while still excluding records from a different policy scope.
• Share each cached answer across workers. A response stored in Valkey by one application worker becomes available to others, allowing later requests to benefit regardless of which instance handles them.

Valkey supplies the shared answer store and filtered vector search. The application defines the embedding used for queries, and ultimately decides which proposed responses to accept.

How semantic caching works

A suitable semantic match skips retrieval and generation.

Let’s follow the second shopper’s question about returning headphones to illustrate how semantic caching works in practice:

1. Check eligibility and scope. The application identifies the request as a general return-policy question about headphones. It extracts the US region, English language, and active policy version from application context.

2. Find a candidate answer. An embedding service converts the question to a vector. The application passes this vector and the metadata filters to Valkey with the command FT.SEARCH, which returns a set of response candidates containing an answer, policy citation, metadata, and vector distance.

3. Check whether the answer fits. The application checks the vector distance and confirms that details affecting the answer match. This catches small text differences like “opened” vs “unopened”. If the relevant details match, the application returns the cached explanation.

4. Generate an answer on a miss. If no candidate fits, the application retrieves current policy and generates a fresh response.

5. Store answers that qualify for reuse. After responding, the application stores the new response in Valkey if it’s suitable for caching. For example, an answer containing shopper-specific information does not belong in this shared cache.

Each answer added to Valkey gives later questions another opportunity to skip retrieval and generation. When the next shopper asks about the same return policy in different words, the application can deliver the explanation sooner and avoid another model call.

Lower costs and faster answers

A semantic cache saves time and money by avoiding retrieval and generation when it serves a reusable answer. A 25% hit rate means one in four eligible questions takes that shorter path.

Lower costs. For every $1,000 you spend on generation, a 25% hit rate could save $250! Of course, caching infrastructure adds some cost, but this is usually far less than repeatedly generating duplicate responses.

Faster answers. Retrieval and inference are two of the slowest types of calls, often taking multiple seconds. Meanwhile, Momento Cache returns hits from Valkey in just milliseconds.

The actual semantic cache hit rate depends on the query patterns in your application. The opportunity grows as more questions can reuse an answer: fewer paid model calls for the business, and less waiting for users.

Turn repeated questions into faster answers

When users ask the same thing in different ways, semantic caching lets one answer serve many requests. Valkey Search on Momento Cache combines a shared answer store with filtered vector matching, helping your application reuse relevant responses, reduce generation costs, and shorten the wait for users.

Bring semantic caching to your application: get started with Momento Cache and put your recurring questions to work.

We say it jokingly, but it’s true. Inference isn’t well-known for being fast. Powerful, sure. But not fast.

We’ve done a lot of experimentation to make it faster. Parallel fetch, zero-copy reads, batched metadata lookups, the whole retrieval path. But there was one particularly intriguing experiment that has a lot of upside, but some huge trade-offs. What if we stored the KV cache remotely instead of locally? There’s just one problem. The KV cache is huge. Transporting it over a network is a non-starter. Right?

We know how it works. GPU memory is fastest, then host DRAM, then the NVMe bolted to the same box, then, if you really have to, go across the network. You go remote when you need to share, but you pay for it in latency.

I’m not onto a novel idea though. Our industry is all over it right now. NVIDIA has an offloading tier in Dynamo. Mooncake is building a store. Xinnor published a piece in June arguing that Lustre with a client-side cache can hold its own against local NVMe. Khawaja Shams made the case that once you disaggregate prefill and decode, the cache has to travel between machines whether you like it or not.

Clearly there’s merit to it. But I didn’t see much about the cost of the tradeoff. How much of a latency hit do you really take storing your KV cache in something like Valkey, and is skipping prefill still worth it after you’ve paid it?

So I took it upon myself to run the numbers.

The setup

The benchmark uses three machines: two identical GPU nodes running vLLM, and a Valkey box sitting one 10 Gbps hop away from both of them.

gpu-a does all the computing for prefill. The DRAM and NVMe tiers live inside it, and Valkey sits across the network. gpu-b is used to determine what happens when a cached request lands on a machine that did not perform the prefill.

The workload is thirty realistically generated legal and medical documents, all trimmed to exactly 10,000 tokens. Every document goes to vLLM twice. The first pass does the prefill and stores the KV cache. The second pass sends the same prompt, and I time how long the first token takes to come back (TTFT). The only thing that changes between runs is where the KV cache lives.

Everything runs vLLM 0.28.0 with LMCache 0.5.4, valkey-glide-sync 2.5.1, and Valkey 9.1.1, serving Qwen2.5-7B-Instruct-AWQ on NVIDIA L4s.

To get accurate benchmark results I had to turn off vLLM prefix caching so the GPU wouldn’t serve every repeat request. I also had to clear the disk tier and drop the page cache before every single run, because Linux keeps recently written disk data in memory. If I didn’t do that, the “NVMe” reads would come out of RAM and I wouldn’t have actually tested the disk speed.

The goal was to beat cold TTFT, which falls between 3.1 and 3.3 seconds. I ran the entire benchmark twice on separate instances a day apart, and got reproducible numbers within 2%. So I’m feeling good about what I found. Let’s take a peek and see how the network did.

Keeping everything on one machine

For this first test, gpu-a does all the work. It computes the KV, stores it in whichever tier is being tested, and reads it back on the second pass. Even in the Valkey run it’s gpu-a writing and reading its own cache, just over a wire instead of a local bus.

You’re reading that right. The network was the fastest. And by quite a large margin, at that. It beat the local NVMe by 2.7x on the same machine. The disk is physically inside the box with no switch in the way, and the cache was still quicker to reach from across the network.

(Ignore the DRAM row for now. Something interesting happened there, and we’ll talk about it later.)

At first I thought I broke something. Maybe I accidentally put the “disk” tier on the EBS root volume instead of the instance store? But lsblk showed the tier on nvme0n1 (the 250 GB instance store), and df showed 16 GiB written to it during the run, which is about right for 300,000 tokens of KV.

After doing some math, it started making sense.

Each document’s cached KV is 39 chunks of 256 tokens at 57,344 bytes per token, roughly 572 MB (39 x 256 x 57,344). While I was sanity-checking the harness, I ran the DRAM tier with two documents (so everything fit in memory), and hits came back in about 82 ms. A DRAM hit barely transfers anything, which means 82 ms is basically the overhead cost. So any time beyond 82 ms is what I would consider to be transfer time.

That means Valkey’s 557 ms is really about 473 ms of transfer. 572 MB in 473 ms is 1.2 GB/s, which is a 10 Gbps NIC running flat out. Compare that to the disk, which moved 572 MB in ~1,438 ms. That’s 400 MB/s, which is odd, because NVMe should be able to read north of 2 GB/s.

So the remote tier is bound by the network, and the local tier is bound by its own software. Clearly a lot of work went into LMCache’s Valkey connector this year. It pulls with eight parallel workers straight into caller-owned buffers. The local disk path hasn’t gotten the same attention yet, and 400 MB/s from an NVMe says there’s plenty of opportunity. I imagine it won’t stay this way for long.

Anyway, from these results we see that putting the KV cache on the network didn’t have the latency penalty I thought it would. It ran as fast as physics allows 🔥. Even if the disk path gets some love next month and starts doing 1.5 GB/s, using the network is still a solid choice. Plus, Valkey has room to go even faster with a bigger NIC.

There is one thing I can’t explain. That 2.18x speedup for the disk is a median, and the per-document numbers show something weird. The first document requested in the cached pass comes back around 11x faster. The second comes back around 3.5x. The other 28 are all right at 2.1x. Oddly enough, both runs were basically the same:

Something is warm for the first request or two, then stops. I don’t know what it is, and I’m not going to make up a reason that sounds good. Whatever it’s doing, it makes the disk look better than the median I’m quoting, so it isn’t driving my conclusion.

Let’s talk about DRAM

In my results table above, the DRAM row says it had a 0.95x “speedup”. I know what you’re thinking: I misconfigured it. I didn’t. Remember that two-document sanity check where hits came back in 82 ms? That was this tier with the same config. The TTFT was 3122.8 ms cold and 82.5 ms cached, which is a 37.93x speedup 🤯.

DRAM isn’t slow. DRAM is the fastest thing in the entire stack by a mile. It just… runs out.

Thirty documents is 300,000 tokens of KV. At 57,344 bytes per token, that’s 16 GiB. I gave the tier 8 GB, and the entire host only has 16. The box simply didn’t have the memory, and it evicted everything before I could even run the second pass.

Not only that, but every tier that served nothing came back 3% to 5% slower than cold, on at least 29 of the 30 documents. A lookup that misses is still a lookup, and a store that’s evicting still costs a store. In this instance, the undersized cache was basically a tax.

And thirty documents is nothing when compared to a real production corpus. If your working set is two documents, use DRAM and stop reading this. If it’s thirty or more, skip the DRAM for something with more capacity.

Expanding to two machines

Everything we’ve covered so far has been a single machine writing and then reading its own cache. But the big question is what happens when a request is routed somewhere else, because it will in production. So I ran the cold pass on gpu-a and the cached pass on gpu-b (the one that didn’t do the prefill).

Unsurprisingly, NVMe went down to zero reuse and 3.2 second prefill on gpu-b because the KV didn’t exist there and there was nothing to access. It makes NVMe seem a lot less exciting once you start using it in a distributed environment.

That said, Valkey went from 5.98x to 5.97x. Reading from a machine that did not compute the KV cost nothing measurable. The worst document slipped from 5.55x to 3.92x, and the medians are effectively the same. The cache doesn’t belong to either node. And that was the whole point.

My takeaways

If your working set fits in host DRAM, use that. A cache hit came back 37x faster than recomputing the prefill. Just be honest with yourself about capacity. Measure your corpus in tokens and multiply by your model’s per-token KV. Don’t eyeball it.

Once it stops fitting, local disk gets you 2.18x on the machine that wrote it, but nothing anywhere else. On a single node that’s a considerable boost. But across a fleet, every entry is isolated to the box that produced it, and your hit rate becomes the odds of getting routed back to the same machine.

The network tier isn’t free on the way in. Cold TTFT ran about 110 to 150 ms slower with Valkey than with DRAM, which is a 3% to 4% cost for storing over the wire instead of in memory. So the tradeoff is that you pay 110-150ms per cache fill in exchange for about 2.7 seconds back on every hit after it.

The most surprising result was that the shared tier was already the faster one here. On this hardware, with this model, at this corpus size, putting the cache on another machine made it quicker. I went into this assuming I’d be arguing that sharing is worth a latency cost. I am happy to have been wrong on that one. Skipping prefill is worth what it costs (and then some).

Your KV cache belongs on the network. Not because it has to be shared (though it does). Because that’s where it was fastest.

Try it yourself

The harness, the 30-document corpus, the tier configs and the infrastructure scripts are all in valkey-kvcache-bench. One command provisions all three hosts, runs the matrix, pulls the results down and terminates everything:

cd infra
.\Run-TierBench.ps1 -AwsProfile <profile> -Teardown

It costs about $2.14 an hour and the full benchmark run takes under an hour.

One warning if you try this yourself - the first time I ran the two-machine test, every tier came back 0/30 (including Valkey). It turns out LMCache’s default cache key hash is Python’s hash(), which gets a different random salt in every process. So gpu-a and gpu-b were generating different keys for identical tokens, which made cross-node reuse impossible. LMCache even warns you at startup with a Using builtin hash without PYTHONHASHSEED set message, and I had completely missed it. But you can fix it by setting this config value:

​yaml pre_caching_hash_algorithm: sha256_cbor_64bit ​

If your shared KV cache ever shows a zero hit rate across nodes, it’s probably that.

Anyway, try it out and if you see something different, please feel free to reach out so we can compare.

Happy coding!

Momento Cache is built for fast-moving teams who want optimal Valkey performance without the hassle of babysitting infrastructure. Cluster capacity and Flex capacity bring single-tenant, fully-managed resources to this self-service infrastructure platform.

Cluster provides direct control over instance type, shards, replicas, and availability zones. Flex automatically optimizes resources within specified bounds. Both were designed for workloads that need stronger isolation, more control, and a higher performance ceiling than Momento Cache’s Serverless configuration.

Avoiding the trap of operational creep

In the AI era, it’s trivial to stand up basic infrastructure at near-zero cost. We’ve all been there: it’s fast, it’s easy, it mostly works. It’s the right solution when you need to ship.

Then, growth hits. Traffic changes shape. Memory fills unevenly. A shard needs to move. A primary fails. Clients stampede. A zero-day patch lands at midnight. Suddenly, operational creep has consumed the time and the token budget that you wanted to spend on building.

And the problems only multiply as you add more features, products, and services. Soon your entire team is stuck fighting against infrastructure as latency, cost, and complexity steadily creep up and to the right.

Fast, reliable, efficient - pick all three

The Momento platform powers critical features for millions of users around the world at companies like Capcom, Coinbase, Paramount, and Snap. Now, Momento Cache puts the full power and flexibility of Valkey at your fingertips, packaging up hard-won production lessons into a streamlined service.

The new Cluster and Flex configurations help you to tailor a Valkey deployment to fit your specific needs. Momento seamlessly operates the lifecycle behind that system: provisioning, health, failover, rolling topology changes, version upgrades, and security patches.

Whether you’re pushing one thousand or one million requests per second, Momento delivers unrivaled performance and resource utilization. As you grow, your infrastructure grows alongside you. For companies with a mature platform org, Momento Cache can even be deployed in a BYOC configuration, as part of your internal developer platform.

Transparent pricing

Momento Cache pricing is designed to be simple, predictable, and cost-effective. Pricing in us-east-1:

• Flex starts at $13 per GB-month of physical Valkey storage
• Cluster applies a 25% surcharge to the list price for every deployed instance
• Data transfer includes 200 GB of ingress + egress each month, then $0.05/GB

Momento Cache supports rapid autoscaling within a specified capacity range, making it easy to reduce the cost of idle resources.

Designing for scale

Momento Cache employs a two-tiered architecture that significantly improves reliability and efficiency at scale. Each valkey cluster sits behind a gateway that handles the hard traffic problems like hot keys and connection storms before they hit the data layer.

The gateway exposes a RESP endpoint, so Momento Cache is compatible with all standard Valkey and Redis clients. It masks the underlying cluster, presenting a single stable endpoint across any topology changes.

block
  app("Redis or Valkey\nclient"):3
  space
  gateway("gateway"):3
  space
  pool("Valkey cluster"):3

  app --> gateway
  gateway --> pool

The gateway is optimized to quickly process TLS, auth, rate limits, and other traffic management concerns at high concurrency and high throughput. It multiplexes client traffic across a pool of warm connections to the Valkey cluster. This reduces connection latency, and enables advanced capabilities like request coalescing.

The result is an efficient system with deliberate separation of responsibilities. The gateway absorbs traffic concerns, while Valkey nodes focus their resources on handling data.

Get started with Momento Cache

Try out Momento Cache in a few short steps with the Momento CLI. While the service is still in preview, you’ll also need to request access via the web console.

First, create an API key in the Momento console, copy the endpoint for your region, and configure the default CLI profile:

# paste the api key and endpoint when prompted
momento configure

Then, create a Capacity Pool. Be sure to provide valid zone IDs for your region:

momento preview pool create \
  --name example-pool \
  --capacity-gib 32..128 \
  --replicas-per-shard 1..2 \
  --zones use1-az1,use1-az2

momento preview pool describe --name example-pool

Once the Capacity Pool status is active, you can create a Database:

momento preview database create \
  --name example-db \
  --pool-name example-pool

Back in the console, open the pool’s Databases tab and copy the regional RESP endpoint. Connect any Valkey or Redis client in standalone mode to this endpoint over TLS on port 6379. The Database name is the username, and your Momento API key or token is the password. Here, we’ll demonstrate with the official valkey cli:

valkey-cli -h <resp-endpoint> -p 6379 --tls \
  --user example-db --pass <momento-api-key>

> SET example-key "ready"
OK
> GET example-key
"ready"

Congratulations! You now have a high-performance Valkey cluster ready to go.

Next, check out the docs to learn more about Momento Cache’s capabilities. Or, if you want to push your cache to the limit, load up cachecannon on a c7g.xlarge instance in the same region and zone as your Database!

Up next

This launch begins the next chapter for Momento Cache. Stay tuned as we port more features from our enterprise services into Momento Cache, including VPC peering and S3 integration!

We’re looking for feedback from teams operating demanding Valkey workloads as we refine the product. If you’re building a fast-growing product, operating a large Valkey or Redis cluster, designing an internal caching platform, or helping teams adopt Valkey, we would love to hear what you need next and where we can help out.

We’re grateful to the engineers and partners who turned years of demanding operating experience into a service anyone can start using today. Special thanks to Dylan Abraham and Jason LaPier for leading the development effort!

But doubling the number of indexes I was using made me nervous. I didn’t know what impact that would have on performance. I’m using Valkey because I needed ultra-low latency, I didn’t want to shoot myself in the foot trying to be clever.

And yes, going from one to two indexes is probably not a big deal. But indexes have a way of piling up. You create one to filter products by category, then add price to it a month later, then the search team wants a vector field, then the payment team indexes their keys, then fulfillment indexes theirs. Before you know it, there are a dozen FT.CREATE statements in the repo with no single owner.

My instinct tells me that performance scales inversely with the count. Every index in valkey-search subscribes to a key prefix. When you write a matching key, that index gets an entry in a mutation queue, and your client stays blocked until every entry is processed. That’s what gives you read-after-write consistency. If you have a dozen indexes, you also have a dozen entries, and a dozen times the wait. Right?

So I built a benchmark to see how much additional indexes cost in valkey-search. And I discovered I was very wrong.

The setup

The server was an r7i.4xlarge with 8 physical cores, running Valkey 9.1.1 and valkey-search 1.2.1. valkey-search sized its own writer pool to 8 threads. I turned persistence off, because I didn’t need a background save forking mid-run throwing a wrench into my latency numbers. Load came from a separate c7i.8xlarge in the same placement group, over 384 connections.

Every write is the same 4.6KB product hash that includes a category, a price, a SKU, a title, a description, and a 1024-dimension embedding. The load generator sends on a schedule and times each request from when it was supposed to go out, not when it managed to (more on this later). Every point runs for 25 seconds, three times, and I capture the median.

Do idle indexes cost anything?

As I mentioned earlier, indexes subscribe to a key prefix. If a key is upserted that doesn’t match on an index, does that slow things down? In other words, my test here determines if the existence of indexes slows down the performance of others.

I ran the same write workload against product: keys in two setups: one index on product:, and another one with the same index plus fifty more indexes on unrelated prefixes.

Nice! I couldn’t measure a difference between them at any rate I tried.

This is because the prefix subscriptions live in a trie. When you write product:88213, valkey-search walks the trie and only notifies the indexes whose prefix matches. An index subscribed to orders: doesn’t hear about it. FT.INFO idx:other0 reports a field called mutation_queue_size, and across the entire run it never left zero, which validates the promise of the trie architecture.

So that hodgepodge of indexes in your repo isn’t what’s slowing your writes down. Only indexes whose prefix matches the keys you write are in the path at all.

The impact of matching indexes

So what’s the performance impact of having multiple matching indexes? I ran the same workload again, this time adding multiple identical indexes directly on the product: prefix.

Every matching index gets an indexing job, and the write isn’t done until all of them are. To figure this out, I took Valkey’s search_ingest_hash_keys counter and divided by the writes that completed in the same window. I took this number and multiplied it by the write rate to calculate the indexing jobs/sec column.

The expensive part, naturally, is switching indexing on in the first place. Going from zero indexes to one cost me 58% of my write rate. The second index cost another 41%. Doubling from two to four only cost 16%. Each additional index cost less than the previous one.

Adding indexes pulls more total indexing work per second out of the same server, 33,636 jobs a second at one index and 67,272 at four. So one or two indexes clearly weren’t saturating the writer pool, because it eventually went on to do twice the work.

Indexing jobs fan out across a pool sized to your physical core count, and the blocked-client handles collapse into a single block on your connection. The write waits for the slowest single job, which is why four indexes don’t cost four times what one does.

Unfortunately I don’t know what the limiting factor is. My hunch is that it’s the per-index bookkeeping that happens on the main thread before a job reaches the pool, but I didn’t measure that, so 🤷.

NOTE - Adding matching indexes won’t appear to add latency until it’s too late. At 2,000 writes/sec, one index and four indexes were within a millisecond of each other. You won’t catch it watching p99 on a healthy system, because what you’re spending is headroom. You find out it’s gone when you need it.

Vector fields hit different

The benchmarks above use TAG and NUMERIC fields, just the normal filter field types. So I went back to the first benchmark, added a single 1024-dimension HNSW vector field to the one-index setup, and re-ran it to find some staggering results.

32,000 writes per second became 2,828. 🤯

That’s an 11x difference in throughput because of a single field. My napkin math says that’s 2-3 milliseconds of CPU per vector insert spread across eight writer threads. To make matters worse, the performance cliffs. I increased the rate on my benchmark by ~40%, and the wheels fell off.

The performance hit goes from five milliseconds to two seconds. The queue went from basically empty to 376 entries deep, and it stayed there for every rate I tried above that. You’re either under the line and fine, or over it and everything is late. If you’re capacity planning, be sure to check whether an index on your hot prefix has a vector field.

Do vector fields affect other matching indexes?

Back to the search feature I was building that required two indexes. A single index can’t do both jobs because of stop words. The text pipeline is configured for the entire index, and it splits words on punctuation before dropping anything in the stop word list. So IT-500 becomes it and 500, it is a stop word, so just 500 is added to the index. Turning on NOSTOPWORDS means your description field will index every the, is, and, and it (plus a lot more) in the catalog. So we need two indexes to have it turned on for one and off for the other.

But that made me wonder what the second index costs when the first one has a vector field. We saw how much of a hit it made to throughput in our earlier benchmarks.

About a three percent difference. The exact match index processed 141,408 text fields during the benchmark run. Both mutations hit the pool together, the write waits for the slower one (the vector). So if you’re already paying the HNSW tax, the lexical index has essentially no additional latency.

My takeaways

So it turns out I was asking the wrong question when I started this experiment. I thought the number of indexes I had was going to slow performance down to a crawl. But it doesn’t. The real question is what fields are inside the indexes that match your keys? Which is a relief, when I think about it.

I also learned a couple of things about benchmarking while I was busy answering the wrong question. 😅

Every sustained writes/sec number in this post could have been bigger. At the top rung of my ladder, the no-index setup completed 128,000 writes a second (but my tables show 80,000). At that run rate, it was making every request wait 1.9 seconds in a queue first. A server running at full utilization drains as fast as it fills, so throughput looks perfect, but at a cost to latency. So the number I used in every table is the highest rate where p99 stayed within reason.

It took me three runs to believe what I saw in that four-row table early in this post. The first run stepped the rate by 1.4x per rung, which put two indexes and four indexes on the same 16,000 rung. Which at first made me think indexes three and four had no performance implications. But in reality, they had both fallen apart somewhere between rungs and the ladder couldn’t show me where. So I re-ran it with 1.19x steps and got a cleaner separation. Then I noticed that ladder started at 14,000, which is already well up the curve, so it never measured what latency looks like when the server is idle. My rule for picking a sustainable rate is relative to that idle number, which meant the second run was grading itself on a curve. The third run started at 2,000 and is the one in the table. A rate ladder can’t resolve a difference smaller than its own step, and it can’t tell you where the knee is if it never saw the flat part before it.

Try it yourself

I have the benchmark, scripts, and results available in GitHub. If you want to check my numbers (or disagree with them!), please do and let me know what you find.

If you want to run the same tests on your own Valkey cluster, you can run these three commands:

FT.INFO <index> # shows mutation_queue_size (write backlog) for the specified index
INFO search # shows search_writer_queue_size for the whole pool
CONFIG SET search.info-developer-visible yes # unlocks per-field-type counters like search_ingest_field_vector

It’s cheap to experiment with your existing clusters because these mutations are reversible. FT.DROPINDEX is instant and your data was never in the index to begin with. Add a shape, measure it, then discard it.

Happy coding!

Unfortunately it’s not that simple. Memories need to be similar, yes, but they also need to be recent. You don’t want memories from a year ago influencing your agent. And outcome is important too. If one approach worked and another failed, you probably want the successful one steering behavior.

I ran into this while building a small demo that puts Valkey Search inside an agent’s inference loop. Every task the agent finishes gets written as a memory. It persists the task text, the approach that worked, whether it succeeded, when it happened, and a vector of the task. Next time a similar task shows up, the agent recalls what worked instead of rediscovering it.

I ended up learning how valkey-search handles this the hard way, because my first FT.SEARCH call did not work the way I expected:

FT.SEARCH idx:memory "(@outcome:{success} @created_at:[1782900000 +inf])=>[KNN 3 @vector $vec]" PARAMS 2 vec <query-vector> DIALECT 2

Left of the => is an ordinary boolean filter of a tag and a numeric range over two fields I indexed alongside the embedding. Right of it is the vector search.

I thought Valkey was going to find the nearest vectors and apply the tag and timestamp filters afterward. But it doesn’t work like that (in a good way). The filter criteria are actually inputs to the query planner. Depending on how many memories they match, Valkey chooses a different algorithm to search the vector index. In other words, adding a filter changes how the search runs, which meant I needed to rethink my initial memory schema. Turns out my retrieval policy was also an index-design decision.

Nobody post-filters anymore

If you’ve used Pinecone, Qdrant, Weaviate, or Milvus, then this should be familiar. All of them decide at query time whether to walk the graph with your filter applied or abandon the graph and brute-force the matching subset instead. Pinecone calls it single-stage filtering. Qdrant calls it query planning. Weaviate calls it a flat search cutoff. Milvus doesn’t really call it anything, it just does it 😂. valkey-search is the same idea, and on the FT.SEARCH vector path it doesn’t implement post-filtering at all.

If you’re a pgvector user, however, filtering happens after the index scan. With the default hnsw.ef_search of 40, a filter that keeps roughly 10% of those candidates might leave you with only 4 results. Iterative index scans were added in 0.8.0 to make that a little better, and they’re off by default.

valkey-search calls it a query planner. It estimates how many keys your filter matches and picks one of two algorithms to perform the search.

If the estimate is small compared to the index, it pre-filters results by walking the qualified key set, computing each distance directly, and keeping a top-k heap. The HNSW graph is never traversed. So valkey-search is essentially doing a brute-force scan over a tiny set.

If the estimate is large, it performs an inline filter. Your predicate is handed to hnswlib as an isIdAllowed functor and evaluated during traversal of the base layer. Non-matching nodes are still visited and expanded, they just don’t get added to the result set.

The cutoff point for one algorithm vs the other is 0.001. So if the number of estimated matching keys is at most .1% of the number of vectors in the index, it will go the brute-force route. Otherwise it uses the inline filter.

For reference, the cutoff point for Milvus is around 7%, which makes valkey-search about 70x stricter. You end up on the inline path more often than you’d think.

Make the filter disappear

On my demo index with a few hundred memories, the threshold works out to well under 1 key, so every recall that matches anything takes the inline path. You’d never notice either way at that scale.

But what would happen with the same schema in production with 1,000,000 memories? The threshold is 1,000 keys. The planner considers the selectiveness of the filter. If @outcome:{success} matches 70% of the index and your @created_at window is also broad, you’re squarely in the inline path. And the planner is right to put you there. If you scope recall to a single tenant with 400 memories, you drop under the threshold, and the query becomes an exact scan. Perfect, fast recall because 400 distance computations is nothing.

Let’s make it more difficult. A filter matching 1% of a large index sits 10x above the cutoff point, so it takes the inline path, where HNSW traverses a significant number of candidates for every one it’s allowed to keep. That results in a lot of extra latency you didn’t account for. And you can’t change that by tuning it, because search.prefiltering-threshold-ratio is immutable unless search.debug-mode is on, and it isn’t in the public configurables table at all.

You can check which path you’re actually on, by the way. valkey-search counts both, named search_prefiltering_requests_count and search_inline_filtering_requests_count. They’re module fields, so you need INFO SEARCH and not plain INFO. You can run your recall query a hundred times to see which one moves.

So your best bet is the index itself. Make the filter act like a namespace. Put a hash tag in the index name, prefix the keys to match, and each tenant’s recall hits one shard against a small index where the filter is mostly irrelevant. Valkey enforces it in both directions, too. A tagged index name requires every prefix to carry the same tag, and an untagged one requires that none of them do. Be careful with this though, because it’s not easy to undo if you change your mind since there’s no FT.ALTER here.

Forgetting is expensive

Removing a vector from an HNSW index calls markDelete and returns. The node isn’t removed from the graph, it’s still visited, it still routes other searches through itself, it just fails the deleted check. search.hnsw-allow-replace-deleted would let the next insert reuse that slot, but it’s false by default and more of a non-production flag. In production the space is stranded until you drop the index.

Luckily, updates are a different story. Modifying an indexed vector routes to an in-place update under the existing label, so the node keeps its slot and just gets its links rewired. Writing a hash field that isn’t the vector doesn’t mess with the graph, and rewriting the vector with identical bytes short-circuits before it gets there.

So updates are cheap, but deletion is where it gets expensive. That includes DEL, eviction, and expiry.

That’s a little scary, because expiry is the recency policy. valkey-search subscribes to generic, expired, and evicted keyspace notifications, so a TTL’d memory really does leave the vector index when the key goes away. Which is what you want, but it’s also what makes graph nodes stranded.

Which means you have to change how you key your memories. Creating a new key every run feels like a safe and reasonable default, and it’s what my demo does with its memory:<id> per completed task. But add a TTL to keep things fresh, and every expiring key leaves a node behind. That’s an expensive default at scale. Instead, use a stable key derived from a task fingerprint, and update it in place as the agent learns more about that kind of task. This means the task costs only one node instead of one per attempt.

There are two recency controls here to consider. The @created_at range decides what you’re willing to believe on any given query, and it doesn’t delete anything. The TTL decides what you’re willing to pay to store. Range should be your first lever and the TTL should be the slower fallback.

Decide before you have data

This demo surprised me twice: the filter I wrote as a query detail decides which algorithm runs, and the TTL I (almost) added to keep memories fresh led to stranding the index.

Everything here is discoverable, at least. The planner code is a short function to read and understand. The filtering comes from hnswlib. Even the deleted node behavior is a comment in the source, and the counters are available in INFO SEARCH. You don’t have to take my word for any of it (but you should 😜).

What you can’t read your way out of is when you have to make decisions. How the index is scoped and how a memory is keyed are day-one calls, made before you have a single memory to check them against. Outside of that dev-only flag, a rebuild is the only way to reclaim space that deletion stranded. Get those wrong and you’re reindexing.

The demo is on GitHub if you want somewhere to start. docker compose up -d gets you Valkey with the search module and an agent that writes its own memories. Run a few tasks through it, then look at INFO SEARCH and find out which path your queries are actually taking.

Happy coding!

I believe we did, and the numbers back it up.

The Room We Wanted

Here is the stat I am proudest of: we had 117 attendees show up out of 124 registrations. That is a 94.4% show-rate. If you have ever organized a tech conference, you know the typical number hovers around 60%. People register, life happens, half the badges never get picked up.

That did not happen here, and it tells you something about who was in the room. Definitely not people collecting free T-shirts, stickers, Mo plushies or vintage green Elemental swag (thanks Kiran!). Of the seven no-shows, six were from sponsoring organizations who had other folks covering the event.

Those 117 people represented 41 unique organizations, from the largest broadcasters, tech platforms, video infrastructure providers… the full spectrum of who actually builds and runs media workloads at scale. When you get that kind of density of real operators in one place, the hallway conversations become as valuable as the sessions. Actually, maybe even more.

We Pre-Screened the Pitches Out

Here is the thing we are strict about. BuffConf is not a place to sell. Every proposed session got human-screened, and anything that read like a vendor pitch got sent back. We prioritized end-user and practitioner voices, the people who have actually been paged at 3am when things flinched.

The result was 22 technical sessions with 29 speakers from 17 companies. And critically, 40% of our speakers were non-vendors. We are talking real production stories from operators like Paramount, Netflix, FOX, CBS Interactive, Meta and Red Bull Media House. Not “here is our roadmap.” More like “here is what we tried, what failed and what finally held up.”

It would be easy to fill an agenda with vendors who have marketing budgets and room for another video nerd conference (all the more when the traditional bigger one is seemingly canceled silently this year… of course you know what I’m talking about!). It is much harder, and also much more valuable, to get the operators to stand up and tell the unvarnished truth about their infrastructure. When they do, everyone in the room learns something they can use in their own work.

What The Presentations Covered

Three main themes ran through the two days, and a handful of sessions captured each one.

Production workflows at scale. I opened with “104 Matches. Five Billion Viewers. Zero Margin for Error,” walking through the FIFA 2026 World Cup numbers and what global live scale actually demands, with no excuses. Spencer Shanson from Paramount then told the story of scaling a FAST platform from a handful of channels to thousands. That kind of journey, from a few streams to an entire catalog of linear channels, is exactly the operational reality most of the room is living on a daily basis, and hearing it from someone who has actually done it is worth more than any architecture diagram. And when “production at scale” means “at Netflix’s scale”, who better than Sunjeet Singh and William Schor could explain how Netflix handles over one billion RPS on 120,000 nodes?

Practical AI, and honest boundaries around it. This was the part I found most refreshing. Nobody stood up and told us AI was going to solve everything. It’s still a journey, and our keynote speaker, Manish Rao, showed us all the progress that AWS Elemental Inference has made since NAB. It’s staggering how three months can mean a world of difference at the current speed of AI innovation. Edwin Rivera from CBS Interactive showed how CBS Sports built an AI-powered “For You Page,” blending editorial curation with embedding-based recommendations to actually move engagement. Content discovery in sports is one of the richest problem spaces in our industry right now. And Jeffrey Kember from NVIDIA walked through what accelerated computing is doing for encoding, quality, and real-time media workflows. A reminder that we are genuinely at the beginning of AI for media, not the end. Across the AI sessions, we also saw token efficiency work borrowed straight from video compression concepts, automated QC loops, and an edge anti-piracy simulation harness. Real implementations, with the scars to prove it, paired with hands-on framework training so engineering teams could take something home and build.

The client-side reality. Our panel on Optimizing Playback SDKs and Device Capabilities brought together Jonathan Colwell from FOX, Joshua Lamb from Red Bull Media House, and David Van der Voort “DVD” from Paramount. Three very different takes on the same daily chaos: supporting every device under the sun, with every usage pattern. That fragmentation is a grind everyone in playback knows intimately, and getting three operators to compare notes in the open was exactly the kind of session you cannot get anywhere else.

We kept the format balanced through all of it. Deep technical talks, real networking time, and interactive workshops where people could get their hands dirty. Room to think, argue, and go deeper.

The Q&A Is Still the Best Part

Same as last year, the Q&A sessions were electric. A technical crowd asked technical questions, and every speaker rose to meet them. Those exchanges are where a prepared talk turns into something better, where you get past the slides into the messy details that only come out when a peer in the audience pushes on exactly the right point.

That is the whole reason we do this in person. You cannot replicate that online.

Thank You

A conference only works because of the people who believe in it before it exists.

To our speakers, from Paramount, Netflix, CBS Interactive, FOX, Red Bull Media House, NVIDIA, Meta, AWS, Visionular, Bitmovin and every company that sent someone to share hard-won lessons: thank you for choosing substance over spin. You are the reason people fly in.

To JP Saibene and Nicolas Gonzalez from Uruguay, to Ali Begen from Tűrkiye and to everyone who traveled a long way to be in Seattle: thank you for bringing the curiosity and the generosity that makes this community what it is.

To the Momento crew who pulled this together, and in particular Hannah, Mike, Kelsey, Maéline and everyone who spent months making a two-day event look effortless: thank you, I’m incredibly grateful and lucky to be working with you all.

And to the people who showed up: this community was the event. Your questions, your stories, your willingness to say “actually, that broke for us too” is what makes this community worth gathering. The streaming landscape keeps moving fast, especially where AI meets live video. We were all here to discuss the future of video, one honest conversation at a time.

See you next year.

That should feel stranger than it does. When we typically think about inference, speed is a whole thing. Time to first token (TTFT), tail latency, and shaving milliseconds off the round trip so a human can stay in the flow are all you see in articles these days. Yet there I was running one of the most token-hungry things from my laptop, and I didn’t really care how long it took.

Meryem Arik, CEO of Doubleword, was able to articulate really well how I felt about it in her recent talk Inference for Async Agents in Production. She posed the simple question “who’s waiting?” that made it all click for me.

When nobody’s waiting, the cost of inference looks entirely different.

Schedule inference like nobody’s watching

In a synchronous system like chat, a human is directly in the flow. Make them wait without any feedback and they start clicking around wondering if anything is happening. In an async system, an agent might fan out, run for thirty minutes, and ping you when it’s done. Nobody is watching the screen. In many instances, it doesn’t really matter whether it comes back in 35 minutes or 45, because the user has gone off to do something else.

To me, this reads like a UX detail, but it drives almost every other decision in your stack.

“Who’s waiting?” is the right place to start. More specifically, how long is the work allowed to wait and what is blocked behind it? Even without a person staring at the screen, a call on an agent’s critical path still has a deadline. Async workflows just give us more room to decide what that deadline actually is.

A human waiting is the most expensive clock in the system. “Fast enough for a person” forces things like how you provision, how much you batch, which chips you buy, and how much idle capacity you keep on standby. Saying “I want an answer now” makes everything expensive. But if you take the human out of the loop, that clock loosens its grip, along with the price tag it was setting for everything else.

We reinvented batch processing

We’ve done this a time or two in the past with things like interactive database queries versus overnight analytics jobs. We prioritize the web request a user is staring at versus the batch job that runs at 2 a.m. The tech industry has been separating “someone is waiting” from “nobody is waiting” since before I was born. 😅

Inference is speed-running what databases and job queues learned decades ago, but in a good way. When a young field’s hardest problem is something the infrastructure world already recognizes, you can jump straight into borrowing patterns that are proven to work.

Meryem frames the core tension as the old trade-off triangle: latency, quality, throughput. You get to pick two of them.

Chat apps pick latency and quality. Async agents pick quality and throughput, and they let latency slide. A long-running agent mostly needs the smartest model it can get so it stays coherent across a two-hour piece of work while it manages subagents and juggles tasks. Delivering tokens at conversational speed doesn’t buy it anything.

Batching is really a scheduling problem

Batch-size math can be tricky. In one example from Meryem’s talk, a model running at batch size one and tuned for maximum interactivity costs roughly $4.65 per million tokens. Push enough concurrent work through the same model and tune for throughput instead, and that cost comes out closer to nine cents per million tokens.

In that example, the same weights and quality produce a 50x difference in cost. That makes a substantial difference to your unit economics.

But there’s a catch. Bigger batches produce their best economics only when you have enough compatible work waiting to keep them full. An empty slot in a batch is silicon you’re paying for that isn’t producing tokens. The hard part is keeping batches full while demand stays bursty and wildly unpredictable.

I’ve made this argument from the other direction before, that GPUs are the most expensive resource in tech and we use them badly. Batch size is the same thing. High utilization is paramount, and you win or lose it in the scheduler.

The queue is the product

I really liked how Meryem put it: most providers lose money on real-time serverless endpoints. It comes back to the “now” tax. To promise a human fast responses against lumpy traffic, you have to provision for the peak and then eat the idle troughs in between. That idle compute is money on fire.

If the work has room to wait, the batch queue becomes a powerful tool. You can push work into the low periods and backfill the troughs that are already being paid for. Spot instances become fair game for soaking up spare compute when the work can tolerate interruption or be checkpointed. SLA-aware routing lets you jump an urgent request ahead of the rest. You can even reorder the queue so requests that hit the same mixture-of-experts weights run back to back and skip reloading those experts every time.

Once you stop committing to “now”, the scheduler/router/whatever is holding the queue becomes the center of the system. It’s where the value collects, and it’s where everything is heading. As hardware and models commoditize, the advantage moves up into routing, placement, and orchestration, and whoever owns the “what runs next” decision owns the economics.

Start labeling the wait

I mark everything real-time out of habit. Every call gets treated as if a human is holding their breath on the other end, even when the “human” is an agent off doing six other things. That results in an unnecessary “now” tax on work that doesn’t need it.

Doubleword has categorized latency into real-time, async (about a minute), and batch (24 hours) buckets. The 50× cost gap we mentioned earlier comes down to which bucket you pull from, and many of us (myself included) reach straight for the expensive one by reflex.

One of the most boring-yet-useful things you can do immediately after reading this article is going through every call your agent makes one at a time and ask whether a human is waiting on each one. You might be surprised how few are. The background call fetching context isn’t waiting. The sub-agent grinding through step four of eleven isn’t either. The nightly summarization job definitely isn’t.

In the near future, we’ll have routers smart enough to infer a request’s latency tier from its context, dependencies, and deadline, so we don’t have to label everything by hand. Once we figure out how to do that efficiently, system costs drop dramatically because the scheduler can optimize around how long each piece of work is actually allowed to wait. Until then, start by giving every model call a realistic latency budget instead of letting the entire workload take on “now” by default.

Everything we’re talking about has been around in the infrastructure world for years. The same instinct drives shared caching at Momento: preserve expensive state, reuse work you have already paid for, and keep scarce resources productive instead of idle. Inference gives those old ideas a very large new price tag.

If you’re wrestling with any of this, we’d love to compare notes.

Happy coding!