Start building with Momento Inference!

Choose models and fine-tune them for your workflows. Serve them from the region you pick, on a stack built for speed. Then, evaluate your optimized models against real workloads and control how they're used in production.

Optimize your inference today ->

Fast, Shared Context Memory for Inference with Valkey

Momento unveils the Large Object Module for Valkey, with zero-copy RDMA and tiered NVMe storage to deliver a fast, shared KV cache for vLLM and SGLang via LMCache.

Daniela Miao headshot Daniela Miao Tony Valderrama Tony Valderrama
Two GPU cards connect by green cables to a shared cache with memory modules above solid-state drives.

The modern AI inference stack has a memory problem. In a fully-utilized GPU cluster, each worker constantly processes new inputs to produce attention state, the raw tensors that drive decoding of output tokens. Once output completes, that data is discarded to make room for the next wave of requests.

Most workloads construct inputs via turn-by-turn accumulation, repeatedly feeding the same long sequences of text to inference. When attention state is lost, workers must waste expensive GPU cycles reprocessing the same inputs to produce the same attention state. Yet, workers generate so much data that even local DRAM and NVMe aren’t sufficient to hold it all.

Recognizing the value of fully-reusable attention blocks, inference engines like vLLM and SGLang have begun pushing this data into a durable, shared store: an emerging AI infrastructure component called context memory.

Valkey is the most widely-deployed free and open-source cache, and a common tool for engineers building efficient, low-latency systems. Now, Momento and AWS have co-developed Valkey’s Large Object module, which introduces innovative new features like RDMA network transfer and tiered NVMe storage, positioning it as a powerful and familiar platform for context memory.

The Context Memory Storage Hierarchy

As storage moves further from the GPU, it becomes cheaper, denser, and slower. Leading inference frameworks such as NVIDIA Dynamo, vLLM, LMCache, and SGLang’s HiCache define four primary tiers in the context memory storage hierarchy:

  • G1. GPU HBM: attention state in active use
  • G2. local DRAM: staging and buffering of active attention state
  • G3. local SSD: hot state reused over short timescales
  • G4. shared storage: shared state accessed by many workers

The G4 shared storage tier typically means accelerated object storage like S3 Express One Zone, WEKA, or MinIO. Recently, the line between G3 and G4 has blurred with new architectures like peer-to-peer sharing (Mooncake) or network-attached NVMe (NVIDIA Bluefield).

Median request latency for Qwen3-8B-FP8 by input token count, on a logarithmic scale in milliseconds. Valkey RDMA tracks close to local NVMe. Local DRAM and GPU HBM are faster, and cold prefill is slowest.

Valkey slots firmly into this space between G3 and G4, providing ultra-low-latency access to a large, durable, shared pool of DRAM and NVMe. In our preliminary tests, a remote Valkey tier was as fast as peer-to-peer transfer among workers, and barely 5% slower than a worker’s local NVMe cache.

Operationalizing Durable Context Memory

Local caching, peer-to-peer transfer, and Valkey all share similar performance characteristics. But, they each enable a different lifecycle for context memory. Local caching constrains attention blocks to individual nodes, while peer-to-peer transfer binds them to a single cluster. Only a remote store allows context memory to survive beyond the lifespan of individual inference workers and clusters.

This separation of storage from compute represents the natural evolution of inference towards a robust, cloud-native architecture. Decoupling resources into distinct layers improves reliability and unlocks more effective capacity management. Now, Platform teams can independently scale components in response to fluctuating workloads. GPU clusters can be dynamically reassigned a different model or configuration without triggering data loss and cold starts.

In addition, a platform team’s operational efficiency improves when they reuse familiar products. Valkey and Redis are the most widely used caching technology in the world. Adopting them as a distributed kv cache equips operators with a hardened, high-performance tool.

There are so many unknown and esoteric challenges with inference at scale. Shared context memory doesn’t have to be one of them.

Three stages of KV cache offloading: local storage, peer-to-peer sharing, and remote persistent storage.

The Economics of Reusable Context Memory

Operational simplicity is only half the case for a remote store. Every block of attention state is GPU work already paid for, and that cost is recovered only if the block remains available when a matching request arrives. Reuse can occur seconds later for a tool call, an hour later when an agent resumes, or days later with a shared system prompt.

Retaining that block incurs a small hourly cost. The cheaper the storage, the longer the period until it exceeds the cost of GPU time for recomputation. DRAM is justified when reuse occurs for minutes to hours, NVMe for days, and object storage for longer still. And because the cache is shared, each block has more chances to earn back its cost.

As models evolve, the balance tips further toward retention. Longer contexts increase the cost of prefill computation. Meanwhile, model architectures deploy new and innovative methods to compress attention state. Each stored byte represents a larger amount of compute, making it worth keeping longer.

Capturing that value at scale requires a datastore that can move multi-megabyte blocks quickly and tier them cheaply.

Evolving Valkey for Context Memory

We built the Large Object module because our customers running inference kept hitting the same wall: multi-megabyte attention blocks that were too expensive to hold in DRAM and too slow to copy over the network. Valkey already had the speed and operational familiarity they needed, so together with AWS, we set out to give it cheap tiered storage and a direct path into GPU host memory.

Valkey is backed by a vibrant community of contributors under the governance of the Linux Foundation, with a particular focus on efficiency and performance at scale. Now, the Valkey Large Object module introduces several capabilities that make Valkey the ideal platform for fast, shared context memory.

The module stores multi-megabyte blocks of attention state across DRAM and NVMe, dramatically decreasing the cost of storage. We added a zero-copy RDMA path that quickly moves these blocks between the server and an inference workers’ host memory. We also shipped an integration with LMCache that makes it easy to use Valkey with vLLM and SGLang.

Together, these let you keep more context, for longer, at lower cost, and get more useful work from the GPUs you already have. To get started, try the Large Object module. Talk to Momento about deploying to your environment with the Momento Valkey Operator, or Momento Cache provides a fully managed service.

Keep Reading