You know how back in the day we used to be messing around at our desk and a teammate asks “What are you doing?” To which our answer was always “I’m just waiting for this to finish compiling.” In 2026, we have a different excuse: “I’m waiting for Claude to respond.”
We say it jokingly, but it’s true. Inference isn’t well-known for being fast. Powerful, sure. But not fast.
We’ve done a lot of experimentation to make it faster. Parallel fetch, zero-copy reads, batched metadata lookups, the whole retrieval path. But there was one particularly intriguing experiment that has a lot of upside, but some huge trade-offs. What if we stored the KV cache remotely instead of locally? There’s just one problem. The KV cache is huge. Transporting it over a network is a non-starter. Right?
We know how it works. GPU memory is fastest, then host DRAM, then the NVMe bolted to the same box, then, if you really have to, go across the network. You go remote when you need to share, but you pay for it in latency.
I’m not onto a novel idea though. Our industry is all over it right now. NVIDIA has an offloading tier in Dynamo. Mooncake is building a store. Xinnor published a piece in June arguing that Lustre with a client-side cache can hold its own against local NVMe. Khawaja Shams made the case that once you disaggregate prefill and decode, the cache has to travel between machines whether you like it or not.
Clearly there’s merit to it. But I didn’t see much about the cost of the tradeoff. How much of a latency hit do you really take storing your KV cache in something like Valkey, and is skipping prefill still worth it after you’ve paid it?
So I took it upon myself to run the numbers.
The setup
The benchmark uses three machines: two identical GPU nodes running vLLM, and a Valkey box sitting one 10 Gbps hop away from both of them.
gpu-a does all the computing for prefill. The DRAM and NVMe tiers live inside it, and Valkey sits across the network. gpu-b is used to determine what happens when a cached request lands on a machine that did not perform the prefill.
The workload is thirty realistically generated legal and medical documents, all trimmed to exactly 10,000 tokens. Every document goes to vLLM twice. The first pass does the prefill and stores the KV cache. The second pass sends the same prompt, and I time how long the first token takes to come back (TTFT). The only thing that changes between runs is where the KV cache lives.
Everything runs vLLM 0.28.0 with LMCache 0.5.4, valkey-glide-sync 2.5.1, and Valkey 9.1.1, serving Qwen2.5-7B-Instruct-AWQ on NVIDIA L4s.
To get accurate benchmark results I had to turn off vLLM prefix caching so the GPU wouldn’t serve every repeat request. I also had to clear the disk tier and drop the page cache before every single run, because Linux keeps recently written disk data in memory. If I didn’t do that, the “NVMe” reads would come out of RAM and I wouldn’t have actually tested the disk speed.
The goal was to beat cold TTFT, which falls between 3.1 and 3.3 seconds. I ran the entire benchmark twice on separate instances a day apart, and got reproducible numbers within 2%. So I’m feeling good about what I found. Let’s take a peek and see how the network did.
Keeping everything on one machine
For this first test, gpu-a does all the work. It computes the KV, stores it in whichever tier is being tested, and reads it back on the second pass. Even in the Valkey run it’s gpu-a writing and reading its own cache, just over a wire instead of a local bus.
| Tier | Cached TTFT | Speedup | Reuse |
|---|---|---|---|
| DRAM (8 GB) | 3345 ms | 0.95x | 0/30 |
| Local NVMe | 1521 ms | 2.18x | 30/30 |
| Valkey (remote) | 557 ms | 5.98x | 30/30 |
You’re reading that right. The network was the fastest. And by quite a large margin, at that. It beat the local NVMe by 2.7x on the same machine. The disk is physically inside the box with no switch in the way, and the cache was still quicker to reach from across the network.
(Ignore the DRAM row for now. Something interesting happened there, and we’ll talk about it later.)
At first I thought I broke something. Maybe I accidentally put the “disk” tier on the EBS root volume instead of the instance store? But lsblk showed the tier on nvme0n1 (the 250 GB instance store), and df showed 16 GiB written to it during the run, which is about right for 300,000 tokens of KV.
After doing some math, it started making sense.
Each document’s cached KV is 39 chunks of 256 tokens at 57,344 bytes per token, roughly 572 MB (39 x 256 x 57,344). While I was sanity-checking the harness, I ran the DRAM tier with two documents (so everything fit in memory), and hits came back in about 82 ms. A DRAM hit barely transfers anything, which means 82 ms is basically the overhead cost. So any time beyond 82 ms is what I would consider to be transfer time.
That means Valkey’s 557 ms is really about 473 ms of transfer. 572 MB in 473 ms is 1.2 GB/s, which is a 10 Gbps NIC running flat out. Compare that to the disk, which moved 572 MB in ~1,438 ms. That’s 400 MB/s, which is odd, because NVMe should be able to read north of 2 GB/s.
So the remote tier is bound by the network, and the local tier is bound by its own software. Clearly a lot of work went into LMCache’s Valkey connector this year. It pulls with eight parallel workers straight into caller-owned buffers. The local disk path hasn’t gotten the same attention yet, and 400 MB/s from an NVMe says there’s plenty of opportunity. I imagine it won’t stay this way for long.
Anyway, from these results we see that putting the KV cache on the network didn’t have the latency penalty I thought it would. It ran as fast as physics allows 🔥. Even if the disk path gets some love next month and starts doing 1.5 GB/s, using the network is still a solid choice. Plus, Valkey has room to go even faster with a bigger NIC.
There is one thing I can’t explain. That 2.18x speedup for the disk is a median, and the per-document numbers show something weird. The first document requested in the cached pass comes back around 11x faster. The second comes back around 3.5x. The other 28 are all right at 2.1x. Oddly enough, both runs were basically the same:
| Position in the cached pass | Run 1 | Run 2 |
|---|---|---|
| 1st document | 11.42x | 10.84x |
| 2nd document | 3.65x | 3.28x |
| the other 28 | ~2.1x | ~2.1x |
Something is warm for the first request or two, then stops. I don’t know what it is, and I’m not going to make up a reason that sounds good. Whatever it’s doing, it makes the disk look better than the median I’m quoting, so it isn’t driving my conclusion.
Let’s talk about DRAM
In my results table above, the DRAM row says it had a 0.95x “speedup”. I know what you’re thinking: I misconfigured it. I didn’t. Remember that two-document sanity check where hits came back in 82 ms? That was this tier with the same config. The TTFT was 3122.8 ms cold and 82.5 ms cached, which is a 37.93x speedup 🤯.
DRAM isn’t slow. DRAM is the fastest thing in the entire stack by a mile. It just… runs out.
Thirty documents is 300,000 tokens of KV. At 57,344 bytes per token, that’s 16 GiB. I gave the tier 8 GB, and the entire host only has 16. The box simply didn’t have the memory, and it evicted everything before I could even run the second pass.
Not only that, but every tier that served nothing came back 3% to 5% slower than cold, on at least 29 of the 30 documents. A lookup that misses is still a lookup, and a store that’s evicting still costs a store. In this instance, the undersized cache was basically a tax.
And thirty documents is nothing when compared to a real production corpus. If your working set is two documents, use DRAM and stop reading this. If it’s thirty or more, skip the DRAM for something with more capacity.
Expanding to two machines
Everything we’ve covered so far has been a single machine writing and then reading its own cache. But the big question is what happens when a request is routed somewhere else, because it will in production. So I ran the cold pass on gpu-a and the cached pass on gpu-b (the one that didn’t do the prefill).
| Tier | One node | Second node | Reuse on second node |
|---|---|---|---|
| DRAM | 0.95x | 0.96x | 0/30 |
| Local NVMe | 2.18x | 0.95x | 0/30 |
| Valkey | 5.98x | 5.97x | 30/30 |
Unsurprisingly, NVMe went down to zero reuse and 3.2 second prefill on gpu-b because the KV didn’t exist there and there was nothing to access. It makes NVMe seem a lot less exciting once you start using it in a distributed environment.
That said, Valkey went from 5.98x to 5.97x. Reading from a machine that did not compute the KV cost nothing measurable. The worst document slipped from 5.55x to 3.92x, and the medians are effectively the same. The cache doesn’t belong to either node. And that was the whole point.
My takeaways
If your working set fits in host DRAM, use that. A cache hit came back 37x faster than recomputing the prefill. Just be honest with yourself about capacity. Measure your corpus in tokens and multiply by your model’s per-token KV. Don’t eyeball it.
Once it stops fitting, local disk gets you 2.18x on the machine that wrote it, but nothing anywhere else. On a single node that’s a considerable boost. But across a fleet, every entry is isolated to the box that produced it, and your hit rate becomes the odds of getting routed back to the same machine.
The network tier isn’t free on the way in. Cold TTFT ran about 110 to 150 ms slower with Valkey than with DRAM, which is a 3% to 4% cost for storing over the wire instead of in memory. So the tradeoff is that you pay 110-150ms per cache fill in exchange for about 2.7 seconds back on every hit after it.
The most surprising result was that the shared tier was already the faster one here. On this hardware, with this model, at this corpus size, putting the cache on another machine made it quicker. I went into this assuming I’d be arguing that sharing is worth a latency cost. I am happy to have been wrong on that one. Skipping prefill is worth what it costs (and then some).
Your KV cache belongs on the network. Not because it has to be shared (though it does). Because that’s where it was fastest.
Try it yourself
The harness, the 30-document corpus, the tier configs and the infrastructure scripts are all in valkey-kvcache-bench. One command provisions all three hosts, runs the matrix, pulls the results down and terminates everything:
cd infra
.\Run-TierBench.ps1 -AwsProfile <profile> -TeardownIt costs about $2.14 an hour and the full benchmark run takes under an hour.
One warning if you try this yourself - the first time I ran the two-machine test, every tier came back 0/30 (including Valkey). It turns out LMCache’s default cache key hash is Python’s hash(), which gets a different random salt in every process. So gpu-a and gpu-b were generating different keys for identical tokens, which made cross-node reuse impossible. LMCache even warns you at startup with a Using builtin hash without PYTHONHASHSEED set message, and I had completely missed it. But you can fix it by setting this config value:
yaml pre_caching_hash_algorithm: sha256_cbor_64bit
If your shared KV cache ever shows a zero hit rate across nodes, it’s probably that.
Anyway, try it out and if you see something different, please feel free to reach out so we can compare.
Happy coding!
