A cache can look healthy while the load on its backend doubles. If its cache hit rate drops from 99% to 98%, the change looks like a trivial percentage point. Frame the same event as cache miss rate, however, and misses have risen 100%, from 1% to 2%. Twice as many requests now reach the backend.
That is the biggest miss in cache hit rate (CHR): the metric can hide the operational impact of a cold or disrupted cache. Tracking cache miss rate (CMR) alongside it makes that impact harder to overlook.
Real-world workloads often have highly skewed data access. Production caches can therefore achieve hit rates above 90% and sometimes above 99.9%. A DynamoDB paper, for example, cites a 99.75% hit rate for its metadata cache. It also describes the bimodal behavior that follows when the cache gets cold. Even the distributed systems experts on the DynamoDB team have experienced the consequences of a falling hit rate.
CMR spikes correlate directly with load spikes, so they quickly put the database impact into context. CHR alone requires arithmetic to expose that load:
Load = CURRENT_CHR / (1 – NOMINAL_CHR)
The calculation is not difficult, but it is enough friction for us to miss the consequences of a CMR spike when we see only a CHR dip. Monitoring CMR makes seemingly minor issues during maintenance windows or scaling events easier to evaluate.
Most operational dashboards already teach us to look for spikes in latency, errors, and other signals. Some metrics, such as request rate, warrant alarms on dips. A cold cache fits the more familiar pattern: its miss rate spikes.
Why cache miss rate spikes
Restarts introduce cold nodes
Deployments and failed nodes can both cause restarts. In either case, a new node enters the system without data: a cold cache.
On AWS ElastiCache, deployments are limited to maintenance windows selected by users, while failed nodes can occur periodically as EC2 instances fail. Even with replication and a multi-AZ setup, node failures and maintenance windows affect CMR. The cache may also take meaningfully longer to warm than the maintenance window, extending the impact on CMR.
Rehashing redistributes keys
Rehashing typically occurs when you scale a cache in or out. Adding or removing a node changes the topology and redistributes its keys.
Early in a rehash, cache clients may disagree about where a particular key belongs. If client A sets a key on node X but client B reads it from node Y, client B gets a miss.
After that early phase, some keys may not yet exist at their new location. Clients continue to get misses until one sets the key there. Several clients may observe that the item is missing and issue redundant set commands. Those commands increase backend load while eroding cache performance.
How to reduce cache miss rate spikes
Replicate the data
Replication creates multiple copies of the data. It increases the potential read throughput for a key and creates multiple places where that key can be found.
Consider a system with three replicas and requests distributed evenly among them. A single node failure would send only 33% of requests to a cold node, where they would return a miss.
Warm a node before serving reads
Cache warming lets a new node observe some set commands before it joins the read path. Facebook’s mcrouter, for example, warms new nodes as they come online. Pinterest uses a similar pattern for its self-managed cache nodes on EC2.
Rather than abruptly replacing a node with an empty one, cache warming replicates a portion of traffic to the new node for a limited period. The node can build a meaningful set of keys before it serves reads. Popular keys often matter most to customers, and their popularity also makes them more likely to reach the new node. Warming does not eliminate CMR spikes, but in most cases, it meaningfully dilutes them.
Cache teams should own avoidable misses
Customers and providers expect some misses during normal operation. Without clear specifications for miss scenarios, however, ownership becomes blurred. Cache teams can largely mitigate misses caused by node failures or deployments, topology changes during scale-in or scale-out, and memory-pressure-driven eviction.
Caching services do not always invest the engineering effort required to limit CMR spikes during deployment and scaling. Instead, providers may blame customers for every miss: “If a customer did not give me the key, how can I be expected to return it?”
ElastiCache autoscaling guidance goes as far as encouraging customers to disable scale-in to reduce the CMR impact of repeated scale-in and scale-out:
Disable Scale-In – Auto scaling on Target Tracking is best suited for clusters with gradual increase/decrease of workloads as spikes/dip in metrics can trigger consecutive scale-out/in oscillations. In order to avoid such oscillations, you can start with scale-in disabled and later you can always manually scale-in to your need.
Reducing scale events improves CMR, but it can leave a wastefully overprovisioned cluster below 10% utilization. Customers ultimately pay the higher bill.
How Momento approaches cache miss rate
CMR is a fundamental focus at Momento. A spike in CMR means greater load on our customers’ databases. By minimizing those spikes, we can give customers true elasticity and continuous availability without maintenance windows. This insight helped drive Momento’s architecture.
Share topology changes through a low-latency messaging bus
Momento’s proxy fleet uses a low-latency, gRPC-backed messaging bus. The fleet can quickly learn about changes in cache topology, which limits how long nodes disagree about where a key belongs.
Keep server-side topology out of the SDK
Modern cache clients often track cache topology. Momento removes that leaky abstraction, which lets us build simpler SDKs that do not notice server-side state changes.
Warm nodes during deployment and scaling
Momento brings nodes into service after they have warmed. Along with abstracting topology changes, this eliminates maintenance windows and lets us deploy continuously without affecting CMR.
The next time your team puts CHR on a dashboard or discusses it in a design review, reframe the conversation around CMR. If a caching vendor offers “elasticity” or “autoscaling,” ask what happens to miss rate during deployments, scale-out, and scale-in. Then explore Momento Cache to see how Momento approaches those events.
