At Amazon, I ran a fleet of thousands of EC2 instances across multiple regions. Many of my lessons about caching came from that experience, and most were learned the hard way.
I helped build a personalized ad insertion service that handled millions of requests per second (RPS). We had a tight latency budget for giving each viewer a personalized experience. If the service took too long, the video would buffer and erode that experience.
The workload was also spiky and somewhat unpredictable. You never knew when a football game would get interesting and more fans would start tuning in. Our first thought was to add a cache. As an engineering leader now at a caching company, my response may puzzle you:
Are you sure you want to add a cache?
My own instinct while working on Amazon-scale services was often to reach for a cache as the default performance tool. Time and again, my team and the teams whose designs I reviewed encountered the same pitfalls.
A cache starts with planning
Adding a cache takes time. Our teams had to provision Amazon ElastiCache clusters, size them, benchmark them, and instrument them. That planning could easily consume a sprint or more before we even began integrating the cache.
Instrumentation is frequently overlooked when teams rush into caching. If implemented incorrectly, a cache may slow you down or increase your error rates. Without instrumentation, you could keep assuming that the cache is helping when it is doing the opposite.
Sizing and benchmarking are just as important. Overprovision, and you waste money. Underprovision, and more screens buffer while the cache becomes slower and more error-prone.
Here is my obvious plug: Momento Cache can make a cache available in seconds without requiring you to size it. That can save developers time on setup and instrumentation. But making a cache easier to use does not mean you should add one blindly. The decision still requires deliberation.
Small hit-rate changes can create large backend spikes
We spend a lot of time lowering cache miss rates (CMR) for our customers, but very low miss rates create their own risk. The DynamoDB paper discusses a cache hit rate (CHR) of 99.75%, which means a CMR of only 0.25%.
If the CHR drops by 0.75 percentage points to 99%, the CMR rises from 0.25% to 1%. That apparently small change creates four times as many misses and sends four times as much load to the backend database. This bimodal behavior can be catastrophic during a cold start. The Roblox outage postmortem provides a real-world example.
Plan how stale data leaves the cache
As soon as you add a cache, you are likely compromising your system’s consistency semantics. You may receive old data without realizing it, and the resulting behavior can be hard to debug. A response does not match the database, and eventually you discover that one small part came from a stale value you did not realize was cached.
Before implementation, answer these questions:
- How will you update values in the cache?
- Will you use a time to live (TTL), or will another component decide when to update cached data?
- Will readers and writers trigger updates, or will a background process refresh the data? What happens if that process stops?
- If you cannot fetch a new value, will you return a stale one? If so, for how long?
- When an item is missing from the cache, will you cache it before or after returning success?
- If an item is missing and the downstream service returns an error, will you cache the error response?
Each answer adds complexity. The cost you were trying to reduce can return as the cost of maintaining spaghetti code. Having seen this pattern many times, I insist on explicit TTLs for every cached item, even when engineers say the data will not change.
Choose where the cache lives
Local caches distribute state across servers
We often use a cache to hide performance and scalability problems. This approach has two major flaws.
First, those problems eventually return. Local caching works well in a small web-server fleet, but the backend sees a different load as the fleet grows. You have to understand what doubling the fleet will do to that backend.
Second, caches add complexity to your code. Storing and retrieving one value is simple. Evicting stale data within one process can also be straightforward because there are no external dependencies. Across a fleet, however, local caching leaves stale values in more places. Clearing a poison pill could require you to restart the entire web-server fleet.
Local caches also add more cold starts. Each deployment or scaling event adds web servers whose caches need to warm, causing a latency hit and more variability. Pinpointing that behavior requires deep, per-server metrics on your dashboards.
A separate cache fleet adds a network boundary
A separate, look-aside cache fleet has meaningful advantages. Web-server deployments do not erode its hit rate or make the shared cache cold. Sharing the cache also builds a common working set across the web servers, which can improve hit rates. This is particularly useful when AWS Lambda is the web-server layer because each Lambda handles only one request at a time.
A separate fleet can also use memory more efficiently. Instead of reserving cache memory on every web server, you can size one caching fleet and reason about it as a separate concern.
Those benefits come with operational work. You must monitor, manage, and scale the cache fleet. You are accountable for its security patches or maintenance windows. Scaling up, scaling down, and replacing nodes during deployments can produce cold caches, so the scaling and deployment story matters.
Local caches reset with each deployment, so you do not have to worry about cached items that are incompatible across web-server versions. A separate cache persists across those deployments. If a deployment changes the cached data format, stale schemas can become poison pills that are difficult to clear.
A separate fleet also adds a remote procedure call (RPC) to the critical request path. That is more involved than a local in-memory lookup. You must tune clients for connection pools, timeouts, retries, and failover.
For example, caching clients typically set timeouts at 60 seconds, meaningfully longer than the time an Amazon DynamoDB query takes to return. That timeout may be appropriate, but it has implications. During maintenance windows, connections may hang until they time out and create backpressure on web-server threads. Tinder’s account of using Amazon ElastiCache explains this problem well.
Clients must handle those conditions correctly, and the original cost goal still matters. Supporting both a caching fleet and the team that manages it can make the overall system more expensive.
Make the tradeoff explicit
Caching is one of the best tools for solving scalability problems. It can help with hot keys, throttling, and the cost of underlying databases. It also adds a new system with its own performance, consistency, failure, and cost behavior.
Before adding one, write down the metric it must improve, how you will instrument and size it, how stale data will leave it, and whether local or separate caching fits your deployment model. If those answers justify the operational complexity, add the cache deliberately. If they do not, keep investigating the underlying performance or scalability problem.
