At Momento, we’ve built a blazing-fast serverless cache. Making our cache simple and delightful is a crucial part of our mission. We strive to be shockingly simple.
What does that mean for you as a developer? On the server side, you shouldn’t have to worry about managing clusters, scaling, replication, or any of the other work that consumes engineering time. You should be able to focus on what differentiates your business. Create a cache and go. Let us worry about the rest.
But that’s only half the story. If you need a cache, you have code that needs to read and write values. That code needs a client library that knows how to talk to the cache service.
Why cache clients are a first-class concern
Most database and cache services provide excellent server-side capabilities, but leave the client side as an exercise for the reader. You might have to sift through several client libraries on GitHub and assess their quality by asking, “Which one has more stars?”
Even after you choose a library, you might spend valuable time tuning connection pools and other settings for your environment. Or worse, you might have to debug an inappropriate configuration after an outage.
Momento Cache takes a different approach. We want your client-side experience to be as delightful as your server-side experience, so we provide official Momento client libraries for most programming languages.
We think of these libraries as our “front door.” They’re often the first place where you form an opinion about Momento. It’s on us to make sure they live up to the same high standards as our server-side product. Our cache clients are a first-class concern.
This post is the first in a series about how we think about and tune our clients. The second installment explains how we’ve tuned the JavaScript cache client, and an upcoming entry will dive into Python.
The irony of writing this series is that we hope you never need to think about the details we cover. We’re doing this work so you don’t have to. If you came here to learn what you need to know before ramping up with one of our cache clients, you’re done. We’ve got you.
If you’re curious about how we polish the SDKs to make sure they shine for you, come along, dear reader.
Timeouts, retries, and backoff matter most on bad days
Timeouts, retries, and backoff strategies are commonly overlooked client-library configurations. These parameters can seem inconsequential until they contribute to a catastrophic outage.
Suppose your cache client has a five-second default timeout for connecting to the cache server. You might not notice a problem for weeks or months while the server is healthy. Then the cache becomes unavailable. Every request now waits five seconds before your code falls back to the underlying data source. This sudden bottleneck can wreak havoc on your entire system.
A poorly configured retry strategy can also produce false cache misses and put unnecessary load on the database server. A poorly configured backoff strategy can create a retry storm, overload your servers, and make a bad situation worse.
Development and production need different defaults
Connection pools, timeouts, and retries usually don’t have one set of values that works well in both development and production.
A development environment often has higher network latency and possibly less network stability. Relaxed timeout and retry values can prevent interruptions from request timeouts that don’t matter during development.
In production, your application servers might issue cache requests with low latency to your cloud provider of choice. You’re likely to want more aggressive timeout and retry strategies so your application can meet its own service-level agreements.
How can a client give you the best of both worlds?
Manual tuning
Some client libraries expose configuration as a set of keys and values that you need to tune. You might say, “I want a timeout of 500 milliseconds, up to 5 retries, a backoff factor of 2, and up to 5 milliseconds of jitter.” Then you have to repeat the exercise for every environment you run.
We will support tuning Momento clients this way if you prefer it or if an unusual environment requires a unique configuration. In most cases, though, we don’t want you to have to do all that work.
Prebuilt static configurations
We are currently finishing a round of performance-tuning work to define prebuilt, static configuration bundles for each supported programming language. Each client library will soon include ready-to-use configurations such as DevEnvironmentConfig and ProdEnvironmentConfig. They should provide a strong starting point for the 90% case in each type of environment.
We’re doing the chaos testing to make sure these prebuilt configurations fit their intended environments. You only need to read the brief descriptions, decide which configuration most closely represents your environment, and start running.
Adaptive configurations without manual tuning
Choosing among a handful of prebuilt static configurations is much less work than tuning every setting by hand. Even then, you still have to identify which configuration best matches your environment.
Imagine instead an adaptive configuration that keeps a sparse histogram of client-side latencies. It could adjust timeout and retry values dynamically based on the real-time latencies it observes in your environment.
We don’t have an ETA for adaptive configurations, but they’re another example of how we think about making the client-library experience shockingly simple. This is hard to get right, and we’re keen to make it production-ready. Stay tuned for updates.
Let the client do the hard work
You shouldn’t have to think about connection pools, timeouts, retries, gRPC configuration, code optimization, or maximizing single-core CPU performance. Continue with our deep dive into tuning the JavaScript SDK to see how we approach that work, and stay tuned for the Python, C#, and Java installments.
