A few weeks ago I handed off a pretty hefty refactor to Claude, closed my laptop, and went out to do some farm chores. I had no idea how long the job was going to take, and I didn’t care. I walked back in a couple hours later and it was done.
That should feel stranger than it does. When we typically think about inference, speed is a whole thing. Time to first token (TTFT), tail latency, and shaving milliseconds off the round trip so a human can stay in the flow are all you see in articles these days. Yet there I was running one of the most token-hungry things from my laptop, and I didn’t really care how long it took.
Meryem Arik, CEO of Doubleword, was able to articulate really well how I felt about it in her recent talk Inference for Async Agents in Production. She posed the simple question “who’s waiting?” that made it all click for me.
When nobody’s waiting, the cost of inference looks entirely different.
Schedule inference like nobody’s watching
In a synchronous system like chat, a human is directly in the flow. Make them wait without any feedback and they start clicking around wondering if anything is happening. In an async system, an agent might fan out, run for thirty minutes, and ping you when it’s done. Nobody is watching the screen. In many instances, it doesn’t really matter whether it comes back in 35 minutes or 45, because the user has gone off to do something else.
To me, this reads like a UX detail, but it drives almost every other decision in your stack.
“Who’s waiting?” is the right place to start. More specifically, how long is the work allowed to wait and what is blocked behind it? Even without a person staring at the screen, a call on an agent’s critical path still has a deadline. Async workflows just give us more room to decide what that deadline actually is.
A human waiting is the most expensive clock in the system. “Fast enough for a person” forces things like how you provision, how much you batch, which chips you buy, and how much idle capacity you keep on standby. Saying “I want an answer now” makes everything expensive. But if you take the human out of the loop, that clock loosens its grip, along with the price tag it was setting for everything else.
We reinvented batch processing
We’ve done this a time or two in the past with things like interactive database queries versus overnight analytics jobs. We prioritize the web request a user is staring at versus the batch job that runs at 2 a.m. The tech industry has been separating “someone is waiting” from “nobody is waiting” since before I was born. 😅
Inference is speed-running what databases and job queues learned decades ago, but in a good way. When a young field’s hardest problem is something the infrastructure world already recognizes, you can jump straight into borrowing patterns that are proven to work.
Meryem frames the core tension as the old trade-off triangle: latency, quality, throughput. You get to pick two of them.
Chat apps pick latency and quality. Async agents pick quality and throughput, and they let latency slide. A long-running agent mostly needs the smartest model it can get so it stays coherent across a two-hour piece of work while it manages subagents and juggles tasks. Delivering tokens at conversational speed doesn’t buy it anything.
Batching is really a scheduling problem
Batch-size math can be tricky. In one example from Meryem’s talk, a model running at batch size one and tuned for maximum interactivity costs roughly $4.65 per million tokens. Push enough concurrent work through the same model and tune for throughput instead, and that cost comes out closer to nine cents per million tokens.
In that example, the same weights and quality produce a 50x difference in cost. That makes a substantial difference to your unit economics.
But there’s a catch. Bigger batches produce their best economics only when you have enough compatible work waiting to keep them full. An empty slot in a batch is silicon you’re paying for that isn’t producing tokens. The hard part is keeping batches full while demand stays bursty and wildly unpredictable.
I’ve made this argument from the other direction before, that GPUs are the most expensive resource in tech and we use them badly. Batch size is the same thing. High utilization is paramount, and you win or lose it in the scheduler.
The queue is the product
I really liked how Meryem put it: most providers lose money on real-time serverless endpoints. It comes back to the “now” tax. To promise a human fast responses against lumpy traffic, you have to provision for the peak and then eat the idle troughs in between. That idle compute is money on fire.
If the work has room to wait, the batch queue becomes a powerful tool. You can push work into the low periods and backfill the troughs that are already being paid for. Spot instances become fair game for soaking up spare compute when the work can tolerate interruption or be checkpointed. SLA-aware routing lets you jump an urgent request ahead of the rest. You can even reorder the queue so requests that hit the same mixture-of-experts weights run back to back and skip reloading those experts every time.
Once you stop committing to “now”, the scheduler/router/whatever is holding the queue becomes the center of the system. It’s where the value collects, and it’s where everything is heading. As hardware and models commoditize, the advantage moves up into routing, placement, and orchestration, and whoever owns the “what runs next” decision owns the economics.
Start labeling the wait
I mark everything real-time out of habit. Every call gets treated as if a human is holding their breath on the other end, even when the “human” is an agent off doing six other things. That results in an unnecessary “now” tax on work that doesn’t need it.
Doubleword has categorized latency into real-time, async (about a minute), and batch (24 hours) buckets. The 50× cost gap we mentioned earlier comes down to which bucket you pull from, and many of us (myself included) reach straight for the expensive one by reflex.
One of the most boring-yet-useful things you can do immediately after reading this article is going through every call your agent makes one at a time and ask whether a human is waiting on each one. You might be surprised how few are. The background call fetching context isn’t waiting. The sub-agent grinding through step four of eleven isn’t either. The nightly summarization job definitely isn’t.
In the near future, we’ll have routers smart enough to infer a request’s latency tier from its context, dependencies, and deadline, so we don’t have to label everything by hand. Once we figure out how to do that efficiently, system costs drop dramatically because the scheduler can optimize around how long each piece of work is actually allowed to wait. Until then, start by giving every model call a realistic latency budget instead of letting the entire workload take on “now” by default.
Everything we’re talking about has been around in the infrastructure world for years. The same instinct drives shared caching at Momento: preserve expensive state, reuse work you have already paid for, and keep scarce resources productive instead of idle. Inference gives those old ideas a very large new price tag.
If you’re wrestling with any of this, we’d love to compare notes.
Happy coding!
