When a system runs out of headroom, you've got two options: buy a bigger box, or run more boxes. We settled that question for the whole internet a long time ago; the web runs on adding cheap machines, not one unaffordable one. And yet the moment someone says "I need more from my AI," the default answer is still the first option. A bigger model. A bigger GPU. More of the scarcest memory money can buy.
I keep bumping into research that points the other way, and none of it frames the point I think it's making. A routing trick here. A paper about coordinating agents there. A tiny model released out of nowhere. Put them together and they're all describing the same thing: horizontal scaling for inference. And horizontal scaling is the only thing that has ever made computing cheap.
Let me be clear about the claim so we don't drift: I think the reason AI keeps getting expensive is that we keep buying a bigger computer instead of running more of the small ones we already have; every piece of the "run more of them" answer already exists. This post is me connecting the dots.
The people already building it
Andrej Karpathy got closest to saying it out loud. His "cognitive core" argument is that the model most of us actually need is small, a few billion parameters or fewer, one that reasons well, uses tools, and looks facts up instead of memorizing the entire internet. On Dwarkesh Patel's podcast, The Lunar Society, Karpathy made the same point: the field is already moving that way; top models grew for years and are now shrinking, and nearly every good small model is a distillation, trained to copy a bigger one.
The version of that idea I actually care about: if skill and memorized facts can be pulled apart, then a small model tuned for one narrow job doesn't need a frontier model's extra hundred billion parameters, because most of those parameters carry knowledge the job never touches.
And Karpathy isn't just talking about it. He built the assembly line. Autoresearch strips his training stack down to one GPU, one file, one number to improve; then it hands the whole thing to a coding agent that proposes a change, runs a five-minute training run, keeps the change only if the score goes up, and repeats overnight with no human in the loop. His long run stacked twenty improvements across seven hundred experiments, on training code he'd already spent years tuning by hand; that run even caught a missing scalar in the attention math that two decades of manual work never noticed.
The part that matters here is the supply side. If a small model is something an agent can measurably improve while you sleep, then a fleet of specialists stops being a hand-made thing and becomes something you grow. A way less glamorous version of the same loop runs my own inference benchmarks, one agent on a schedule and another shipping fixes behind quality gates; the lesson so far is about the gates. Six of its performance fixes once quietly broke retrieval, and the failing test was the only thing that noticed. A different project of mine taught me the other half of that lesson: 716 tests passing, typecheck clean, and six real defects that showed up only the first time the thing ran against a real Slack conversation. Gates catch what they were written to catch; they are not evidence that the system works.
The pieces are all sitting there
Take them one at a time and none of this is exotic. That's kind of the point.
The router is just a load balancer. The vLLM Semantic Router puts a small trained classifier in front of a pool of models and sends each request where it belongs: easy questions to the fast path, hard ones to the reasoning model. The insight I keep coming back to is that routing is a sorting problem, and a sorter can be hundreds of times smaller than the models it sorts between. A good receptionist doesn't need to be the smartest person in the building; they need to know which door to send you to. Caveat: this piece is less solved than it sounds. In production the router shares a GPU with the models it routes, and making it cheap for long inputs is an open research problem on its own.
The orchestrator can be learned, and it can be small. Sakana's Fugu puts a ~0.6B coordinator and a 7B planner in charge of a pool of worker models, handing out Thinker, Worker, and Verifier roles instead of a hand-wired pipeline. Two things you shouldn't over-read: the workers in that pool are frontier models, so this is evidence for small coordinators, not small workers. And "rivals frontier models" is generous; Fugu Ultra scores 73.7 on SWE-Bench Pro where Claude Fable 5 scores 86 on Sakana's own table. What it does prove cleanly is where the intelligence needs to live: in the planning and the checking, which are genuinely open-ended, and definitely not in the sorting.
Teams work if you respect the cache. Nous Research's Hermes agent shipped a Mixture-of-Agents mode where an aggregator reads several models' answers and combines them into something better, and the team beats its own strongest member on their benchmark. The implementation detail is the telling bit: the other models' answers get appended at the very end of the prompt, below a stable prefix that never changes, so the prompt cache never breaks and the only cost of the team is the extra calls. The quiet lesson is that the KV cache is the real currency of inference, and every serious multi-model design ends up organized around not wasting it.
Teams aren't free, and now we've got numbers. A Google Research study tested 180 agent-system configurations. Adding agents helps when a task splits into parallel parts; a centralized team beat a single agent by roughly 81%. It actively hurts when a task is sequential; every multi-agent setup they tested made results 39 to 70% worse, because the agents talking to each other ate the budget the reasoning needed. Error amplification is the scarier one: parallel agents with nobody checking amplified errors about 17x, while adding an orchestrator that reviews the work cut it to about 4x. The lesson lands on the same spot every time: the value lives in coordination and checking, and a team without them is worse than no team at all.
The fix for the worst failure is infrastructure, not a smarter model. From Spark to Fire studied error cascades: chains where one agent's mistake becomes another agent's input and multiplies, until a group vote settles on the wrong answer as agreed truth. Their answer is provenance. Give every message a link to its parents, so when a detector flags one message as bad, you can quarantine everything descended from it before it reaches the user. In their tests that raised the share of runs where a planted error never reached the final answer from ~32% to ~89%. This is the one that makes me optimistic, because it's a message log with parent pointers; not a smarter model. And message logs are something engineering already knows how to build. I've been building the unglamorous version in a separate project, and the shape holds. Every action is an event in an append-only log; an event's trust class is derived from its kind rather than declared by whoever wrote it, so an agent cannot mark its own output as a human decision; and the rule that an outbound client message requires a human principal is a database constraint, not a policy document, so it holds for someone poking at the database with the application nowhere in sight. That is what provenance looks like once it stops being a diagram: a type you can't lie in, and a check that survives the app being bypassed.
The workers are small enough to throw away. Gemma 3 270M is 125MB at 4-bit, follows instructions out of the box, and is designed by its own authors to be fine-tuned for one job rather than used as a generalist. Tuned that way, models this size get close to far larger general models on narrow jobs like sorting and extracting text, at a fraction of the memory; and they stop working the moment the task changes, say the creators themselves. Fragile is fine if something is watching, and the routing and checking layers above are exactly that something. I have my own measurement of how quietly that fragility hides. Compressing a 30B coding model's KV cache down to three bits scored 100% on my intelligence checks, which used prompts of about fifty tokens; at 2.5K tokens of context the same setup completed factorial as def factorial(n: int): int): return n * n * x. Keeping 16 of 48 layers in full precision brought back return math.factorial(n). The failure did not exist at the size I was testing at, which is the general shape: cheap parts fail silently, and the layer above has to be the thing that notices.
The part everybody skips
Put those pieces side by side and a familiar diagram shows up. The classifier router is a load balancer. The specialists are workers. Mixture-of-Agents is a quorum polling several answers. The learned orchestrator is a scheduler. Every component of a horizontally scaled service is being reinvented for inference, one paper at a time; mostly without anyone saying the phrase out loud.
But horizontal scaling has a requirement that's easy to miss until it breaks something: your workers have to be stateless, or their state has to live somewhere cheap. The web spent years prying session state out of its servers and into stores like Redis, precisely so any worker could handle any request and workers could be killed and replaced freely.
A model's "session state" is its KV cache. And right now that state is welded to the most expensive memory in the building. Every loaded specialist occupies GPU memory whether it's working or not, and every swap means paying the full cost of re-reading the context from zero. That's the exact property that makes horizontal scaling too expensive, and it's why the fleet of specialists has stayed a diagram instead of a deployment.
The fix is the same one it's always been: move the state somewhere cheap. Save the KV cache, and the prepared weights, to disk. Waking a specialist back up becomes a file read, not a recompute from scratch. This is already how it works at the big end. LMCache moves KV caches out of GPU memory, through RAM, down to local disk and remote storage, and the wins are dramatic where context is long; their disk benchmark takes time-to-first-token from 6.3s cold to 0.15s warm, because loading saved state off disk beats re-reading a long prompt even though disk is slow. And if you use a hosted provider you're already living in this economy without the diagram; cached input tokens cost a tenth of fresh ones, because the provider is doing exactly this trade for you. The price sheets make the point more bluntly than the architecture diagrams do. A cache read is 0.1x base input, so a cached prefix pays for its write after one or two reads. Gemini bills $0.50 per million tokens per hour simply to hold a context cached. A managed agent session is billed $0.08 per session-hour of wall clock while it is merely running, on top of every token it spends. Warm state is metered by the hour wherever you look for it. A file sitting on a disk is not.
Here's the part I actually find exciting. If a specialist's entire warm state is a file, then how many specialists you can have is limited by disk. Disk is cheap and grows to terabytes on hardware you already own. How many run at once stays limited by compute. You stop asking "how much GPU memory do I need to hold my models?" and start asking "how big is my library, and how much of it is hot right now?" That's a caching question. We are good at caching questions. That's the whole trade: RAM for disk, scarce for cheap, the move that turns multi-agent systems from a demo into something with a real cost curve.
What I built, and where it stands
I've been prototyping the small end of this myself, and the numbers are now specific enough to argue with.
Prefill is the cost, and skipping it is the win. My clearest evidence came out of a run that wasn't about caching at all: pushing a 30B coding model to a 512K-token context on one 48GB laptop. Prefilling those 524,288 tokens took 6,671 seconds, about 111 minutes. Generating the 32-token answer afterwards took roughly 2 seconds. Reading the context, not answering from it, is where effectively all the wall clock goes; and it gets worse the further you go, with per-chunk throughput falling from about 163 tokens/second at 65K to about 58 at 512K, because each new chunk has to attend over everything before it. Every design in this post that avoids re-reading a prompt is chasing that one ratio.
Save-and-wake works, and the saving is measurable. A 0.5B specialist's KV cache freezes to disk encrypted, wakes back up, and keeps generating. On a 6,145-token prefix, the resumed session pushed 12 prefill tokens through the model where a cold rebuild pushed 6,156: 513x fewer, on real hardware, against a threshold of 3x that I wrote down before running it. The same measurement on stock full-precision caches came out at 551x (8 tokens against 4,409 on a 4.4K prefix), and there the forked resume also beat re-prefilling on wall clock, deep copy included: 119ms against 696ms. I count prefill tokens rather than seconds deliberately. The token ratio holds whatever GPU you run it on, and it's the number the design depends on.
Routing is as cheap as the receptionist argument needs it to be. A classify on that 0.5B model, prefill plus one logit, measured 10.7ms at p50 and 11.3ms at p95, against a budget of 150 and 300ms I'd fixed in advance; a later run of the same gate came back at 7.9 and 8.0ms. A short tool-use reply came in at 422ms. Sorting costs about ten milliseconds.
Residency is bounded by disk, not RAM. The piece that turns those facts into a fleet is a registry holding N specialists, each with its own resumable session, with only a capped hot set resident: an LRU with logged eviction, snapshots keyed by a hash of the specialist's current log-derived context, and stale snapshots collected so the disk stays bounded too. A missing or stale snapshot is never fatal; it rebuilds from the log. That rule is what makes the whole arrangement safe to run, because it means the log is the truth and the cache is disposable, so a fragile saved state can cost you time and never correctness. The number that makes eviction a real policy is the residency budget: about 812MB resident per 0.5B 4-bit specialist. This part is built and tested and wired into nothing that serves traffic; I'm not going to call a parked module a deployment.
The same session layer runs against the 30B coding model as six REST endpoints, and the whole lifecycle, create, fork, rewind, save, load, is in a public benchmark log: forks are instant reference copies, rewind drops 35 tokens from a context in constant time without re-reading anything, and a saved session wrote 47 layers into under a megabyte and came back. The encryption is doing real work, because a KV cache contains the conversation that made it, and an unencrypted snapshot is your private data sitting on disk dressed up as a performance optimization. The snapshots are AEAD-encrypted at rest and the registry refuses to run against a store that has no keystore, so there is no configuration in which the plaintext path is the convenient one.
Compression is what makes the state small enough for the trade to be worth making at all. At 256K tokens on that laptop, a full-precision KV cache is about 10.5GB and thrashes; the same cache at 4-bit is about 2.6GB, and that headroom is the difference between failing around 256K and passing at 512K on the same machine.
What I still owe is wall clock on the disk path specifically. A slow enough disk read could eat a 513x token saving while the token-ratio gate sails through, and the timing gate that would catch that is written and not yet run. Until it is, cheaper is proven in tokens and assumed in seconds. The other thing I have not proven is the interesting half of routing: my learned router picks a specialist by embedding the request and taking the nearest centroid, deferring when the best match is weak or the top two are close, and its measured win is against an injected embedder in tests rather than a real one on real traffic.
And the uncomfortable bits for anyone this excites: the router is a single point of failure, a poorly calibrated one that doesn't know when it doesn't know, quietly making everything downstream worse. Saved caches are fragile by design, valid only for one exact pairing of weights and prompt prefix; which is why Hermes's never-change-the-top-of-the-prompt discipline and the save-to-disk story are two forms of the same idea. Error cascades are why I think provenance ends up as necessary as the load balancer, part of the base design rather than an optional extra. And the gap in the evidence is worth repeating, because it's the space between this post and reality: learned orchestration is proven with big workers, small workers are proven on narrow jobs, and nobody has published the system that does both at once.
Scaling up buys capability in a straight line, on hardware that gets scarcer the higher you climb. Scaling out is the only shape that has ever made computing cheap, and every piece of it already exists somewhere in the literature. What's missing is the unglamorous work: residency management, deciding what stays loaded and what gets evicted, which in my case is an LRU and a garbage collector; router calibration; and wiring the whole thing together. The supervisor I run my own fleet of agent sessions through is the same kind of object, and it's made of nothing exotic: exact-match signals, dedupe, a debounce so a firehose arrives as one digest, buffered events that survive a restart, and a single-instance lock. That's ordinary systems work, the kind platform engineering already has good answers for. Anyone who wants to can just pick it up. I know I've been doing exactly that.