An LLM router is a layer that sits in front of a pool of models, evaluates each incoming request, and forwards it to the model best matched to what that request requires. Requests within the capability range of smaller models are directed to them. The frontier tier is reserved for requests that require it.
Coding agents are the clearest case for routing, because a single working session contains an unusually wide range of difficulty. An agent building a feature will read files, grep for symbols, list a directory, rename a variable, add a form field, write a migration, generate a test fixture, update a config file, and write a docstring. It will also, somewhere in that session, need to reason about a race condition or decide how to restructure a module boundary. The second category justifies frontier pricing. The first category does not, and in a session of that shape it accounts for far more requests than the second.
When the agent is pinned to a single model, every one of those requests is billed at the same rate. The difference on any individual call is small enough to ignore. Across a working day, a team, and a month of sessions, it becomes the dominant component of the bill.
Routing addresses that directly. The difficulty is entirely in the evaluation step, because how a router decides which model should handle a given request determines whether it saves money, whether it slows the agent down, whether the code it produces holds up, and whether any of that remains true six months later. Those four outcomes are the axes on which every existing routing approach either succeeds or fails, so they are stated below as explicit conditions.
The four conditions a router must satisfy
The conditions interact, which is what makes them hard to satisfy together. A router can buy quality by spending more on its decision, which costs economy and latency. It can buy economy by downgrading aggressively, which costs quality. A router that meets three of the four trades a visible problem for an invisible one.
Economy
The combined cost of running the router and the models it selects must land substantially below what the same traffic would have cost against a frontier model. This is the condition most routers are compared on, and at first glance that is the right approach, but the arithmetic behind it is less forgiving than it appears.
Whatever the router spends choosing a model is charged against every request, including the ones it sends to the frontier model anyway, where the decision bought nothing. That overhead comes out of the price gap the router exists to capture, so a router with meaningful decision cost has to save considerably more than its headline number before it breaks even. Approaches that require a full model inference pass to make the selection are one way this goes wrong.
Latency
Routing happens before generation begins, so whatever it costs in time is added to every request in the system and sits in front of the user on all of them.
The benchmark for that latency is the fastest model in the pool. A frontier model working through a hard problem takes seconds, and against that number almost any routing overhead disappears. A small model answering a file lookup or a symbol query can return in around a hundred milliseconds, and that is the call the routing decision has to stay out of the way of. A router that adds four hundred milliseconds to a hundred-millisecond response has quadrupled the time to answer. The requirement is that routing latency sit an order of magnitude below the shortest downstream call, which puts the target in the tens of milliseconds or lower.
Quality
Output quality from the routed system must match or exceed what sending every request to a single frontier model would have produced. A router that downgrades slightly too aggressively produces acceptable output on ordinary requests and degraded output on the difficult tail, which means cost metrics improve while the system gets worse. Catching this requires continuous measurement against a held-out baseline, plus per-request logging of which model handled which step. Without the second, a failed session is difficult to attribute to anything.
Durability
The router has to stay current as new models are released, and it has to do so without accumulating maintenance work that offsets what it saves.
New releases land at price points that did not previously buy that level of capability, so work that required the top tier last year becomes a mid-tier request this year. A routing configuration calibrated in January can still be working in April, still saving against its original baseline, and pointing at a lineup that no longer represents the best available trade. Durability comes down to what is required to incorporate a new model: automatic absorption, a retraining cycle, and a human writing new rules are three different ongoing costs.
Why a well-designed router can exceed the performance of every model it routes to
Routing gets framed as a cost reduction technique with quality held constant, which understates what it does. The quality ceiling is higher than the strongest model in the pool.
No single model performs best on every request. When a set of frontier models is evaluated against the same broad test set, each one misses items the others answer correctly, and the misses are not randomly distributed. In coding specifically, models differ in which languages and frameworks they handle well, in how reliably they follow a diff format, in long-context retrieval across a large repository, in tool-calling discipline, and in whether they over-edit when asked for a narrow change.
A router that picks well can therefore beat the best model in its pool. Send a Rust question to the model that is strongest at Rust and you get a better answer than the highest-scoring model overall would have given, because that model is not the one best at Rust. Do that across a whole workload and the routed system ends up ahead of every model it selects from. This has been measured on MMLU-Pro, where routed setups scored higher than the strongest individual models available to them while costing a fraction as much per token.
Pinning an agent to one model therefore leaves measurable performance unrealized regardless of budget, which is an argument for routing that applies even where cost is not a constraint.
Common routing approaches and where each one fails
Routing is already performed in production in several forms, ranging from a configuration file to a trained model. Taken in order of sophistication, each one addresses the weakness of the one before it. Each also fails at least one of the conditions above, and in several cases the failure is structural rather than a matter of implementation quality.
Static assignment
The simplest approach assigns models to categories of work in advance. A developer decides that planning and architecture go to a frontier model, and that file reads, edits, and test generation go to something cheaper. It is a configuration file, it costs nothing to run, and it adds no latency, so economy and latency are satisfied without difficulty.
Quality is where it breaks, and the reason is that categories are too coarse to describe what is inside them. "Edit a file" covers renaming a variable, which is a find-and-replace, and it covers changing a function that forty other places in the codebase call, where all forty have to keep working afterward. "Change the database" covers adding a column nothing else touches, and it covers restructuring a table the rest of the system depends on while it stays online. Each pair lands in the same bucket and gets the same model, so the hard one is handled by something too weak for it. Categories fine-grained enough to avoid this are too numerous for anyone to maintain.
The second failure follows from the first. The mapping is hand-authored, so every new model release and every change in how the agent decomposes work requires someone to revisit it. That revision competes with feature work for attention, so the configuration ages in place.
Rule-based routing
A rule set adds granularity by evaluating each request individually. Conditions inspect signals such as prompt length, file extension, keyword matches, which tool the agent is invoking, or how many files are currently in context. This is an improvement over sorting by category, and on narrow, well-characterized workloads it performs acceptably for a long time.
The limitation is that every available signal is a proxy for difficulty rather than a measurement of it, and the proxies fail in both directions. A long prompt can be a large pasted file with a trivial question attached to the end. A short prompt can describe a subtle concurrency bug in nine words. File extension indicates language rather than difficulty. Keyword matching identifies phrasing rather than substance. Each rule is defensible on average and wrong on a meaningful minority of requests, and the minority is not the part of the distribution you can afford to get wrong.
So rule-based routing inherits the quality ceiling of static assignment in a more elaborate package, and it inherits the maintenance burden along with it. Every rule is a piece of hand-written logic that has to stay true as the agent, the codebase, and the available models all change underneath it.
A language model as the router
This approach hands each request to a language model and asks it which model should handle the work. It is appealing because the decision is finally being made by something that can read and understand the request, which addresses the proxy problem directly.
It fails on a constraint that cannot be tuned away. A small model used for the decision keeps cost low and routes unreliably, which fails quality. A larger model routes well and adds enough cost to consume the savings the routing was meant to produce, which fails economy. There is no size that satisfies both.
The latency failure is worse and independent of the size question. Every request requires a complete generation pass over the full input before it reaches the model that will answer it, so the decision is priced and timed like an answer. On requests routed to a small model the router can take longer than the model it selected.
A learned classifier
The strongest of the common approaches trains a model to predict, for a request it has not seen before, which model in the pool will handle it well. The classifier is small and purpose-built, so it costs little to run. Quality holds close to what sending everything to the strongest model would produce and can exceed it, for the reasons in the previous section, because the prediction is fine-grained enough to distinguish the nullable column from the zero-downtime backfill.
Latency depends on how the classifier is built. Being small is not sufficient on its own, and a classifier that takes four hundred milliseconds to return a verdict is slower than several of the models it might select. Getting the decision into the tens of milliseconds is achievable with the right architecture, quantization, and serving setup, and it is engineering work that has to be done deliberately rather than a property that comes free with the approach.
Durability is the weakness. What the classifier learned is bound to the specific models present in its training data. A model released last week has no representation in it at all, and adding or replacing a model in the pool requires collecting fresh performance records and retraining before the router can use the new option. The router therefore lags the frontier by the length of that cycle, permanently, and the lag is costly in coding, where model capability moves fast and where a new release can be materially better at exactly the work the router is deciding about.
None of the four approaches clears all four conditions. Static assignment and rule-based routing fail on quality and durability. A language model as router fails on latency and on either economy or quality depending on which way it is tuned. A well-optimized learned classifier fails only on durability, which makes it the closest of the four and locates the remaining problem precisely: the classification approach is sound, and binding it to a fixed set of models is what limits it.
How a router differs from an LLM gateway
The two terms get used interchangeably, and the conflation matters because it causes teams to adopt one while believing they have addressed the problem the other solves.
An LLM gateway is a unified access layer. It provides a single API across multiple providers and handles authentication, rate limiting, provider failover, and availability management. Its routing decisions are based on operational signals: price, uptime, and latency. Send this request wherever it is cheapest, or wherever the provider is currently up. Gateways such as OpenRouter, LiteLLM, and Portkey operate on this basis, and they solve the access problem well.
A router evaluates the content of the request. It determines what the request requires and selects the model most likely to handle it correctly at acceptable cost, so the decision is a function of the request itself. The two are complementary and are sometimes delivered together, since a router needs a gateway's provider access to be useful at all. A gateway on its own leaves the cost problem described at the start of this article untouched. It makes multi-provider access easier to administer without changing which model handles which step.
How to evaluate a routing system
Three of the four failure modes above do not announce themselves, so they have to be checked for. Several properties of a routing system are observable from the outside, without access to its internals or its training methodology.
Decision logging
Each decision should record the request, the models considered, the model selected, the confidence in that selection, the cost, and the routing latency. In agent workloads this record is what makes a failed session diagnosable, since it identifies which step was handled by which model and where the reasoning first went wrong. A router that treats its own decisions as opaque is asking to be trusted on the one condition that is hardest to verify independently.
Model incorporation process
Ask what happens when a new model is released. Automatic incorporation, a retraining cycle, and a human authoring rules are three answers describing three different products, and the difference between them is the durability condition in practical form.
Disclosed routing latency
Overhead should be published as a figure in milliseconds and compared against the response time of the fastest model in the pool.
Quality-controlled savings
Any router can cut costs by sending everything to the cheapest model available, so the claim to look for is cost reduction with quality held flat or improved against a defined baseline, and for coding workloads the useful measure is task completion rather than token-level similarity to some reference output.
Feedback from production traffic
Stronger implementations feed evaluation results back into subsequent decisions, so selection accuracy improves on a specific codebase and workflow rather than staying frozen at whatever the system knew on the first day.
When routing does not justify its complexity
Two situations call for skipping routing.
The first is low volume. When monthly inference spend sits in the low thousands of dollars, the absolute savings from routing are small, and the engineering time spent configuring and maintaining it would return more somewhere else. A single mid-tier default model is a reasonable choice at that scale, and revisiting the decision when spend grows is easier than unwinding a routing setup that never paid for itself.
The second is workloads where nearly every request is difficult. A session consisting entirely of debugging distributed systems behavior, or of designing a novel algorithm, has a difficulty distribution skewed hard toward the top, so there is little traffic available to downgrade and the router spends its time confirming that the frontier model is needed again. This describes particular sessions more often than it describes a team's overall usage, and separating the two changes the conclusion.
Routing produces the largest benefit when the difficulty distribution is heterogeneous and lower-difficulty requests dominate. Agentic coding sits close to the ideal case, because the mechanical work of navigating and modifying a codebase surrounds every difficult decision.
Key takeaways
- A router must satisfy four conditions simultaneously: economy, latency, quality, and durability. Three of the four fail invisibly, since only cost is measured by default.
- Routing latency should be measured against the fastest model in the pool, not the slowest. A decision that takes 400ms is slower than several of the models it might select, so the target is tens of milliseconds or lower.
- Because model strengths are uneven across languages, formats, and context lengths, an accurate router can perform above every individual model in its pool rather than merely matching the strongest one.
- Static assignment and rule-based routing fail on quality and durability, since categories and surface signals only approximate difficulty. A language model as router fails on latency structurally. A learned classifier clears the first three conditions when it is optimized for serving latency, and fails on durability regardless, because what it learned is bound to the models it was trained on.
- Routing produces the largest benefit when the difficulty distribution is heterogeneous and lower-difficulty requests dominate, which describes a wide range of agentic coding workloads.
Frequently asked questions
What is the difference between an LLM router and an LLM gateway?
A gateway provides unified access across model providers and routes on operational factors such as price, uptime, and failover. A router evaluates the content of each request and selects the model most likely to handle it correctly. Many products provide both, since a router depends on a gateway's provider access to reach the models it selects.
How much latency does routing add to a coding agent?
Implementations built for low overhead add 10 to 50 milliseconds per call. The comparison that matters is against the fastest model in the pool, since a small model can answer a file lookup in around a hundred milliseconds and the routing decision has to stay well below that. Approaches that use a full language model to make the routing decision run a complete generation pass over the input before dispatching, so on requests bound for a small model the decision can take as long as the response.
Can routing be used with open-source or self-hosted models?
Yes. A router selects on model performance characteristics rather than hosting arrangement. Self-hosted models can serve high-volume, low-difficulty agent traffic such as file inspection and simple edits, where the marginal cost per request on owned hardware is close to zero. Mixed pools combining commercial and self-hosted models are common.
What is the difference between model routing and load balancing?
Load balancing distributes requests across identical instances of the same model to manage traffic and maintain availability. Model routing distributes requests across different models based on what each request requires. Load balancing optimizes for throughput and availability, model routing for cost and quality per request. Production systems generally use both, at different layers.
Does routing reduce the quality of generated code?
A correctly implemented router should hold quality flat or improve it, and uneven model strengths across languages and task types make improvement achievable. Verifying it requires measuring task completion against a held-out baseline on an ongoing basis. A single evaluation at configuration time will not catch the gradual degradation that aggressive downgrading produces.
