A lot of the discussion around AI model routing starts from a common idea: don’t use the most powerful model if a cheaper one can do the job.
Microsoft Research has explored this multiple times. In Hybrid LLM, BEST-Route, One Head, Many Models and, more recently in 2026, Switchcraft, the objective is broadly the same: route each request to the lowest-cost model that can still meet the required quality or correctness threshold.
You can see the same broad idea in systems such as GitHub Copilot’s automatic model selection.
It is something like a Minimum Viable Model principle.
Recently I ran into a paper, The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs, that made me realize that agentic workflows make this problem harder.
Small differences in per-step reliability can translate into very large differences in the probability of successfully completing a long sequence of actions.
If a model is 99% reliable at each step, then over 100 independent steps:
0.99¹⁰⁰ ~ 37%
At 99.9%:
0.999¹⁰⁰ ~ 90%
And real agents can be worse than this simplified example, because errors are not necessarily independent: a mistake can enter the context and influence later decisions.
So the routing question changes from:
What is the cheapest model that can answer this request correctly?
to:
What is the cheapest model that can complete the expected trajectory with an acceptable probability of success?
That is a different optimization problem.
The cheapest model per token is not necessarily the cheapest model per answer. And the cheapest model per answer is not necessarily the cheapest model per completed task.
A weaker model may generate more reasoning tokens, require more retries, make more tool-call mistakes, accumulate bad context, or eventually need to be replaced by a stronger model.
In coding agents like copilot Model switching during a long-running session is not free, because rerouting can mean processing a lot of context again. GitHub therefore performs automatic model selection at “natural boundaries” rather than continuously changing models during an active task: GitHub Copilot automatic model selection.
So a routing decision made near the beginning of a task can have consequences much later.
Microsoft Research’s Switchcraft describes routing as selecting the lowest-cost model subject to correctness.
But for agentic systems, that would translate to:
Selecting the lowest-cost model subject to trajectory-level correctness.
A model that is only slightly less reliable may look almost as good on a benchmark or on a single question. But over a long chain of actions, those small differences compound. This is also the paper’s argument for continued scaling: “scaling model size leads to large improvements in the number of turns a model can execute successfully.”
So perhaps the real routing question is not simply which model can do this task? but which is the cheapest model I trust to get all the way to the end of the job?
