The_Wrong_Question

Every benchmark announcement comes with an implicit claim: this model is best. Best is doing a lot of work in that sentence. Best at what? Best under what conditions? Best at what cost, over what volume, in whose infrastructure?

The production question is never "which model is best?" It is: which model handles this category of work correctly, at the cost that makes the operation sustainable, within the governance constraints I have decided to live with? These are not refinements of the ranking question. They are a different question entirely. Ranking answers do not transfer.

I have been running a thirteen-agent fleet across roughly fifteen active projects since late 2025. The routing decisions I made when I built that fleet were not made by consulting benchmarks. They were made by answering three questions for each agent: what is the task profile, what is the cost at production volume, and what does sovereign control over the model require? The answers produced a tiered system that is still running as designed. Not because I picked winners. Because I matched models to work.

The_Three_Criteria_That_Actually_Govern_Routing

Task fit. This is the obvious one, and still the most misapplied. Task fit is not about whether a model can do the work. It is about whether the model's capability profile matches the demands of the task category. A code review requires precise reasoning over structured text with low tolerance for hallucination. A social caption requires register calibration and cultural sensitivity. These are different cognitive profiles. Running both through the same frontier model is not wrong - it is wasteful, and it misses a governance opportunity.

In the fleet, Nix (code review and implementation), Quinn (QA), Tom (research synthesis), and Jo (operations and audits) run on Sonnet. The task profile for each agent is reasoning-heavy, technically specific, and consequential - errors are visible and load-bearing. Haiku runs Vera (copy), Lena (social), Maya (design operations), and Cleo (content). Their task profile is judgement-heavy but less technically consequential - the cost of a weak first draft is a revision cycle, not a broken deployment.

This is not about capability ceilings. Haiku is capable of more than Vera needs from it on most turns. The point is that overprovisioning capability wastes tokens and, more importantly, obscures the signal. When you run a Haiku-tier task on Sonnet, you get more model than the task requires. That does not make the output better. It makes it harder to detect when the output is wrong, because wrong output produced fluently is the failure mode you cannot catch by feel.

Cost-per-token-at-volume. This is where the routing decision becomes economic rather than qualitative. At low volume, the price difference between tiers is negligible. At production volume - thousands of agent turns per week - the difference is material.

The fleet runs Sonnet as its override ceiling for subagents. Nothing below the main session runs on Opus. The main session uses Opus during planning and architecture (plan mode), then drops automatically to Sonnet for execution. This is not a budget constraint imposed from outside. It is a deliberate architecture: Opus-grade reasoning for decisions, Sonnet-grade execution for actions, Haiku for work that requires speed and volume over depth. The cost curve of that architecture is significantly different from a flat "use the best model for everything" policy - and the output quality is not noticeably degraded, because the task profile for most agent work does not require frontier-tier reasoning.

Governance and sovereignty. This is the least discussed routing criterion and, in a multi-vendor world, increasingly the most important. Routing is not just a capability decision. It is a dependency decision. Which vendors do you have active relationships with? Which models run in infrastructure you control? Which choices lock you into a specific provider's pricing model or availability guarantees?

The fleet is deliberately multi-vendor. The main fleet runs on Anthropic's Claude models. Sam - the external Codex CLI agent - is OpenAI's Codex, invoked via CLI wrapper as a separate lane. That is not a hedge against Anthropic. It is an architecture decision: some categories of work benefit from a model specialised for mechanical iteration and natively integrated with the development environment. Maintaining that lane independently preserves the option to route there without reorganising the fleet. Sovereignty means keeping the routing decision yours - not defaulting to one provider because switching costs are high.

The_External_Lane:_What_Sam_Is_Actually_For

Sam deserves his own paragraph because the temptation to misread him is real. He is not a second opinion on tasks the main fleet could handle. He is not a vendor pivot or a backup when Anthropic is slow. He is the answer to a specific question: what is the right tool for mechanical iteration work - repetitive code transformations, multi-file renames, boilerplate generation - that neither Sonnet nor Haiku is optimised for?

Sonnet can do this work. Haiku can do some of it. But Codex, integrated natively into the development environment via CLI, handles it with lower latency and at lower cost for that specific task profile. The architecture decision to add Sam was not "OpenAI sometimes beats Anthropic." It was "mechanical iteration is a distinct task category that warrants its own tool, and the right tool happens to be external." The fleet now routes that category explicitly. Sonnet is freed for the work it is actually suited to.

What_the_Industry_Is_Now_Confirming

Two recent product releases make the structural logic of this routing split concrete at an industry level.

Microsoft shipped MAI-Thinking-1 and MAI-Code-1-Flash in the same suite. MAI-Thinking-1 is positioned as a frontier reasoning model - preferred over comparable models in blind evaluations on complex reasoning tasks. MAI-Code-1-Flash is optimised for low-latency agentic coding, integrated directly into GitHub Copilot and VS Code. Two models, same vendor, explicitly divided on the reasoning-vs-cost-optimised axis. This is the same split the fleet has been running operationally. Microsoft named it as a product architecture, not an inference.

NVIDIA's Nemotron 3 Ultra confirms the economics. 550 billion parameters, 55 billion active via sparse mixture-of-experts architecture. Day-zero availability on SageMaker JumpStart. Glean is already deploying it for cost-effective agentic workflows. The significant figure here is not a benchmark score - it is an economic signal: NVIDIA's own positioning for Nemotron places it at frontier-adjacent capability - materially lower cost and latency, without a material gap in the task profiles that dominate agentic fleet work. That makes the routing split economically legible to anyone still unconvinced by the task-fit argument alone. You can route frontier-adjacent reasoning at a fraction of the cost, if the task profile fits. That is not a reason to stop thinking about routing. It is a reason to think about it more carefully.

The FutureAGI analysis from May - three frontier models (Opus 4.7, GPT-5.5, Gemini 3.1 Pro) functionally equivalent at the edge on production workloads - still holds. The W23 evidence fills in the other half. The frontier has converged; the below-frontier tier is now differentiating on specialisation and economics rather than raw capability. Routing into that tier is no longer a compromise. It is a design choice.

Stop_Asking_Which_Model_Is_Best

The benchmark conversation will continue. New models will ship, new scores will be posted, new leaderboard positions will be celebrated. The fleet will not notice.

This is not arrogance - it is the point. When the routing decision is separated from the ranking question, model releases become inputs rather than directives. A new frontier model ships: check the task profile, check the cost curve, check the sovereignty implications. Adjust the routing if the answer changes. Do not rebuild the architecture. The fleet I run has absorbed several model generations without a structural change, because the routing logic does not depend on which model is currently winning.

The benchmark conversation answers a question that production routing does not ask. It will keep producing answers to that question. The question a production system asks is narrower and more tractable: what does this agent do, how often, at what cost, within what constraints? When you have answered those questions, the routing mostly follows. You do not need a leaderboard. You need a task profile, a cost model, and the discipline to match them.

That is the difference between a fleet that runs and a fleet that chases.