EXPERIMENT_027 // MODEL-ROUTING.COST/LATENCY.BENCHMARK
A fixed task set of five experiential-design briefs runs against three models — Claude Opus 5, GPT-5.6 Sol, and Gemini 3.6 Flash — and a deterministic rubric scores each response, demonstrating the real cost, latency, and quality trade-off behind routing a task to a bigger model instead of a cheaper one.
LOADING EXPERIMENT...
Model routing is usually argued from a benchmark leaderboard or a vibe, not a measurement against your own task shape. This experiment fixes that: five short experiential-design briefs — a budget-constrained activation concept, wayfinding sign copy, pop-up KPIs, a weather contingency plan, a fabrication trade-off — run once each against three models through the same @diabolicallabs/llm-client call shape. A deterministic keyword-rubric scores every response, and the real cost, latency, and quality land in a bundled, committed results table rather than a live per-visitor demo.
HOW IT WORKS
Claude Opus 5, GPT-5.6 Sol, and Gemini 3.6 Flash each answer the same five briefs through @diabolicallabs/llm-client — no direct provider SDK import. Qwen 3.8 Max, the model originally scoped for this slot, was substituted: the fleet has no Alibaba/DashScope API key and the toolkit has no Alibaba provider. Gemini 3.6 Flash already serves as this repo's vision provider (EXP_011 Spatial Brand Mirror), making it the fleet-relevant, actually-callable substitute — stated plainly in the methodology section, not hidden.
Each task carries 3-4 scoring criteria, matched by case-insensitive keyword group against the response text — the same no-model-call convention as this repo's linter-family experiments (ASI-Checklist Linter, Trust-Tier Compliance Linter). Quality score = matched criteria / total criteria x 100, reproducible from the response text alone, with no fourth paid LLM call spent on judging the other three.
The 15-call canonical result set (5 tasks x 3 models) is committed to the repo as a fixture and rendered on this page by default — no live endpoint fans out to three paid providers on every visitor. On the committed run, no single model won on cost, latency, and quality simultaneously; the comparison charts and per-task table show the measured trade-off directly, not a picked winner.
The harness originally set a shared sampling temperature and disabled reasoning per call — both broke on the live models. claude-opus-5 and gpt-5.6-sol reject a custom temperature outright, and each provider accepts a different subset of reasoning-effort values, with claude-opus-5's extended thinking silently consuming the entire token budget before producing visible output when left unset. Both are fixed per-provider in config.ts, caught by running the harness for real before bundling any result, not by assuming the call shape would just work.
WHAT THIS PROVES
A same-call-shape, same-task, same-rubric comparison turns "which model should this route to" from a leaderboard argument into a measured trade-off — on this run, the cheapest and fastest model matched the flagship's quality score on 4 of 5 tasks, which is exactly the kind of result a routing decision should be checked against before defaulting to the most expensive tier.
A benchmark harness is not exempt from the same live-verification discipline as any other integration: two silent failure modes (a rejected temperature parameter, thinking tokens eating the entire output budget) would have produced a broken or misleading result set if the harness had been trusted on first write instead of run for real and inspected before the canonical fixture was committed.