EXPERIMENT_038 // LOCAL.QWEN3.8-27B.AGENT-ROUTING.BENCHMARK
22 real historical fleet-dispatch decisions — redacted briefs paired with their actual `delegate:` ground truth — run through a self-hosted Qwen3.8-27B on Ollama and through the fleet's default cloud model, scoring which specialist each one predicts and measuring the real accuracy, latency, and cost trade-off of routing that one decision locally instead of to the cloud.
LOADING EXPERIMENT...
Alibaba's Qwen3.8-27B shipped with strong, corroborated benchmark numbers for a model small enough to run entirely on local hardware. A leaderboard score answers a different question from "can this replace one specific cloud call in my own fleet." This experiment picks the fleet's own agent-routing decision — which of 11 named specialists should own a given task, the exact call this site's own orchestrator persona makes routinely — and runs 22 real historical instances of it (redacted briefs, real `delegate:` ground truth from proj-plan history, not synthetic prompts) through both a local Ollama-hosted Qwen3.8-27B and the fleet's default cloud model, side by side.
HOW IT WORKS
Every task is a redacted paraphrase of an actual commissioned brief from this project's own planning history, with its ground-truth label taken directly from that brief's own frontmatter `delegate:` field — never invented. Specialist names and any cross-repo specifics are stripped from the task text itself (the classification target the model has to predict), and all 22 source briefs are scoped to this repo, whose architecture is already fully public in its own AGENTS.md, so nothing paraphrased here discloses anything not already published.
Both arms see the exact same system prompt (the fleet's 11-specialist roster with one-line role descriptions) and the exact same per-task prompt. The cloud arm calls `gpt-5.4-mini` through `@diabolicallabs/llm-client`, matching this repo's own configured default. The local arm calls a self-hosted Ollama daemon directly over HTTP — `@diabolicallabs/llm-client` has no Ollama provider, and the fleet's `local-mcp-service` gateway has no `qwen3.8` entry yet, so this is a direct integration, documented as a scope-respecting judgment call rather than a workaround.
A response is scored correct only if it names the real historical delegate, parsed from an `ANSWER: <Name>` line when the model follows the requested format, or from the first valid roster name found anywhere in the response otherwise. A response naming no valid roster member scores as an unparseable miss rather than being silently dropped — the same no-judge-call, no-silent-drop convention as this site's other benchmark and linter-family experiments.
Live-measured before committing to the full run: the same one-line classification prompt took 92 seconds with the model's extended-thinking mode left on (212 tokens of chain-of-thought for a trivial answer) versus 24 seconds with `think: false` — a roughly 4x latency difference on a task that gains nothing from reasoning. Every local-arm call in the committed run sets `think: false` explicitly.
WHAT THIS PROVES
On the committed 22-task run, local Qwen3.8-27B scored 81.8% routing accuracy (18/22) against the cloud default's 77.3% (17/22) — the self-hosted model was not "dramatically worse," it was marginally more accurate on this exact task, while running roughly 5.6x slower per call on average (6.0s vs 1.1s) and at $0 metered cost against the cloud arm's $0.00676 total across all 22 calls. Three of the four wrong predictions were shared by both arms on the same tasks — a real signal about where this specific roster is genuinely ambiguous (infra-adjacent frontend work reads as backend to both models), not a local-model weakness.
The honest trade-off this run actually measures is latency and operational surface, not correctness: a locally-run 27B model can match a fleet's default cloud tier on a real, recurring classification decision, but currently at several seconds of added latency per call and outside the fleet's existing gateway (a new-model registration gap flagged plainly rather than routed around). That is a genuinely different decision than "is the smaller model good enough" — it already is; the open question is deployment plumbing, not model quality.