The_number_that_isn't_the_number_I_needed

Alibaba's Qwen3.8-27B landed on 14 August 2026 with a headline benchmark of 61.7% on SWE-bench Pro against 53.4% for Claude Opus 4.6 Max. That's a real result on a real coding benchmark - the 61.7% figure is corroborated across VentureBeat, Northflank, and DataNorth AI; the Opus 4.6 Max comparison is explicit in DataNorth AI. It's also not what I needed to know.

My fleet doesn't run every sub-task on Opus, and coding isn't the sub-task I was trying to route away from the cloud. What I actually needed: on the specific, repeated decision my own dispatch layer makes - agent-routing classification - does a model I can run on my own hardware for free hold up against the model my fleet calls by default. That's a different question, with a different answer. So I built the test instead of citing someone else's.

The_setup:_one_real_decision,_run_twice

Every commissioned brief in my planning history carries a delegate: field in its frontmatter, recording which of my fleet's 11 named specialists actually did the work. I scoped the task set to 22 briefs about the Labs repo itself, redacted specialist names and cross-repo specifics, and ran both arms against the identical system prompt (the fleet's roster, one line per specialist) and the identical per-task prompt.

Local Qwen3.8-27B ran through a self-hosted Ollama daemon, called directly over HTTP. Cloud gpt-5.4-mini ran through @diabolicallabs/llm-client, matching the model my fleet is actually configured to use by default. A pilot of 8 tasks ran first, per the experiment's own stop condition: local scored 6/7 correct (85.7%), cloud scored 7/8 (87.5%), neither result was "dramatically unusable," so the full 22-task run went ahead.

What_the_full_run_measured

Local Qwen3.8-27B scored 81.8% (18 of 22) on agent-routing classification. Cloud gpt-5.4-mini scored 77.3% (17 of 22). The local model was more accurate, on this task, on this run, not by a wide margin, but not by rounding error either.

The trade-off showed up somewhere else. Local calls averaged 6.0 seconds; cloud calls averaged 1.1 seconds, roughly 5.6 times faster. Local cost was $0 metered across all 22 calls; cloud cost was $0.00676 total. The latency gap was the constraint, not accuracy.

The_wrong_answers_told_me_more_than_the_right_ones

Four of the local model's 22 predictions were wrong. Three of those four exactly matched the cloud model's wrong answer on the same task: both read two infrastructure-adjacent frontend tasks as backend work (naming Sable instead of Nix), and both named Reid instead of Cleo on a content-audit task with editorial framing. That's evidence about my roster's boundary ambiguity, not about the smaller model's competence. The one task where the two arms disagreed with each other, both wrong, was the only miss that didn't replicate.

I also found a real tuning knob I hadn't accounted for going in. The same one-line classification prompt took 92 seconds with the model's extended-thinking mode left on, versus 24 seconds with it explicitly disabled, for the same one-word answer. Every local call in the committed run sets think: false. That's not a footnote; it's roughly a 4x latency difference on a task that gains nothing from reasoning, and it's the kind of default that silently taxes a benchmark if nobody checks for it first.

What_I'm_actually_deciding

The honest constraint this run exposed wasn't model quality. It was plumbing. My fleet already has a local-inference lane, local-mcp-service, that routes to promoted models like qwen3.6:27b-mlx and qwen3.6:27b-coding-nvfp4 for execution-typing work. That gateway has no entry for Qwen3.8-27B yet. Registering one is a separate, cross-repo change, so this benchmark called Ollama directly rather than block on that work or quietly route around it. I'm naming that gap here rather than letting the benchmark imply a deployment path that doesn't exist yet.

That reframes the actual question. "Is the smaller model good enough" is not open anymore, on this task, it already is. The open question is whether I want a several-second-per-call local lane wired into the gateway for a routing decision my cloud tier currently clears in about a second, and whether the tasks where that trade-off is worth taking are the tasks I'm already routing to the free-tier local lane for other reasons.

Continuity_with_the_routing_thread

This isn't a standalone result. It's the same question the Model-Routing Cost/Latency Benchmark asked when it compared providers on cost and fit rather than benchmark scores, and the same trade-off the Reasoning Budget Router made explicit when it started spending inference budget deliberately instead of uniformly. Qwen3.8-27B is one more data point in a fleet that's already treating "which model runs this" as an operational decision with its own evidence, not a default nobody revisits.