The_Wrong_Question

When an agent's tool call comes back malformed, the reflex is to rewrite the prompt. Add more examples. Tighten the instructions. Repeat until the failure stops showing up, or at least stops showing up in the cases you happen to be testing.

That's the wrong question, asked with real conviction. Every failure becomes an instruction problem, every fix a prompt edit, and every fix covers only the failure you just saw. The next one arrives differently and the cycle repeats. I have watched this pattern eat entire afternoons across my own agent fleet, and the fixes never generalise, because the actual defect isn't in the words. It's in the architecture that has no layer built to catch the failure in the first place.

Reliability_Is_a_Systems_Problem

This is exactly the argument Dr. Eric Siegel makes about predictive analytics systems generally: a model's value isn't its raw capability, it's what happens around the model when it's wrong. A prediction without a decision layer underneath it is a curiosity, not a system. The same logic applies directly to agentic pipelines. A frontier model without a deterministic validation layer underneath it isn't a reliable system either, no matter how good the model's benchmark scores look. The category error is precise: treating model quality and system reliability as the same variable.

Where_the_Guardrails_Actually_Go

The architectural alternative is concrete, not aspirational. Deterministic logic sits at the boundary, doing the things that don't need a model at all: schema validation, type checking, required-field enforcement, timeout handling. Classifier ML sits at the edges, doing narrow, well-scoped judgement calls, like sorting a response into valid or invalid, or flagging an output for a second pass. The frontier model sits in the middle, doing the actual reasoning and generation work it's good at, inside a structure that doesn't trust its output until that output has passed through the layers on either side.

None of this is exotic. It's the same separation of concerns any backend engineer applies to user input: never trust the client, validate at the boundary, fail loud rather than fail silent. Agentic systems rarely follow this discipline, mostly because the "input" in question is a model's own output rather than a user's. It's tempting to skip the scepticism. Don't.

Why_I_Built_the_Benchmark

Vendor benchmarks like Terminal-Bench and SWE-Bench test whether a model can complete a task. They don't test what happens when a production pipeline hands that model a real-world tool call and the response comes back subtly wrong: malformed JSON in a stream, a tool call that silently drops a parameter, structured output that validates loosely but is semantically wrong. None of these failures show up on a leaderboard.

EXP_009 skips model capability. It tests what happens when a model's output meets a strict contract, using task patterns from my own twelve-agent system rather than benchmark-friendly prompts.

The_Setup

Fifteen standardised tasks, split across four categories: simple tool calls, parallel invocations, structured JSON generation, and multi-step reasoning. Each task is a JSON spec, prompt, Zod schema, example outputs, and tool definitions, tagged as synthetic or production so vendor-neutral patterns stay distinct from real extracts.

All three models, GPT-5.5, Claude Opus 4.7, and Gemini 3.1 Pro, run every task in parallel via Promise.allSettled, with a 30-second timeout per task and a 10-minute limit on the full suite. The runner validates every raw response against the Zod schema and scores three things: pass or fail, latency in milliseconds, and validation errors where they exist. An invalid JSON response or a missing required parameter counts as a fail even when the model's underlying intent was clearly sound. That's a deliberate choice. Intent doesn't ship to production; a parseable, schema-valid response does.

Results stream live over server-sent events as each task completes, no polling, and the full result set, all fifteen tasks times three models, exports as JSON with raw responses, validation errors, latency, and suite configuration attached, so the run stays inspectable and reproducible after the fact.

What_the_Dashboard_Actually_Showed

Here's the finding that mattered more than any per-model score: the most instructive failure on this dashboard wasn't a model failure at all.

In late May 2026, the entire Claude Opus 4.7 column on a production run started failing every tool-calling task with an HTTP 400 error. On the surface, that looked exactly like the story everyone tells about model reliability, one frontier model quietly worse at tool use than the other two. It wasn't. The actual defect was in the harness: all three provider adapters were passing a stubbed, non-standard object as the tool's input schema instead of a real JSON Schema, and Anthropic's API was the one that correctly rejected it rather than silently accepting a malformed contract. OpenAI and Gemini were accepting the same broken schema at the time, just not enforcing it yet. Fixing the adapter to pass the tool's actual schema, rather than a pass-through stub, resolved it across all three providers in a single change.

That's the architecture argument, made in production data rather than in the abstract. If the fix for that outage had been "try a different model," the actual defect, a malformed contract at the boundary, would never have been found, and it would have resurfaced the moment any pipeline used the same stub pattern against any provider strict enough to enforce its own schema.

The second finding is smaller but points the same direction. Gemini 3.1 Pro's column returns missing_api_key rather than a model response in this benchmark's environment, a configuration gap, not a statement about Gemini's reliability. It fails before the model is ever asked to do anything. OpenAI's and Anthropic's columns, by contrast, are live and verified against the schema on every run.

Model-Specific_Versus_Architectural

This is the distinction the dashboard exists to draw, and it's worth being precise about it, because the two failure types call for opposite responses.

A model-specific failure shows up on a single provider and doesn't reproduce elsewhere: one model consistently mishandling a parameter type, or timing out on a task category. Those are prompt-fixable or fixable by switching models. You can route around them.

An architectural failure is one that shows up regardless of which model sits behind it, because the defect is in the contract, not in the reasoning. The malformed inputSchema stub exemplifies this: it broke Claude first because Claude enforced the contract, and was one dependency bump from breaking the others - not a model defect, a contract violation. No amount of prompt engineering touches that class of failure, because the model was never the layer where the defect lived. The fix has to happen in the harness.

Vendor benchmarks can only ever surface the first kind. They test the model in isolation, by design. They cannot see a contract violation between your harness and a provider's API, because that contract doesn't exist until you build the pipeline that creates it. That's the gap this benchmark is built to close, and it's the gap most reliability conversations skip past entirely.

What_This_Means_for_Production_Pipelines

Don't start with "which model is more reliable." Start with "what happens in my pipeline when a response fails validation." If the honest answer is "I'm not entirely sure," that's the actual finding, and it's a finding no model swap will fix.

Concretely: validate every tool response against a real schema before downstream logic, not a stubbed placeholder. Make failures loud rather than silent. An HTTP 400 you can see beats malformed output propagating three steps further before something breaks obscurely. Test your harness's contract with the same rigour you'd apply to testing a model. The incident above didn't happen because Claude was unreliable. It happened because nobody tested what the harness was actually sending.

None of this means model choice doesn't matter. It means model choice is a downstream decision, made after the architecture is sound, not a substitute for building the architecture at all.

The_Practitioner_Test

Here's the test I now apply to any reliability fix before I accept it: if the fix is in the prompt, you've named a symptom, not a cause. A prompt edit that happens to stop a specific failure from recurring in your test cases isn't evidence the underlying defect is gone. It's evidence you haven't found the defect yet.

Reproducibility

The task suite lives in the Labs repository, organised by category. The dashboard exports full run results as JSON with raw responses, validation errors, and latency, so the findings above aren't claims to trust blindly. Run the same fifteen tasks against your own provider stack and verify which failures belong to your code and which to the architecture underneath.