EXPERIMENT_009 // AGENTIC.RELIABILITY.DASHBOARD
Head-to-head tool-calling reliability test across three frontier models. Run 15 tasks simultaneously against GPT-5.5, Claude Opus 4.7, and Gemini 3.1 Pro. Track pass rates, latency, and JSON validity in real time.
LOADING EXPERIMENT...
A benchmark suite tests identical tool-calling tasks against three frontier models in parallel. Fifteen standardised tasks drawn from Diana’s actual orchestration patterns — simple tool calls, parallel function invocations, structured JSON responses, and multi-step reasoning with tool use — execute with a 30-second timeout per task, 10-minute total suite limit. Each task validates the response against a Zod schema, scoring JSON validity, tool-call correctness, and parameter accuracy. This is not a synthetic benchmark. Production agent pipelines fail on patterns that vendor benchmarks do not capture.
HOW IT WORKS
Fifteen standardised tasks live in src/data/experiments/exp_009/tasks/ across four categories: simple tool calls, parallel invocations, structured JSON generation, and multi-step reasoning with tool use. Each task is a JSON spec: the prompt, a Zod schema describing the expected output shape, example outputs, and tool definitions. Each task is tagged with `source: 'synthetic'` or `source: 'production'` so readers can distinguish vendor-neutral patterns from real-orchestration extracts drawn directly from Diana’s 12-agent system. The suite is anonymised and generalised for reproducibility.
The runner accepts a task and model config, executes the model, and validates the raw response against the Zod schema. It returns three metrics per task: pass (boolean), latency (milliseconds), and validation errors (if any). An invalid JSON response or a missing required tool parameter counts as a fail, even if the model’s intent was sound. The runner also captures the raw output and error state for inspection. All three models run in parallel via Promise.allSettled per task.
A POST endpoint kicks off the suite run. The frontend connects to an SSE stream that emits results as tasks complete. Each event carries the task ID, model name, pass/fail badge, latency, and validation errors. The aggregate stats bar updates live: cumulative pass rate, mean latency, JSON validity rate, and tool-call accuracy. No polling, no progress bars — raw result data as it flows.
The full results set is exportable as JSON: all 45 task runs (15 tasks × 3 models), with raw responses, validation errors, and latency data. Researchers can inspect the failures, reproduce the results, and feed the data into their own analysis pipelines. The export includes timestamp, suite configuration (timeout values), and model versions so results remain interpretable over time.
WHAT THIS PROVES
Vendor benchmarks (Terminal-Bench, SWE-Bench) test against standardised tasks. Production agent systems fail on specific patterns that standardised benchmarks do not capture: malformed JSON in streaming responses, tool calls that silently drop optional parameters, structured outputs that technically validate but semantically miss. This experiment generates original data using real orchestration patterns drawn from a live 12-agent system. The results are reproducible and open.
The dashboard also reveals failure modes that are invisible in a pass/fail aggregate. Gemini 3.1 Pro, for example, currently fails with missing_api_key rather than returning a response — that is real production data, not a software bug. OpenAI and Anthropic results are live and verified against the schema. The combination of all three shows which models handle the given task patterns reliably and which reveal gaps. That is the question builder systems actually ask.