EXPERIMENT_029 // EFFORT-LEVEL.A/B.HARNESS
Runs a fixed six-task benchmark at four reasoning-effort levels against Claude Opus 5, recording real cost, latency, and a rubric-scored quality result for every task-effort pairing, then charts the trade-off curve.
LOADING EXPERIMENT...
Click "Run sweep" and a fixed set of six hand-authored tasks — two trivial, two medium, two hard — each runs for real against Claude Opus 5 at four reasoning-effort levels (low, medium, high, and a deliberately unsupported "none"), using `@diabolicallabs/llm-client`'s `reasoningEffort` call option (v6.3.0+). Every one of the 24 (task, effort) cells records real USD cost, latency, and a deterministic rubric-scored quality result, then a trade-off chart plots average quality against average cost per effort level — showing whether paying for more reasoning effort actually buys a better answer, task by task.
HOW IT WORKS
`reasoningEffort` is a per-call option on `LlmCallOptions` (v6.3.0+) — the same Claude Opus 5 endpoint is called four times per task with only that one field changed. This experiment extends EXP_018's model-tiering framing one level down: instead of swapping to a cheaper model, it swaps how hard the same model thinks before answering.
Two of the six tasks (the bat-and-ball problem and a lily-pad doubling problem) are classic Cognitive Reflection Test items — questions with a fast, wrong, intuitive answer and a slower, correct, reflective one. They exist specifically to make an effort/quality relationship visible: a low-effort pass is more likely to reproduce the intuitive wrong answer than a high-effort one, unlike the trivial and medium tasks where even minimal effort usually lands the right answer.
Every task prompt ends with an explicit "answer with only the number/word" instruction, and the rubric scores the response's final line against the task's expected-answer pattern: 100 for a match on the last line, 50 if the correct answer appears earlier in the response but not as the final line, 0 if it never appears. No second model call judges the first — mirroring EXP_018's classifier, which is also a deterministic check rather than an added LLM round-trip.
"none" is swept deliberately even though Anthropic's API does not accept it for Claude Opus 5 — `@diabolicallabs/llm-client` rejects the combination before any network call (`LlmError` with `kind: 'bad_request'`), and that specific error kind is what the harness uses to label the cell "not tested" rather than silently omitting it from the results grid or miscounting it as a real failure.
WHAT THIS PROVES
Reasoning effort is a genuine per-call parameter now, not a model swap: running the identical request at low vs. high effort against the same Claude Opus 5 endpoint produces measurably different cost, latency, and — on the two CRT tasks specifically — correctness, proving effort is a usable lever a real system could tune per task rather than a theoretical knob that only exists in provider documentation.
It extends EXP_018's practitioner framing from picking a cheaper model to picking a cheaper effort level on the same model: the same fleet instinct that routes trivial tasks to a smaller tier applies one level down, to how hard a single model should think before answering — and the two CRT tasks make the case concretely, since an under-effort pass reproduces the fast wrong answer while a higher-effort pass more reliably reaches the correct one.