The_question_I_actually_wanted_answered

I set out to answer a narrower question: when I next build a Labs experiment or a fleet tool that needs to call functions, should I expose those functions as JSON schemas the model fills in, or as typed code the model calls directly?

The paper that prompted the question, "The Bitter Lesson of Tool Calling" (arXiv 2608.06370), reports that programmatic tool calling matches or exceeds native JSON tool calling in 11 of 14 tested models on BFCL v4, and stays stable under context degradation where JSON-based calling measurably degrades. The mechanism it names is control flow: a model writing code gets loops, conditionals, and batching for free. A model filling in a JSON schema gets one call, one round trip, one wait for the next turn, every single time.

That's a compelling argument on paper. It is not, by itself, a reason to change how I build things. A benchmark I haven't run myself, against models I don't actually use, on tasks that aren't mine, tells me what happened somewhere else. So I built the comparison as a Labs experiment instead of taking the paper's word for it: six data-lookup tasks over a fixed mock order dataset, run twice each against GPT-5.4 Mini and Claude Haiku 4.5, once through JSON tool calling and once by having the model write a single function that calls typed stubs directly inside a sandbox. That's EXP_030, live in the Playground.

What_the_build_exposed_before_the_benchmark_ran

The JSON-mode arm needed a working multi-turn conversation: model proposes a call, harness executes it, result goes back as a new message, model asks for the next call if needed. That should have been the easy half of the harness to build.

It wasn't, because @diabolicallabs/llm-client - the single entry point every LLM call in my fleet goes through - has no native tool_result content block. The message type supports system, user, and assistant roles. It has no way to say "here is what that tool call returned" inside a properly typed multi-turn exchange. To keep the JSON-mode arm running at all, the harness had to synthesise tool results as plain user-role text and feed them back that way.

That is not a neutral workaround. A synthesised text result is a different shape of input than a provider's native tool-result format, and it's a real behavioural difference between what the JSON-mode arm was measuring and what native JSON tool calling actually looks like in production. It is, in the strictest sense, a confound. I had two options once I saw it. Patch the toolkit inside the experiment PR, quietly make the workaround disappear, and publish a comparison that reads as cleaner than the ground it was measured on. Or leave the toolkit alone, flag the gap for the maintainers in the PR description, and put the confound in front of anyone reading the experiment page.

I took the second option. Not because it produces a tidier headline, but because a benchmark that hides its own worst assumption is worse than no benchmark at all. The whatItProves section on the experiment page says it plainly: this comparison couldn't be built cleanly, and that fact outlasts the specific numbers the run produced.

What_the_numbers_said,_with_the_caveat_attached

On the committed run, JSON mode reached 100% success at 2.8 round trips per task. Programmatic mode reached 83% success at one round trip and half the latency. The misses were scoring-strictness errors, not model failures.

Read narrowly, that's a reproduction of the paper's central finding at a scale I can actually verify: programmatic tool calling collapsed every multi-step and batch task in the set into a single model turn, where JSON tool calling needed a fresh turn for every tool it touched. On six tasks and two cheap, fast models, the round-trip gap the paper reports held.

Read honestly, the programmatic arm had a structural advantage: it never needed to represent a tool result as a message, because code can call stubs directly inside a sandbox. JSON tool calling carries a typed tool-result dependency that has to work correctly, or the comparison is already unfair.

The_decision_that_actually_anchors_this

I did not decide, on the strength of one benchmark run, that every future tool I expose to an agent should be a typed code stub instead of a JSON schema. That would be exactly the kind of absolute prescription the paper's own findings don't support, 11 out of 14 models, not 14 out of 14, and a scoring-strictness gap on the programmatic side that I have no evidence resolves cleanly outside a six-task fixture.

What I did decide, in the PR that shipped EXP_030, is narrower and more durable. When a tool surface in this fleet needs multi-step chaining, batching, or conditional logic, over more than one or two calls, programmatic exposure is now the default I reach for first, because the round-trip cost of JSON tool calling is real and I've now measured it on my own tooling rather than taken it on faith. And when a build surfaces a gap in shared infrastructure like llm-client, the toolkit gets flagged, not silently patched around inside whichever experiment happened to hit it first. A workaround that disappears into one PR is a workaround the next person building on the same toolkit doesn't know exists.

The paper gave me the mechanism. The experiment gave me the numbers. The toolkit gap gave me the actual production decision: disclose the confound, flag the fix upstream, and let the comparison stand on ground I've been honest about rather than ground I've quietly levelled.