EXPERIMENT_030 // PROGRAMMATIC-VS-JSON.TOOL-CALLING.BENCHMARK
Six data-lookup tasks over a fixed mock order dataset run twice each against two models — once through native JSON tool-calling, once by having the model write code that calls typed function stubs directly in a sandbox — reproducing arXiv 2608.06370's finding that programmatic tool calling needs far fewer LLM round trips per task than the JSON tool-calling pattern most agent frameworks default to.
LOADING EXPERIMENT...
Most agent frameworks default to native JSON tool-calling: the model proposes a tool call, the app executes it, the result goes back for another model turn, repeated once per tool invocation. arXiv 2608.06370 argues a model writing code that calls typed function stubs directly resolves multi-step tool chains in a single generated script instead. This experiment runs both arms against the same 6 fixed tasks (multi-step lookups, conditional chains, batch aggregations) over the same 5-order mock dataset, through the same two models, and records success rate, LLM call count, tool-invocation count, latency, and cost for each.
HOW IT WORKS
6 tasks run against GPT-5.4 Mini and Claude Haiku 4.5 through @diabolicallabs/llm-client — no direct provider SDK import. JSON mode calls withTools() in a loop, executing each requested tool call against the real implementation and feeding results back as the model asks for more. Programmatic mode asks for one JavaScript function, runTask(), that calls the same underlying functions directly; that function then executes once in a Node vm sandbox exposing only the 6 instrumented stubs — no require, no process, no network.
@diabolicallabs/llm-client's LlmMessage only supports system/user/assistant roles — there's no native tool_result content block for continuing a withTools() conversation after executing a tool call. The JSON-mode loop works around this by synthesizing tool results as plain user-role text, which is a real behavioural difference from a provider's native multi-turn tool format and the biggest confound in this comparison. Flagged for the toolkit's maintainers in the PR, not fixed from an experiment build.
Each task has one correct, data-derived answer (an order ID, a computed total, an OVER/UNDER verdict) — success requires every expected fact to appear in the final answer text, after normalising comma-formatted numbers. No LLM judge, no partial credit toward the headline success-rate metric — a model that computes the right number but omits a required ID still fails the check, and that trade-off is documented plainly rather than loosened to flatter either mode.
On the committed run, JSON mode reached 100% success on both models at roughly 2.8 LLM round trips per task; programmatic mode reached 83% success at exactly 1 round trip per task and roughly half the latency. Programmatic mode's only misses were both models correctly computing the right total but naming an event instead of the required order ID — a scoring-strictness cost, not a wrong answer. Neither mode is declared the outright winner; the comparison charts and per-task table show the measured trade-off directly.
WHAT THIS PROVES
The round-trip-count gap the paper reports is reproducible at fleet scale with cheap, fast models and a small fixed task set: programmatic tool calling collapsed every multi-step and batch task into a single LLM call on this run, while JSON tool-calling needed one model turn per tool call it made — a real cost and latency difference an agent framework's default calling convention should be chosen with, not assumed.
A benchmark that measures a real gap in your own shared tooling is as valuable as one that measures the models: the harness could not build a fair multi-turn JSON tool-calling arm without discovering and documenting that @diabolicallabs/llm-client has no native tool-result message type — a finding that outlasts this run's specific success-rate numbers.