Tool-call benchmark
Ten questions from the easier half of the v2 board — ones most lanes already answer when they may read as much as they like — asked again with a budget the model is told up front: at most 20 tool calls, as few tokens as possible. An answer counts only when it is exact and arrived inside the budget. Everything below is counted from the recorded runs: calls, turns, tokens, wall time and the provider's bill. One lane is written up on its own: Grok on Era.
What is being measured
The hard board asks whether a model can find the right reading of ambiguous data. This board asks a different question about work a model can already do: how cheaply can it do it? Dossiers, joins across two or three connectors, a rate over a filtered set, a root-cause read — the ordinary questions an agent at a customer answers all day. Without a budget the same lanes spent anywhere from six to a hundred and fifty tool calls on them; with one, planning the reads becomes part of the task.
The budget
Each question is preceded by the same brief:
The cap is enforced by the harness, not just stated: past it, every read returns a refusal telling the model to
submit what it has or abstain. submit_answer is free. Refused reads are still counted, so a run that
asked for more than its budget shows as an overrun even if it then answered correctly from what it had —
and that answer does not score. The token half of the brief is an instruction, not a rail: prompt and output
tokens are recorded per run and reported, and the price chart is where a lane that ignores it shows.
Scoring
The headline is the solved-within-budget rate: runs whose answer matched the key exactly and whose tool calls did not exceed the cap, over all runs of the lane (ten questions × three repeats). Wilson 95% intervals on that count. Beside it: the plain exact rate (correct regardless of budget), overruns, abstentions, and the price per solved answer — the lane's whole bill divided by its solved count, so a lane that solves half the board for a dollar a run pays two dollars an answer. Each question also carries a par: the fewest calls any lane solved it in, so a lane's call counts read against what was possible rather than against the cap. Provider refusals and harness errors are not answered, shown separately, never scored as a miss.
Harnesses
Every lane runs langgraph — LangGraph's ReAct agent with the benchmark system prompt, gateway
meta-tools, a persistent run_python sandbox and submit_answer (one run_python
call is one tool call however much it pages inside). The native loop (same prompt and meta-tools, no code
execution) holds the cap too, but has no lanes on the current harness. ADK lanes are not on this board: that harness does not enforce a cap, and a budget the loop does not hold would
not be the same experiment.
Corpus and replay
Every lane ran against the committed replay corpus (datasets/segments/seg-fintech-large, what the
partners environment serves); the corpus hash and the cap are recorded per run directory and a lane
off either is refused by the scorer, so the numbers replay from the repository alone.