Benchmarks · model report

Grok on Era

xAI's grok-4.7, served through Amazon Bedrock (grok-4.7-bedrock), on the tool-call benchmark: ten ordinary cross-system questions about a simulated fintech, each under a declared budget of twenty tool calls, three repeats, LangGraph harness with run_python. Grok solved 24/30. Of the four lanes compared here it is the cheapest per solved answer, $0.32 against Sol's $0.42, Sonnet's $1.25 and Astra's $1.39. It is not the cheapest lane on the whole board: GPT-6 Luna, Opus 5.5 and Sonnet 5.5 solve as much or more for less. Every number below is counted from the recorded runs on the committed corpus.

What it did better

The thirty runs, set beside Astra's, Sol's and Sonnet's on the same questions.

1. The same answers for less

Grok solved six questions 3/3: the stalled-chain owner, the courted-but-never-won accounts, the stalling diagnostic stage, the pipeline leader's annual value from the documents, the SLA credits by ticket load and the fix-owner dossier. On those six its runs averaged $0.26. Astra averaged $1.01 and Sol $0.32 on the same six, and both also went 3/3 on all of them. Sonnet averaged $0.68 and solved 12 of the 18 runs. Grok was the cheapest of the four on eight of the ten questions. Sol was cheaper on the courted-but-never-won accounts, and on the stalled-chain owner Grok was the most expensive of the four.

Most of the saving is list price: Grok bills $2 in and $6 out per million tokens, against Sol's $4 / $20, Sonnet's $2 / $10 and Astra's $10 / $50. Grok read about as much as Astra (121k prompt tokens a run against 114k) and more than Sol (82k), and wrote about twice their output tokens (2.4k against 1.2k and 1.1k). Next to Sonnet, which has a similar input price, the saving comes from efficiency: under a third of the prompt tokens (Sonnet 389k), less than half the output tokens and four more solved answers.

2. It writes code

43% of Grok's tool calls were run_python, 6.4 scripts a run. Astra ran 5.3 a run (52% of its calls), Sonnet 7.7 (47%) and Sol 3.2 (30%). Grok also ran fastest of the four: 42 seconds a run, against Sol's 83, Sonnet's 103 and Astra's 120.

3. It always answered

All thirty runs submitted an answer: no abstentions and no run refused past the cap. Sonnet overran nine times and submitted 22. On seven of the ten questions all three repeats gave the identical answer. Sonnet did the same on seven, Astra on eight, Sol on nine.

Where it lost

One definition of "open"

open_urgent_share_by_tier, 0/3. The question asks what share of each tier's accounts has an urgent ticket that is still open. Grok answered 61 / 60 / 50 on every repeat, which matches counting statuses new, open, pending and hold as open. The key counts status open alone: 42 / 25 / 33. Eleven lanes solved it at least once. The joins, pagination and tiers were the same each time; only the reading of that one word differed, and it was applied the same way on every repeat.

Three single slips

The other three losses were one repeat each. On the busiest-caller dossier, one run reported 465 call minutes; the two solved repeats reported 473. On the root-cause blast radius, one run had the same values as its two solved repeats but wrapped them in an extra answer/note object, and was graded wrong. On the named-reference question, one run said 45 accounts and 224 messages, where the two solved repeats said 53 and 477.

No spare budget

Grok averaged 14.8 calls a run, against Astra's 10.3 and Sol's 10.7. On two questions, the SLA credits and the named reference, every run used exactly twenty calls. It finished inside the cap every time, but with no calls to spare.

The hardest question

hard_named_reference_stated, 2/3. Astra and Sonnet were 0/3 and Sol 1/3. Across the whole board nine lanes solved it at least once, and only Sonnet 5.5 solved all three repeats.

Without a budget

This section is the direct xAI model (grok-4.7), from an earlier run. The cost advantage depends on the cap. Grok was also started on the hard board without a cap and was stopped on cost after twelve rows. The LangGraph lane scored 4/10 exact for $107.55: $10.76 a row, 30 to 117 model turns and 57 to 160 tool calls per row, and one row cost $27.80 on 4.2M prompt tokens. The no-code lane scored 0/2 for $14.45. Those twelve rows are kept in the repository as an unscored partial (eval/results/grok-4.7_hard_partial.json). The hard-board scorer refuses an incomplete 8 × 3 grid, so Grok has no row on the hard board and no 90/10 total. Without a cap Grok keeps reading until it is sure, and even at $2 a million tokens that is expensive. Given a budget, it stays within it and is cheap.

Read this with