Nemotron 3.5 Super on Era

The Query and Compute tasks are 33 factual questions about one simulated fintech. The only way to answer them is through the company's tools: REST and MCP calls to Salesforce, HubSpot, Zendesk, Jira, Gong, Slack and S3. A model has to find the right records across systems, read every page, join, group and aggregate them, and get the arithmetic right: counts, sums, medians, "which account had the most". Two questions have no answer in the data, and the right response is to say so. Each question has one exact key, computed from the same records the tools serve. Every model answers each question several times, and the score is the share of runs that were correct.

Nemotron scored 89.9%, the highest of the ten models we have run on these questions. Claude Opus 4.8 is next at 76.8%. With 33 questions the intervals still overlap, so the exact order is not settled, but Nemotron is clearly in the top group. It also never refused a question that had an answer; every other model did so between 3 and 17 times.

Compared with Claude Opus 4.8 it did better on counting, pagination, joins, grouped and aggregate questions, cross-system joins, superlatives, multi-hop chains and time-based questions. Its weak spots are the one distribution question (0 of 3), time-based questions (half right) and the Gong call questions. Outside these 33 questions it also missed every document-store question and the Slack and Jira counts.

The hard board has eight questions where the convenient reading of the data gives the wrong answer. A rule is mentioned once on a sales call and has to be applied to the records, or two systems disagree and only one of them is current. Nemotron sits in the middle here. With the LangGraph harness it solved 2 of 24 runs, the same as GPT-5.6-sol, and without a harness it solved none. Claude Fable 5.1 and GPT-6 Astra solve half or more.

Some questions can't be answered from the connected systems: an account's credit rating, its current MRR, where an analyst firm placed the company. The right move is to abstain: abstain: true plus a note on what was searched. On 90 held-out runs of such questions Nemotron got 64 right. In most of the other 26 it searched well and then turned "nothing found" into an answer: an empty field became "MRR is 0", or the note said nothing exists while the answer field still held the string "None".

Post-training starts from a finished model and changes its behaviour with a small amount of targeted data. Alignment is post-training aimed at how the model behaves, rather than what it knows. Supervised fine-tuning (SFT) is the simplest form: show the model examples of the behaviour you want and train it to reproduce them.

The training loop is the easy part of SFT; the data is the hard part. You need many examples in the format the model works in (tool calls, tool results, the final structured answer), and you need to know each one is right. Era gives you that for free, because the grader is the label.

After tuning, every abstention has the right shape, answer: null, abstain: true, and no run overflows the context window. It searches, decides and stops. Two cases that flipped:

A targeted behaviour in a 121B model changed with 366 examples and 60M trainable parameters, and labelling the examples cost nothing. Frontier models get their behaviour from post-training too, at a scale few teams can afford. A model with Nemotron's cost that you can align to your own data and failure modes closes much of the gap to models that cost five to twenty times more per correct answer. How much of the gap it closes is an empirical question, and Era is how you measure it.

Most agent fine-tuning data is either written by people (small and expensive) or judged by another model (only as good as the judge). Era works differently. Generation starts from the question: the generator picks the answer, plants the records that produce it across the connected systems, and gives the grader the key. Code checks every submission. A question is unanswerable because the fact was never placed anywhere. The systems speak vendor-faithful REST and MCP, so a training trace looks exactly like what the model will see in production. Generate as many companies as you need, hold some out, and your evaluation stays clean. See how Era post-trains agents.

Post-training pays off most in rich interactive environments, and most models are too expensive to use there. An Era episode is tens to hundreds of tool calls and hundreds of thousands of tokens. Nemotron's hybrid Mamba-2/MoE design suits that: 12B active parameters, a per-token state that barely grows with context, and a 262K window it can actually use. It is efficient to serve and the weights are open, so you can run the whole loop yourself: measure, train on graded rollouts, merge, and measure again, all against a company that doesn't exist.