Paper breakdown: Terminal-Bench 2.1 — 28 of 89 tasks were repaired, and scores moved by up to 12 points

Terminal-Bench 2.1 is a repair of a benchmark that was already supposed to be clean. Buchanan, Zhou, Linderman, and Ré say version 2.0 had been through human review, model-assisted auditing, and red-teaming before release, and that 28 of its 89 tasks still needed a fix. The fixes ship as Terminal-Bench 2.1. The abstract says accuracy rose by over 10 percentage points. The discussion says an individual score can move by as much as 12 points, while the ordering of 19 agents stays highly correlated.

This page was published November 8, 2026. The PDF, the HTML, the arXiv record, the 2.1 repository, pull request 53, and the Terminal-Bench homepage were read that day.

What did the repair change?

The repair changes tasks, not models. Section 5 says the audit found issues in 28 tasks, roughly a third of the 89. The paper sorts the defects into three piles, and a task can sit in more than one. Figure 3a prints the shares. Table 2 prints the counts.

Sixteen divided by 28 is 57%. Twelve divided by 28 is 43%. Eight divided by 28 is 29%. The three counts sum to 36 because of the overlap. The example the paper gives for a mismatch is query-optimize: the grader demanded a byte-for-byte copy of a database file, and the instructions never said so. Drift is a task that broke after release because a package, an API, or a web page moved. Environment issues are budgets and images that were too tight for the known-correct solution. The repair rule for a resource limit is at least 1.5 times what the oracle and the grader actually used.

The sources of the 28 are also printed. Eleven tasks had already been flagged in Terminal-Bench 2.0 Verified, five more had been discussed in public issues and pull requests, and 14 were new to this audit. The paper says two tasks appear in more than one of those lists, and 11 + 5 + 14 − 2 = 28.

Z.ai's earlier audit, which this paper cites, is a different measurement. The paper says environment fixes alone, on all 89 task images, moved Claude Opus 4.5 from 52.4% to 57.9% in that earlier report. That 5.5-point gap is Z.ai's number. It is not a row in Table 7.

Which of the 28 tasks actually moved?

Table 7 is the number a reader can check. Each rate is the pass rate of that task averaged across the paper's agent-model pairs. After the repair, the paper says no task in the table is still at zero for every agent. The largest drop is overfull-hbox, from 53.3% to 36.7%, which the appendix explains as a broken build that had been giving credit the repaired grader refuses.

The five tasks that go from at most 15% to over 50% are the ones the main text says account for roughly 62.7% of the absolute pass-rate change on the modified tasks. Adding the absolute changes on those five rows gives 305.9 points. Adding the absolute changes on all 28 rows gives 487.5 points. 305.9 / 487.5 is 62.7%.

The signed changes sum to +377.7 points, or 13.5 points per repaired task. That is the mean on these 28 rows. It is the wrong number to paste next to a model's name. The other 61 tasks are outside the table. If they were unchanged, the same +377.7 points would be 4.2 points across all 89 tasks. The abstract's "over 10 percentage points" is larger than 4.2, so it is not this diluted average. The discussion's ceiling, 12 points on an individual agent, is a third quantity. The paper never prints the 19 accuracies that would let a reader see which agent moved by 12.

Claude Code with Claude Opus 4.6 is the one agent the text names for a rank change: it sits more than 7 points above the fitted line and climbs three places. Codex with GPT 5.4 sits below the line. The two weakest agents, both under 40% on version 2.0, lose accuracy on 2.1. The paper says that is what you would expect when 19 of the 28 repairs were hard tasks. It does not name those two agents.

Did the ordering of agents survive?

Mostly. Spearman ρ = 0.958 between version 2.0 and version 2.1 across 19 agents. The task-bootstrap 95% interval is 0.881 to 0.979. The fitted line in the text is TB2.1 = 2.1 + 1.04 × TB2.0. A correlation that high means the broad order is stable. A move of three ranks, which is the maximum the paper reports, is enough to change which of two close agents you would call the leader.

Table 6 is the configuration list, and it has 20 rows, not 19. Three use Claude Code (Opus 4.6, Sonnet 4.6, GLM 5.1). Four use Codex (GPT 5.3 Codex, GPT 5.4, GPT 5.4 mini, GPT 5.5). Two use Gemini CLI (Gemini Flash 3, Gemini Pro 3.1). Eleven use Terminus-2, including Gemma 4 31B and Kimi K2.6. Qwen 3.5 9B and Qwen 3.5 27B appear in the model table and in the budget curves, and they are not in Table 6. This page stops at that count. It does not guess which row was left out of the 19.

The auditor-agreement table is a different 89. Claude Code (Opus 4.6) and Codex (GPT 5.4) agree on the primary defect label, including "no defect," at κ = 0.80 on all 89 tasks. Cheaper judges agree less. Table 1 prints three pairs, each over the same 89 tasks:

Against the final 2.1 release, the workflow's precision is 57.1% and its recall is 71.4%. The paper says those figures are retrospective, because the workflow helped build the repair set. They are a check on triage, not a claim that an unattended agent found every bug.

How much of the harness gap is the clock?

At the default timeout, the harness is a large part of the score. The abstract says extending the time limit closes a gap of about 20 points on GPT 5.4 and shrinks a 30-point gap on GPT 5.4 mini to 5 points. Figure 6 prints eight accuracies for Codex and Terminus-2 on Terminal-Bench 2.1. One pairing reproduces those gaps exactly, within ordinary rounding:

On GPT 5.4, 74.2 − 53.9 is 20.3 points at the default clock. At 32×, Terminus-2 is at 75.3 and Codex is at 74.9, so the gap has closed and the order has flipped by 0.4 points. On GPT 5.4 mini, 67.0 − 37.1 is 29.9 points, and 68.2 − 62.9 is 5.3 points. Codex barely moves when it is given more time. Terminus-2 uses the extra time. The paper's conclusion is that a fixed budget is an operating point, and that a harness comparison read only at that point can be a comparison of speed.

Figure 3b says something compatible about failures. More than half of failed trials at the default budget ended in a timeout. The range the text prints is 68% under Claude Code and 18% under Gemini CLI. The appendix adds that 11.8% of graded timeouts still pass, because a timed-out trial is scored on the files the agent left behind. About 27% of upstream trials were filtered out before that chart, as runner and provider failures rather than agent outcomes.

GLM 5.1 is the efficiency outlier the text names. On the budget sweep it solves 8 more tasks on version 2.1 than on 2.0 and overtakes GPT 5.4, whose coverage barely changes. The Qwen curves barely change at all. A longer clock and a repaired task set can reorder agents even when the default-time ranking looks stable.

Terminus-2 is the baseline harness: a single-agent ReAct loop with a shell and file tools. The ReAct paper is the reasoning-and-acting note on this site. Claude Code and Codex are the provider harnesses in the same figure. A harness guide that starts from the product side, rather than from this benchmark, is How do you evaluate an AI agent harness?. The November harness board on the live site is a different version, and the November harness benchmark reads that board.

What was on GitHub the day this page was written?

The paper says Terminal-Bench 2.1 is at harbor-framework/terminal-bench-2-1, and that the discussion of the revisions is pull request 53. Both were checked on November 8, 2026.

The README's 26 and the paper's 28 were both public on the same afternoon. The directory count confirms the benchmark is still 89 tasks. It does not say which two tasks the README left out, or whether the README is counting a narrower kind of edit.

The homepage check is the reason this page refuses to become a leaderboard. At 13:42:54 GMT, https://www.tbench.ai/leaderboard/terminal-bench/2.1 returned a redirect to /?version=2.1. The HTML of that response still said Terminal-Bench 4.0 and did not include score rows. The README still links a 2.1 board and says only maintainer-run submissions are being added. Until that board is in the HTML, a percentage on a third-party "2.1 leaderboard" is someone else's run.

Two earlier notes on this site already use the words Terminal-Bench 2.1 for other experiments. Evolver reports a diagnostic slice of 20 tasks per configuration. You don't need long context runs the 89 tasks inside mini-swe-agent and studies context, not grader repairs. A percentage from either note is not a cell in Table 7, and Table 7 is not a submission to Terminal-Bench 4.0.

The audit also left Terminal-Bench for a spot check. On 34 tasks drawn from an 80-task Harbor-Index snapshot, covering 29 source benchmarks, the workflow flagged 14 and manual review confirmed 12, which the paper calls 86% precision. Reviewers also found defects the workflow missed, and two bugs in shared grading code touched 13 tasks. The paper treats 2.1 as a checkpoint. The last sentence of the discussion says the pipeline can keep running, so version 2.1 is not a finished board.