AI AgentsInfo
Submit your tool

Checklist

AI agent harness evaluation checklist: twelve sourced checks

Twelve checks a team can answer about its own agent harness. Each one is tied to a sentence a company published, or to a field on an official board. Mark yes only when you can point at the test, the trace, or the row.

Sources read . The Google, NVIDIA, and Anthropic posts were re-read that day. The SWE-bench and Terminal-Bench fields are the ones stored on the November 2026 harness benchmark, read November 8, 2026.

Marked yes: 0 of 12. Still open: 0. Unanswered: 12. This count is your worksheet. It is not a SWE-bench score, a Terminal-Bench score, or a rank. Nothing you mark is saved.

What do you check before you quote a public board?

Dogfood first, write one check for an action you can see, and keep an end-to-end destination suite.

  • The agent can already do routine work in its own codebase.

    Google’s September 9, 2026 post says evaluations belong after dogfooding. Until the agent can handle routine work in its own codebase, a suite is early.

    Google Developers Blog, September 9, 2026

  • You have a written check for one observable action, such as a clarifying question, a validator before “done,” or a canonical link.

    The same post treats those three actions as behavioral checks. The check reads a step in the trace, rather than a final paragraph.

    Google Developers Blog, September 9, 2026

  • You still keep an end-to-end suite as the destination score.

    Google says behavioral checks and end-to-end suites are complementary. Macro benchmarks verify the destination. Behavioral checks are the partner you use while you change the harness.

    Google Developers Blog, September 9, 2026

When can you accept a change to the harness?

Start from a stored failure, change one part, and require the held-out tasks to improve as well.

  • The test started from a stored trace or a failure you can point at.

    NVIDIA’s November 4, 2026 walkthrough points NeMo Compass at stored traces, then writes Harbor tasks for a failure the traces already show: a send without confirmation.

    NVIDIA Technical Blog, November 4, 2026

  • The edit you are judging changes one part of the harness.

    In that walkthrough the harness edit is one instruction change, in Hermes SOUL.md. The post’s closing step is to try one change, then rerun the tests.

    NVIDIA Technical Blog, November 4, 2026

  • Some tasks were hidden from whoever proposed the edit, and you accept the change only when the development tasks and the held-out tasks both improve.

    NVIDIA withheld two of four Harbor tasks from the coding agents that proposed the edit, and says to accept a change only when it improves both the test cases and the held-out set.

    NVIDIA Technical Blog, November 4, 2026

What keeps a trial from lying?

A clean environment, a written grader, and a task that states everything the grader checks.

  • Each trial starts from a clean environment.

    Anthropic’s January 9, 2026 post says each trial should be isolated. In internal evals, Claude gained an advantage by reading git history left by earlier trials.

    Anthropic, January 9, 2026

  • A grader is written down, and it scores a transcript or the environment’s final state.

    Anthropic defines a grader as logic that scores some aspect of performance, a transcript as the full record of the trial, and an outcome as the final state in the environment. A last sentence that says the job is done is not the outcome.

    Anthropic, January 9, 2026

  • The task states everything the grader checks, and a reference solution passes those checks.

    Anthropic says agents should not fail because a filepath or another detail was missing from the task, and that a reference solution should pass the graders so you know the task is solvable.

    Anthropic, January 9, 2026

How do you read a public harness score?

Name the harness, keep the date on the source, and keep unlike boards in separate columns.

  • Any score you repeat names the harness on the same row.

    The November 2026 harness benchmark copies the harness column from the official SWE-bench and Terminal-Bench boards. A percent without that column is a different claim.

    AI Agents Info, November 2026 harness benchmark

  • You kept the date printed on the official source next to the number.

    That benchmark stores each result date. The Terminal-Bench 4.0 board record it copied says the board was updated 2026-10-23. The page was read November 8, 2026.

    Terminal-Bench and the November 2026 harness benchmark

  • You kept SWE-bench release 1.x and release 2.x apart, and you read Terminal-Bench accuracy with its uncertainty. Cost stays its own cell.

    This site’s November 2026 harness benchmark, reading swebench.com, says release 2.x calls tools while release 1.x reads the action out of the model’s text, and that those generations are not necessarily comparable. Terminal-Bench publishes accuracy with a 95% half-width. The benchmark copies cost as its own field.

    SWE-bench, Terminal-Bench, and the November 2026 harness benchmark

Which pages sit next to this checklist?

The long guide is How to evaluate an AI agent harness. How to stand up the tasks is How to build an eval harness for AI agents. Public rows are the November 2026 harness benchmark. The paper shelf those notes live on is AI agent papers. A revenue table, which is a different question, is verified revenue. Install counts are which AI agent framework people install. Live counts are the trackers. A store an agent can quote is how to make your store readable by AI shopping agents, next to the six-month shopping report.

Sources

Source: aiagentsinfo.com, AI agent harness evaluation checklist: twelve sourced checks, updated .

Cite or embed this

The citation names this page and the date above. The embed is a blockquote with a link back here. Nothing here is a required badge.

Citation

AI Agents Info. "AI agent harness evaluation checklist: twelve sourced checks." Sources read November 10, 2026. https://www.aiagentsinfo.com/tools/ai-agent-harness-evaluation-checklist.

Embed

<blockquote cite="https://www.aiagentsinfo.com/tools/ai-agent-harness-evaluation-checklist">
<p>Twelve yes-or-no checks for an AI agent harness, each tied to a public post or board. The count stays in the browser. It is not a SWE-bench or Terminal-Bench score. Sources read November 10, 2026.</p>
<footer><a href="https://www.aiagentsinfo.com/tools/ai-agent-harness-evaluation-checklist">AI agent harness evaluation checklist: twelve sourced checks</a>, AI Agents Info.</footer>
</blockquote>

What else do people ask about this checklist?

It is a worksheet of twelve sourced checks. It does not save answers and it does not publish a score.

What is an AI agent harness evaluation checklist?

Twelve yes-or-no checks for the software around a model. Each check quotes a public post or an official board: Google’s September 9, 2026 harness post, NVIDIA’s November 4, 2026 NeMo walkthrough, Anthropic’s January 9, 2026 evals post, and the SWE-bench and Terminal-Bench fields copied on this site’s November 2026 harness benchmark.

Does the checklist produce a benchmark score?

No. It counts how many checks you marked yes. That count is a worksheet. It is not a percent resolved on SWE-bench and it is not an accuracy on Terminal-Bench.

Is the worksheet saved?

No. The answers stay in the browser until you leave or clear them. There is no account.

How is this different from the harness benchmark?

The November 2026 harness benchmark copies public SWE-bench and Terminal-Bench rows, each with the harness and the date on the source. This page is the list of questions to ask before you treat one of those rows as a reason to change your own harness.