Garry Tan Harness Evals: $0.17 to $0.44 per Solved Task (Nov 2026)

Garry Tan's harness evals ask which agent harness to run once the model is held still. The matrix report in garrytan/g-harness-evals answers that, for 20 public tasks run on November 5, 2026, the pass rates cannot be told apart. The costs can. OpenClaw on Opus 5.5 cost $0.4435 per solved task. Claude Code, on the same model and the same tasks, cost $0.2170.

AI Agents Info read the README, the scoreboard, and that matrix report on November 9, 2026, at main commit 1c26ae4. The GitHub API that morning listed 11 stars, 2 forks, and the MIT license. The numbers below are copied from those files. A pass rate in a different paper is not a row in these tables.

What did the Garry Tan harness evals measure?

They measured pass rate, cost per solved task, and time, with the model and the 20 tasks held fixed, and only the harness swapped.

The matrix report says the run is 7 cells and 560 trials: 20 public v0.3 multi-user dev tasks, 4 repetitions each. It ran from 2026-11-05 07:02Z to 12:06Z, one Ubicloud standard-16 VM per cell, trials one at a time. Every cell was a plain harness. The report says no trial produced a gbrain event, in 560 of 560 event logs.

Four cells used Opus 5.5, which the report names as anthropic:claude-opus-5-5. Three used GPT-6.1 Sol, openai:gpt-6.1-sol. Claude Code and Codex are each model's own harness. The report says they cannot run the same model, so they are never compared with each other.

Pass rate in the report is a task mean. Each task contributes its own share of passing scored tries, and those shares are averaged. It is not the pooled fraction of passes over trials. The 95% ranges are task-clustered bootstrap intervals, 10,000 draws, seed 7, so a few lucky tasks do not narrow the range. Cost per solved task is the cell's whole spend at list price through the project's metering proxy, divided by scored passes. Spend includes trials the scorer then excluded. Time is the median wall-clock time of one trial.

The harness, in the glossary on this site, is that bundle of loop, tools, memory, and permissions. The benchmark entry is the frozen task set plus the scoring rule. This matrix is one benchmark. It is not the only one, and the last section says what it is not.

Can the pass rates tell the harnesses apart?

No. All six preregistered tests came back "can't tell apart," and the report prints Holm p = 1 for every one of them.

The report's own summary is the sentence to quote: on these 20 tasks, the run cannot tell any two harnesses apart on pass rate, on either model. The biggest gaps it names are on Sol, where the model is shared. OpenClaw is 3.9 points over Codex, interval [+0.0, +11.8]. Hermes is 3.8 points over Codex, interval [+0.0, +10.0]. Both intervals include zero. Each primary pair differed on 1 to 4 tasks.

The reason the tests cannot see a smaller gap is the task set. Eleven tasks passed on every repetition in every cell. Three tasks never passed in any cell. That is 14 of 20 tasks with the same score everywhere. The scoreboard calls the set saturated: half or more of the tasks scored the same in every cell, so the tasks, not the harness, decide most of the score. The minimum detectable difference was 7.7 to 14.2 points at alpha 0.05. The report says the run had less than an 80% chance to detect a gap under that range.

"Can't tell apart" is the glossary line in the scoreboard. It is the preregistered test, adjusted for the number of comparisons, finding no difference. The scoreboard says it does not mean equal. The report's verdict on why the result is null is FIX-METHOD. The tasks hit a ceiling. The report says to raise that ceiling before this matrix runs again.

QM is not one of the six tests. The report says QM ran on the default Docker bridge rather than the internal network, so its comparisons are labelled untested.

What did each harness cost per solved task?

OpenClaw on Opus cost $0.4435 per solved task. The other Opus point estimates were $0.1853, $0.2170, and $0.2434. On Sol the three point estimates were $0.1653, $0.1869, and $0.2222.

The table copies the matrix report's per-cell results. Pass rate is the task mean, printed next to the raw passes so the two are not divided into each other. OpenClaw on Opus is the row where they disagree: 60 divided by 78 is 76.9%, and the task mean is 76.3%.

The seven spend figures sum to $99.0057. The report's prose rounds that to $99.01 of a $156.27 cap. VM list price, from the same table, is $6.3601, and the report says that cost sits beside the cap rather than inside it. Passes sum to 415. Scored trials sum to 554. The 6 trials in the gap from 560 recorded to 554 scored are the cap stops below.

The README rounds cost per solved task to $0.19, $0.22, $0.24, and $0.44 on Opus, and to $0.17, $0.19, and $0.22 on Sol, and it ranks by whether those 95% ranges overlap. On that rule every row is rank 1, with a mark on OpenClaw's Opus row. OpenClaw's range, [$0.33, $0.63] in the README's rounding, overlaps Hermes's [$0.18, $0.36] and clears Claude Code's [$0.16, $0.30] and QM's [$0.13, $0.28]. The README says the cost differences are descriptive, with no test. The test is the second run, further down.

Two cells did not finish the plan. A cap stop, in the report, is the metering proxy refusing a request because the worst-case reservation would pass the shard's lease. The trial is recorded as model_error and is not retried.

The task id in the report is sy-03-marrow-fieldworks-ticket-numbers. On Sol, that task drops out of the comparisons, so those pairs use 19 tasks. The report says OpenClaw on Opus averaged $0.333 per trial against a lease formula that assumed $0.270, and OpenClaw on Sol averaged $0.156 against $0.125. It also records two timed-out Sol requests charged at their full reservations, $2.8755 together. One of those, $1.4432 on cn-01-tamsin-cold-chain-site r4, was logged after the trial's meter closed. That is why the Sol cost interval is built around $0.1964 while the headline, which keeps the charge in the ledger, is $0.2222.

Where does Hermes Agent land?

Hermes Agent has the highest printed pass rate on Opus, 78.8%, and the tests still cannot tell it apart from Claude Code or OpenClaw. On cost, the second run can tell it apart from Codex and cannot tell it apart from Claude Code.

The pin is Hermes Agent v2026.9.24. On Opus the cell is 63 passes out of 80, $0.2434 per solved task, $15.3320 spent, median 64.6 seconds. On Sol it is 60 of 80, $0.1869, $11.2113, median 81.5 seconds. The README's rank rule, overlapping cost ranges, puts both of those rows in rank 1 with the other cells.

The same README says Hermes sends more per request on Sol: 423 kB more tool schemas per trial than the comparison it is drawing, and 16.7 requests per trial against Codex's 11.6. It gives that as the reason Hermes costs $0.023 more per trial than Codex in the separate cost run, interval [−$0.030, −$0.016] on the Codex-minus-Hermes contrast. The scoreboard's waterfall table is the place those calls are marked as tested. On Opus, Hermes at $0.190 per trial "can't tell apart" from Claude Code at $0.180. On Sol, Hermes at $0.141 "costs more per trial than Codex" at $0.117.

The matrix report also notes a side channel that did not become a model call. Hermes probes the provider for its model list. The proxy refused 2,640 of those probes on Opus and 1,540 on Sol. The report says no model request from Hermes was refused, and the refusals cost nothing.

Hermes on this site is already a row in other people's experiments. The HarnessFlow note copies that paper's runs of Hermes v2026.9.7, an earlier pin, on different benchmarks. The SpecCompact note is about compaction inside a Hermes-style summary. Neither table is this matrix. The repository discloses that Y Combinator, which Garry Tan runs, is an investor in Hermes.

Why did OpenClaw cost about twice as much on Opus?

OpenClaw's Opus ledger was $26.6127 and Claude Code's was $13.2346. Dividing those two totals gives 2.01. The report's cost per solved task is $0.4435 against $0.2170, which is 2.04.

The README's account of where the money went: about 81% of OpenClaw's extra cost over Claude Code on Opus is cache writes, because OpenClaw re-sends a much bigger system prompt and bigger tool results. On Sol, the same paragraph says most of the gap to Codex is output tokens, 282,492 against 32,226, and that $0.036 per trial of the gap is one timed-out stream charged at its full reservation. Those token counts are the README's. This page did not re-sum the trial logs.

The matrix itself marks the Opus cost gap as descriptive. The waterfall run is the preregistered cost test. There, OpenClaw on Opus cost $0.345 per trial against Claude Code's $0.180, and the scoreboard's call is "costs more per trial than Claude Code." The README prints the paired contrast as $0.165 less per trial for Claude Code, interval [−$0.206, −$0.128]. The difference of the two printed per-trial averages is $0.165. Median time on the matrix was 99.4 seconds for OpenClaw on Opus and 40.1 seconds for Claude Code. On the waterfall run those medians were 90.3 and 34.1 seconds, and the scoreboard says wall time there is confounded with the machine.

OpenClaw's Sol headline, $0.2222, is not twice Codex's $0.1653. The ratio of those two point estimates is 1.34. The waterfall call on that pair is still "costs more per trial," $0.225 against $0.117, and the README calls the paired gap $0.107, interval [−$0.270, −$0.002], barely clear of zero.

The repository discloses that OpenAI is now the key benefactor of OpenClaw, and that OpenClaw's creator, Peter Steinberger, works for OpenAI. It also discloses that OpenAI began through YC Research. The cells are still the cells in the table.

What did the second run say about cost?

It tested cost per trial. On Opus, OpenClaw cost more than Claude Code, Claude Code and Hermes could not be told apart, and Claude Code cost more than QM. On Sol, Codex cost less than OpenClaw and less than Hermes.

The scoreboard names the run waterfall-frontier-2026-11. Same 20 tasks, two tries each, the same pins. Each cell ran alone on its own VM. The calls below are the scoreboard's, and they are cost contrasts. They are not pass-rate tests. Three Opus cells each printed 80.0%, which is 32 of 40. That repeated 80% is the saturation story again, on a smaller sample.

QM's "cheaper" call carries the network caveat. The scoreboard says QM ran on the default Docker bridge while the other cells ran on the internal network. The README's paired figure for the Claude Code side of that contrast is $0.041 more per trial than QM, interval [+$0.020, +$0.061]. The difference of the printed averages is $0.042. This page quotes the interval from the README and the averages from the scoreboard, and does not treat $0.041 and $0.042 as two measurements.

Two further caveats are on the scoreboard and not in the matrix table. On Sol, the proxy billed every uncached prompt token at the cache-write rate, above OpenAI's input list price, because OpenAI caches prompts on its own. And OpenClaw on Sol in this run includes one stream that timed out before its final usage, charged at the full reservation.

What should a ranking use, and what should it leave alone?

Use the tested cost calls, and leave the pass rates unranked.

The sentence the files support: on this public 20-task set, pass rate did not separate the harnesses, and a second run found OpenClaw more expensive per trial than Claude Code on Opus 5.5 and more expensive than Codex on GPT-6.1 Sol. Hermes Agent's Opus cost per trial could not be separated from Claude Code's. Codex was cheaper per trial than Hermes on Sol. QM's lower Opus point estimate was not tested on a shared network.

The sentence the files do not support: Hermes won, or Claude Code beat OpenClaw at the task, or 78.8% is better than 71.3% in a way the experiment could see. The intervals on the pass rates are 32 to 39 points wide. The report says so.

A reader who wants a harness for a reason other than this task set has other pages, and they answer different questions. They are collected on the agent harnesses hub. How do you evaluate an AI agent harness? is the method: one visible action, then the whole job. What is an AI agent harness? is the definition. Best AI agent harness is licence, list price, and platform from official pages, last verified November 2026. It does not use these pass rates. The November harness benchmark report copies official SWE-bench and Terminal-Bench numbers. Terminal-Bench 2.1 is a repair of 28 of 89 tasks in a different suite. HarnessFlow searches for a harness and scores it against pinned Claude Code, Codex, OpenHands, and Hermes on that paper's benchmarks.

Claude Code's permission file is a different November 9 page. Don't ask again is about what a saved approval writes to disk. It is not a row in this matrix. The product pages are Claude Code and Codex.

The disclosures in the README, which this note is not in a position to audit, are these. Garry Tan runs the repository and the evaluation. gstack and gbrain are his projects, and they are not in the seven cells. QM is a project of Y Combinator, which he runs. Y Combinator is an investor in Hermes. OpenAI began through YC Research. OpenAI is now the key benefactor of OpenClaw, and OpenClaw's creator works for OpenAI. The report says the receipts are in the tree so the trials can be regraded. This note did not regrade them. It copied the tables the repository had already published.