Paper breakdown: MCPHallu — why MCP agents fail on unreachable goals

What does the main table say about the 14 models?

Table 5 is the result. Scores run from 0 to 1, and higher is better. The paper prints a mean of 0.642 and a pass rate of 47.8%. The best overall cell is 0.728. The introduction says no model scores above 0.728 on average. The 14 overall numbers below add up to 8.992, and 8.992 divided by 14 rounds to 0.642. Bootstrap intervals are in the paper’s Appendix F.5. The main text says the overall intervals have a half-width of about 0.03 to 0.04. One interval from that appendix, for the leader, is printed here because it is the number a reader will try to over-read: Claude Opus 4.6 overall is 0.728, with a 95% interval of 0.695 to 0.759.

The HTML and the PDF agreed on every cell of Table 5 when both were read on November 8, 2026.

The highest cell in each column, read off that table: overall 0.728 and Context Forgetting 0.730 and Tool Misuse 0.885 and Medium 0.707 and Hard 0.687 are Claude Opus 4.6. Branch Collapse 0.867 and Easy 0.808 are Claude Opus 4.5. Unreachable Goal 0.585 is Claude Sonnet 4.6. GLM-5.1 is the lowest overall score, 0.539. Gemini 3 Flash is the lowest Unreachable Goal score, 0.284.

Unreachable Goal is the lowest of the four subtype scores on 12 rows. On Qwen3.5-397B, Context Forgetting is 0.445 and Unreachable Goal is 0.539. On Kimi-K2.6, those two cells are 0.454 and 0.498. That is the “12 of the 14 models” sentence in section 5.2, visible in the table. Easy to Hard falls on 13 rows. Claude Sonnet 4.5 is the row that does not: Medium is 0.651 and Hard is 0.686.

The model names are the paper’s names. This table is not a claim about any model the November 5 paper left out.

What are the four subtypes, and how are they detected?

Table 2 is the taxonomy. Two subtypes share the reasoning axis and use different triggers. The paper says that is why both stay in the benchmark: across models, Branch Collapse and Unreachable Goal correlate at r = -0.12.

The paper says the set is representative, not exhaustive, and that subtypes can co-occur in one trajectory. This release of Tool Misuse instantiates wrong tool selection. Faulty invocation, with the right tool and invalid arguments, is listed as future work. Context Forgetting covers a stale, approximate, or substituted value.

A behavioral evaluation checks one observable action. MCPHallu is a benchmark: a fixed set of tasks, a scorer, and a comparison across models. The glossary entry for the protocol the agents speak is Model Context Protocol. The loop they ran inside is the one the paper cites from ReAct. The note on that paper is ReAct.

Where do the 358 tasks and 36 servers sit?

Table 3 is the domain cut. The representative servers are examples, not the full server list. Table 8 later names all 36 and totals 296 tools. Section 4.2’s phrase for the same inventory is “roughly 300 tools.”

Fourteen plus 6 plus 5 plus 9 plus 2 is 36. Seventy-five plus 80 plus 69 plus 61 plus 73 is 358. Figure 6’s caption says Unreachable Goal is the largest subtype, at 103 tasks, and the other three range from 82 to 89. The three feasible subtypes are the other 255 tasks. The paper says that in the appendix, and 358 minus 103 is 255. Table 6’s 1,442 Unreachable Goal runs are 103 times 14. Table 7’s 1,176 Tool Misuse runs are 84 times 14, so Tool Misuse is the subtype of 84 tasks inside that 82-to-89 range. This page does not assign the remaining 89 and 82 to Branch Collapse and Context Forgetting, because the caption states the range and does not print those two counts in a sentence.

Servers run in Docker with a sandboxed filesystem and allow-listed network egress. A fresh container starts each episode, so a stateful write does not leak into the next run. Credentials are per server in Table 8. Some need an API key, some none, MongoDB a connection string, Google Workspace an OAuth token. The paper says no credentials are distributed with the release.

Construction is three stages. Tool grounding reads schemas from the live servers. Adversarial synthesis uses Claude Sonnet 4.6, with a subtype-specific prompt, to write candidates that have a checkable correct behavior. Filtering checks executability, a real structural trigger, self-contained instructions, non-triviality, and label consistency. A Claude Code solver on Claude Sonnet 4.6 runs the label check, separate from the GPT-4o judge, so one model is not both the only writer and the only grader. Three annotators, each with at least two years on agent evaluation, review tasks that survive. A task is accepted at 2 or better, on a 0-to-2 scale, for realism, label, and ground truth, from at least two of the three. The benchmark keeps 73% of the candidates that were generated.

Why do unreachable goals fail?

The abstract’s sentence is the finding: agents often continue issuing tool calls, or claim successful completion, rather than recognizing infeasible goals. The average on this subtype is 0.458. The pass rate is 23.2%.

Table 6 bins every Unreachable Goal run by how many tools the agent called. Pass rate falls as the call count rises. Completion language in the final answer, words such as “successfully,” “saved,” and “created,” moves the other way through the 8-to-15 bin, then drops at 16 or more.

The paper’s prose says the pass rate is 0% once the agent issues four or more calls. That matches the last three rows. It does not match a reading of Figure 7 as the same table. Figure 7’s counts are 342, 475, 283, 182, and 264, and its zero-call pass rate is 65.2%, not 65.0%. Those figure bins add to 1,546. The table bins add to 1,442. This page treats Table 6 as the source for these percentages.

The early-stop rule in the scorer is built for this pattern. For Unreachable Goal, the structural score is 1 if the agent declares infeasibility within a probing window of k = 2 tool calls, then decays, and it is 0 if the agent never declares infeasibility. The window is there so a short probe is not punished. A long trajectory that never says “this cannot be done” gets nothing from that component.

Appendix Table 13 prints a coarser cut: 307 runs that keep acting for 4 or more calls with no infeasibility declaration, pass rate 0.0%; 210 runs that keep acting for 1 to 3 calls, pass rate 0.0%; 448 runs that stop and declare infeasibility, pass rate 48.0%. Those three rows sum to 965, not to 1,442, so they do not cover every run in Table 6. Table 14, limited by its caption to non-empty responses, is the language cut: completion only, 195 runs, 2.1% pass; limitation only, 304 runs, 44.4% pass, mean 0.668. The four language rows sum to 1,366.

The useful reading is the one the bins agree on, even where the counts do not. Stopping early and saying the goal cannot be met is associated with passes. Continuing, and talking as if the task finished, is associated with zeros. Blanket refusal is not the same skill. The three feasible subtypes are 255 of the 358 tasks and are scored by completion. On Tool Misuse, calling neither the correct tool nor a confusion tool passes 8.0% of the time.

What does tool misuse look like once the right tool is in the trace?

Table 7 splits Tool Misuse trajectories by which tools were actually called. Finding the right tool is not the result. Avoiding the near-miss tool is.

Correct only versus correct plus confusion is 84.7% versus 32.3%, with the required tool present in both. Mixed traces are also the long ones: 10.70 steps against 3.54. The confusion score penalizes the set of distinct confusion tools, not the number of times one of them is retried. One look at the wrong tool costs the same, on that term, as several looks at the same wrong tool.

The ideal trace the appendix describes is short. Tool Misuse tasks have a sufficient call sequence with a median of 2 calls, range 2 to 5. Branch Collapse and Context Forgetting sit at a median of 7. Nineteen of the 84 Tool Misuse tasks need more than one distinct tool, for example a log lookup before the call that answers the question. The appendix’s worked contrast is git_git_show on a commit, which returns that commit’s patch, against git_git_diff with target HEAD, which answers a different question.

Context Forgetting has its own split, in Appendix Table 15. Stateless runs, in General, Analytics, and Financial, pass 59.2% of the time, mean 0.713, 7.17 steps, over 644 runs. Stateful runs, in Productivity and Coding, pass 24.6%, mean 0.431, 11.46 steps, over 504 runs. The paper’s point is that the memory failure is often a failure to write the remembered value into the artifact the task scores, not only a failure to hold the value in the context window.

Why is the overall leader not the leader on every subtype?

Table 16 ranks each model inside each subtype. Rank 1 is best. Spread is the gap between a model’s best and worst subtype rank. The mean spread is 6.86 out of a possible 13. The abstract’s “6.9 of 13” is that figure rounded. Claude Opus 4.6, the overall leader, is rank 7 on Branch Collapse, rank 1 on Context Forgetting, rank 2 on Unreachable Goal, and rank 1 on Tool Misuse. The discussion section says this in one line: the overall leader ranks seventh on Branch Collapse.

DeepSeek-V4 is the sharp version of the same fact. It is rank 2 on Branch Collapse, with a Table 5 score of 0.853, and rank 13 on Unreachable Goal, with a score of 0.302. A team that picks a model from the overall column will not have picked the best model for every failure mode in this paper.

The attribution audit is the check on whether the label and the observed failure are the same thing. Appendix Table 9 counts 80 failed runs. Rows are the intended subtype. Columns are what the auditors saw. The diagonal is 16, 21, 13, and 17, which is 67. The paper reports kappa 0.78 for that audit.

Eight of the 13 off-diagonal runs land in the Unreachable Goal column. The paper’s note is that a model which gets stuck on a feasible task sometimes stops and wrongly reports the goal as unreachable. That is a real overlap, and it is why 0.642 should not be read as four cleanly separated abilities that never touch.

What the evaluation held fixed

The agent framework is a ReAct-style loop for every model. The context is 128K tokens even when the provider’s native window is longer. Gemini 3 Flash’s native window in the inference table is 1M. GPT-5-chat’s is 400K. DeepSeek-V4’s is 128K. All of them run the benchmark at 128K. The tool-call budget is 20. Temperature is 1.0 in every row of that table. Environments reset between episodes. The paper says a plan-and-execute scaffold and an infeasibility-aware prompt, checked in an appendix, leave the two hardest subtypes the same.

The vendors and the context windows, from Appendix Table 12:

Fourteen models times 358 tasks is 5,012 trajectories, which is the count in the abstract. One run per model per task is the arithmetic of that sentence.

Expert humans, on the stratified 40-task sample, score 0.88. The paper sets the ideal agent at 1.0, the score for never triggering the designed hallucination, and says the humans come close. The comparison it prints against the best model uses the full-benchmark 0.728, not a 40-task score for that model. The Unreachable Goal gap is printed as 0.82 versus 0.58. Table 5’s full-set figure for Claude Opus 4.6 on that subtype is 0.576.

Where the repository and the paper disagree

The paper’s code link is J1zz/mcp_hallu. On November 8, 2026, the GitHub API reported the repository public, default branch main, created February 27, 2026, last pushed May 7, 2026, with 4 stars and 0 forks. The license SPDX id was Apache-2.0. The LICENSE file on main is the Apache License, Version 2.0. The README badge on that same file says MIT. The README also still says the submission is double-blind and that author information is withheld. The November 5 paper names nine authors. The README’s tool line is 302. Table 8’s total is 296, and adding the per-server tool counts in that table also comes to 296.

The README displays a NeurIPS 2026 E&D Track badge that links to neurips.cc. The paper read for this note is the arXiv preprint submitted November 5, 2026. The badge is not, by itself, a sentence in that paper saying the work was accepted.

Anyone reproducing the tables should pin the paper, not the README’s summary statistics. The tasks and the scoring code may still be in the repository. The license badge, the anonymity notice, the tool count, and the push date are older than the preprint this page is about.

How to read it next to the other notes on this site

MCPHallu holds the agent loop still and changes the task, so the failure mode is the thing that varies. HarnessFlow searches for a different harness program. Evolver keeps Claude Code and Codex and injects a strategy asset from an earlier trace. Neither paper’s table is a row in Table 5, and Table 5 is not a score for those systems.

What is an AI agent harness? is the definition of the loop MCPHallu did not search over. How do you evaluate an AI agent harness? is the guide for checking one action inside that software. A harness here is that runtime. MCPHallu’s judge-and-script scorer is a benchmark of model behavior in a shared loop, not a ranking of harness products.

What is BioBasicsBench? is the other new benchmark note on this site, from November 8. It scores raw wet-lab evidence. It does not call MCP tools, and its model list is a different list. The benchmark tracker is where a cell goes when this site has a source, a date, and the harness that produced it. MCPHallu’s cells live on this page, with the November 5 paper as the source.

The paper, the HTML, the PDF, and the repository:

• MCPHallu on arXiv
• HTML
• PDF
• J1zz/mcp_hallu

Related pages: research, harnesses, how agents work, task performance, Model Context Protocol, benchmark, and behavioral evaluation.