Harness Matters: 32.55% web-agent attack success

What the six harnesses did under attack

The rates below are the authors’ measurements inside WebTrap Farm. An attack success means the unsafe action was carried out. The denominator drops tasks labeled capability failure, which are attacks the agent also could not complete in the benign twin. The paper says the overall column is a weighted average across attacks. It is not the unweighted mean of the three surface columns, and the six harness overalls do not themselves average to 32.55. That 32.55% is the aggregate the paper prints for the open-source runs, and it is also the bottom-right cell of the model table.

MUP is malicious user prompts. MPI is malicious prompt injection. DWD is deceptive website design. The two commercial rows use the product’s own model. They are not crossed with the seven backends.

SeeAct and Skyvern 2.0 are the highest open-source overalls, 42.31% and 41.99%. Agent-E is the lowest, 20.24%, and Skyvern 1.0 is next at 21.80%. The unweighted mean of those six overalls is 194.26 divided by 6, which is 32.38% to two decimals, not the printed 32.55%. The 32.55% figure is the paper’s weighted aggregate, also printed as Table 5’s overall cell. The ordering is not the same on every surface. Skyvern 1.0 is the highest on malicious user prompts, 32.75%, and the lowest on malicious prompt injection, 8.04%. Agent-E is lowest on user prompts, 8.48%, and still above 20% on both web-side surfaces. A harness that looks safer on one surface is not the safer harness on the others.

The commercial overalls, 16.85% and 15.14%, sit below every open-source overall. The nearest open-source overalls are Agent-E at 20.24% and Skyvern 1.0 at 21.80%. On deceptive website design the commercial rows are not the lowest: Gemini is 25.51%, above Agent-E’s 23.01% and Skyvern 1.0’s 24.02%, and Claude is 22.12%. The paper’s sentence is that the commercial systems show stronger end-to-end safeguards than the open-source harnesses and still have meaningful vulnerabilities under deceptive design.

The same harnesses, split by model

Table 5 pools the open-source harnesses and splits by the model inside them. The overall row is the 32.55% figure.

A low rate on malicious user prompts does not travel to the web. Claude-Opus-4.5 is 3.25% on user prompts and 36.03% when the instruction is in the page. GPT-5.2 is 5.52% on user prompts, 36.48% on prompt injection, and 33.97% on deceptive design. Qwen3-VL-30B is the reverse shape: 57.94% on user prompts, the highest cell in that column, and 27.78% on prompt injection, below several of the closed models. The paper’s reading is that model safety does not transfer uniformly across surfaces, so a model has to be scored inside the web-agent loop.

Defense success rate is the complement of attack success rate on this denominator. The printed grand overalls add to 100.00: 32.55% attack success and 67.45% defense success. That arithmetic is the two printed overalls. It is what the definitions imply when every non-capability-failure is one or the other.

Table 6 is defense success rate with the model held fixed. The paper’s example is the GPT-4o column: Agent-E 83.79%, Skyvern 2.0 42.89%, a gap of 40.90 points. Claude-Opus-4.5’s column overall is 77.10%, and Agent-E under that model is 75.19%, which is below GPT-4o and below Qwen3-VL-235B in the same harness.

Skyvern 1.0 is not uniformly strong. Under GPT-4o it is 61.94%, below both Browser Use rows. Under Claude-Opus-4.5 it is 91.55%, the highest cell in the table. The harness effect and the model effect are both visible in the same grid, which is what the regression is for.

The regression

Table 7 fits a mixed-effects logistic regression to the calibrated attack outcomes. Harness, model, their interaction, and attack category are fixed effects. Task is a random intercept. Degrees of freedom match a cross of six harnesses and seven models, with one level of each factor as the reference: 5, 6, and 5 times 6.

The model term is the largest chi-square. The harness term is still far past conventional significance, and so is the interaction, which is the statistical form of “this model is safer in one harness than in another.” The paper says all three p-values are below 10^−45. The checks it reports against the capability filter do not remove that conclusion: modeling whether a task was evaluable at all keeps all three effects below 10^−31, and relabeling capability failures at the extremes keeps them below 10^−39. Those checks are bounds. The paper does not print the exact p-values for them.

The repeats are a separate stability check. Three full runs of all six harnesses, on Qwen3-VL-235B and on GPT-5.1, have an average standard deviation of 1.73 percentage points. That is the paper’s statement about run-to-run variation under temperature 0. It is not a confidence interval on the 32.55% aggregate.

Attacks with no extra attack text

For the web-side sets, a naive variant adds no injected instruction and no extra deceptive sentence. The unsafe element can still be on the page. Figure 3 compares those naive runs with the runs that do include explicit attack content.

On malicious prompt injection, SeeAct’s naive attack success rate is 50.00%, against 49.37% for the other attacks. On deceptive website design, Browser Use (Text) is 41.27% naive and 36.30% on the other attacks. The paper says explicit attack wording can sometimes trip a model’s safety training, while a naive page gives the model fewer cues. The example in the text is a credit-card field on a course-selection task with no attack sentence. The agent treats the field as part of the task. The authors call that a harness-intrinsic failure: the loop does not check whether the action is necessary for the user’s goal before it executes.

Where a successful defense shows up

Section 5 reads logs from defense-success runs and marks the stage where the defense becomes effective. Conditional defense success rate is not the overall defense rate. It is the share of the remaining security-relevant tasks whose defense is localized at that stage, after defenses localized earlier have been removed. A zero at a later stage can mean the defense already happened, or that this stage never blocked anything. The tables print the component that harness uses at the stage.

Malicious user prompts

Raw-prompt forwarding localizes nothing at goal interpretation, which is what that design does: the user text is passed on. An LLM interpreter is not enough by itself. Agent-E localizes 82.71% there. Skyvern 2.0, which also has an LLM interpreter, localizes 8.13%. On 100 sampled user-prompt tasks with GPT-4o, swapping Agent-E’s interpretation prompt for the less detailed Skyvern 2.0 prompt moves that stage from 75.61% to 53.85%, a drop of 21.76 points. Those ablation rates are not the full-set rates in the table.

Planning rates do not line up with single-controller versus multi-role. Browser Use (Vision) is 70.31% and Agent-E is 69.50%, on different planner structures. The paper ties the difference to risk-awareness cues in the reasoning prompt. Removing those cues, still on 100 GPT-4o tasks, drops Browser Use (Vision) from 73.91% to 43.48%, which is 30.43 points, and Agent-E from 92.86% to 69.77%, which is 23.09 points.

At execution, the two LLM grounders still localize some defense, 26.94% and 23.51%. Every rule-based grounder in this table is 0.00%. The paper says a rule maps a planned action to a browser command without a semantic check, while an LLM grounder can trip a guardrail again. It also says those grounding prompts do not implement an explicit safety check, so the protection is incidental.

Web-side attacks

Agent-E localizes 78.99% of web-side conditional defense at perception. The paper associates that with structured extraction that filters page content before the planner sees it. The four multimodal perception rates in the table are all below 20%, and Browser Use (Vision) is the lowest at 9.96%. Relaxing Agent-E’s filter so every element is exposed, on 100 sampled web-side tasks with GPT-4o, moves the perception stage from 56.94% to 50.70%, a drop of 6.24 points. That is a smaller move than the user-prompt ablations.

Planning is where Skyvern 1.0 localizes 78.17%. Agent-E’s multi-role planner localizes 10.65% on this surface. The paper’s trace reading is that one role can summarize an injected instruction into a plan that the next role treats as trusted. Removing the goal-anchoring instruction from Skyvern 1.0 drops the stage from 70.73% to 54.05%, which is 16.68 points. Giving Agent-E’s planner the original user instruction, in addition to the upstream plan, raises the stage from 19.35% to 43.59%, which is 24.24 points.

Execution is thin on this surface. SeeAct is 1.63% and Skyvern 2.0 is 4.07%. Rule-based grounders are 0.00%. The paper’s explanation is that a web-side attack has already been broken into clicks and field fills that look legitimate one at a time, so the grounder rarely sees a reason to stop.

The case study is one Skyvern 1.0 trace on a job-application form that asks for a credit card at the end. The intent stage writes a guardrail against sensitive information. Early attempts refuse the field. After the agent misses the completion signal and retries, it later fills the card number, with the same guardrail still in the prompt. The paper treats that trace as evidence that prompt-level guardrails can erode over a long execution. It does not report how often that pattern occurred.

How this compares with harnesses as they stand on November 10

The paper’s open-source rows are configurations, not the November 10 default branch. Browser Use showed 120,474 stars on the GitHub API that day, with a push at 02:34:17 UTC, and the repository description names both the Python library and Browser Use Cloud. Skyvern showed 23,376 stars and a push at 05:01:12 UTC. The paper does not name the commit it ran. A star count is not a security score. It is evidence that the projects in the table are still the public projects, and that their trees have moved since a paper that does not pin a revision.

The commercial pins are older than the docs loaded the same day. Google’s computer-use guide lists four models: gemini-4-argon, gemini-3.8-flash as the recommended model, gemini-3.5-flash-lite, and gemini-3-flash-preview. gemini-3.7-flash is not one of those four names. The page also describes opt-in screenshot scanning for prompt injection, and configurable safety policies. Anthropic’s computer-use tool page shows an example with claude-opus-5-5 and the toolset computer_toolset_20260801. The paper’s Claude row is claude-haiku-4-5-20251001. A lower attack rate on those pins is not a measurement of the November 10 APIs. This site’s computer-use guide is about whether a product can drive a screen. It is not a security table, and it should not be read as one.

Task-score harness work answers a different question. HarnessFlow searches for an executable harness with a higher score on six benchmarks, with the model held fixed. WebTrap Farm does not search. It scores named harnesses on whether an unsafe action happened. A harness that wins a task benchmark can still be the one that finishes the unsafe click. Skyvern 2.0’s 85.25% task-completion rate next to a 58.01% defense success rate is the paper’s own illustration.

The guide to evaluating a harness separates a check of one visible action from an end-to-end task score. WebTrap Farm is end-to-end, and the end is an unsafe action. Someone buying or building a web agent can use both: a task score says the job can finish, and a calibrated attack rate says how often a capable run does the wrong job. Neither number is a substitute for a log of the specific action a deployment must block, such as a payment or a password.

What the experiment controlled

Names and numbers in this section are printed in the November 9 paper unless a link says otherwise.

The attack set is three categories. Malicious user prompts are 160 tasks, built from five goals, eight base prompts each, and four content strategies. Malicious prompt injection is 158 tasks. Deceptive website design is 187 tasks. The three counts in Table 11 sum to 505. Methods and contents were included only when at least two earlier papers had tested them. Web-side tasks include a naive content type. One construction exception is printed: default selection is not paired with sensitive-information disclosure, because a preselected option is not a form the agent has to fill.

Scoring for clicks, downloads, permissions, and form values uses the page instrumentation. Open-ended text uses GPT-4o, checked on 672 of 6,720 outputs. The usability twin keeps the interaction and removes the adversarial trigger. For a malicious post, the twin asks for a benign post on the same kind of page. For a web-side download, the twin removes the malicious content and asks for the download directly. Defense success requires the twin to succeed.

Agents run at their documented defaults. The authors say they do not change core architecture, reasoning logic, or built-in safety mechanisms. Temperature is 0. One completed execution is required per task. The three-repeat check covers two of the seven models. Logs used in Section 5 are enabled on the open-source agents and, the paper says, do not change runtime behavior. Commercial agents are scored from the page, because their internal stages are not available. That is why the stage tables do not include the two commercial rows.

The survey behind Table 1 is 68 GitHub projects with at least 1,000 stars, plus 40 papers. The six evaluated harnesses are a subset chosen to cover the component types, not a random sample of the 68. SeeAct is raw-prompt, multimodal, single-controller, and LLM-grounded. Both Browser Use configurations are raw-prompt and rule-grounded, one textual and one multimodal. Agent-E is an LLM interpreter, textual perception, multi-role planning, and a rule grounder. Skyvern 1.0 is an LLM interpreter, multimodal, single-controller, and rule-grounded. Skyvern 2.0 is an LLM interpreter, multimodal, multi-role, and LLM-grounded.

Limits that change how to read the rates

The paper’s limitations section names three bounds. The attack set is three categories, so a tool-poisoning result is not in these tables. The sites are controlled, so a live page with changing layout is not in these tables. Runs vary, and the authors’ answer is the 1.73 percentage-point average standard deviation on the repeated slice, plus temperature 0 and a large task set.

The denominator is the second bound, and it is in the metric definitions rather than the limitations paragraph. Attack success rate excludes capability failures. Quoting 32.55% as “the agent was attacked successfully on 32.55% of tasks” adds a claim the table does not make. The paper never prints the capability-failure count that would turn 32.55% back into a share of all tasks.

The commercial comparison is a third bound. Two pinned model names, not the model list on the vendor docs as of November 10, sit below the open-source overall and still fail deceptive design at 25.51% and 22.12%. Google’s current guide also offers an opt-in prompt-injection scan that this paper does not turn on or off, because that control is not part of the gemini-3.7-flash setup it ran.

Appendix E draws a line the results section is easy to slide past. The prompt swaps and the filter relaxation are controlled ablations on 100 tasks. The broader contrasts, text versus screenshot, one planner versus several roles, rules versus an LLM grounder, are differences between whole harnesses that also differ in other ways. Agent-E’s low attack rate is not, by itself, an estimate of what textual perception would do inside Skyvern.

The code release does not contain the result databases. The repository was last pushed October 8, 2026. Until those databases are public, the tables are the authors’ report of their own runs. The ethics appendix says the paper does not release exploit-ready procedures. This page follows that boundary: the categories are named, and the payload text is not.

Read the paper

The November 9 paper, the code, and the two computer-use docs checked against the commercial pins:

• Harness Matters on arXiv
• HTML
• PDF
• WebTrap Farm
• Google computer use
• Anthropic computer use tool
• Browser Use
• Skyvern

Related pages on this site: agent harnesses hub, how agents work, task performance, research, What is an AI agent harness?, How do you evaluate an AI agent harness?, Can an AI agent use your computer?, HarnessFlow, harness, harness engineering, guardrail, tool use, benchmark, Best AI agent harness, and the November harness benchmark.