GraceBench describes GUI workflows as transition graphs rather than grading only the final screen. Its ten application graphs contain 2,909 canonical states and 3,116 recorded transitions. The framework aligns observed video or UI state with those graphs and measures deviations, loops, stagnation and recovery over 3,718 execution trajectories. The key…
Inspect the ClaimsCase · DiggingBeagle record
Successful GUI-agent runs can conceal deviations and recovery failures
Scope
Benchmark trajectories on ten GUI applications, not a measured breach, unauthorized payment or generalized failure rate of production computer-use agents. Deviating from one recorded reference path does not necessarily violate task permissions.
Assessment profile
Assessment method- Setting
- evaluation
- Exploitation
- not established
- Evidence
- Grade C
- Remediation
- not applicable
- AI role
- AGAINST AI
Basis and provenance
The primary abstract supports the cohort, UI Transition Graph approach and instability distinction, while the retained Claim cites the 663-of-1294 breakdown. Full trajectory artifacts and scoring code were not independently checked. Impact remains ungraded because the available evidence does not establish realized harm or a defensible general severity for the assessed deployment.
Assessed Oct 16, 2026 using diggingbeagle.assessment/1.
Evidence basis: CLM-GRACEBENCH-TRAJECTORY-RECOVERY-01 · CLM-GRACEBENCH-TRAJECTORY-RECOVERY-02 · CLM-GRACEBENCH-TRAJECTORY-RECOVERY-03 · GraceBench: trajectory-level GUI-agent evaluation
At a glance
When an agent completes a task, reviewing only the final outcome can conceal detours that consumed time or changed unrelated state. Process metrics are therefore useful diagnostic signals, but they require external authorization rules and effect logs before being interpreted as security outcomes.
Read the implicationsThe rate of unauthorized external effects among trajectories marked as deviating has not been established. Alternative valid GUI routes may be penalized by reference-graph alignment despite achieving the intended task. Reproduction on other applications, operating systems and long-lived production sessions remains necessary.
Limits and uncertaintyFull account
GraceBench describes GUI workflows as transition graphs rather than grading only the final screen. Its ten application graphs contain 2,909 canonical states and 3,116 recorded transitions. The framework aligns observed video or UI state with those graphs and measures deviations, loops, stagnation and recovery over 3,718 execution trajectories.
The key distinction is between eventually finishing and reaching the goal by a stable route. Among 1,294 successful traces, 663 include a measured deviation or loop. The highest task-completion model in this cohort also has the weakest measured recovery. These are benchmark-defined path instabilities, not evidence that 51.24% of successful agents stole data or used an unapproved tool. A path can legitimately differ from the reference, while a harmful side effect might also occur on a superficially acceptable path. Permission and external-effect telemetry would be needed to distinguish the two.
Timeline
- Oct 14, 2026disclosure
Public disclosure
Claims & evidence
CLM-GRACEBENCH-TRAJECTORY-RECOVERY-01In GraceBench, 663 of 1,294 successful traces, or 51.24%, contained at least one measured deviation or loop.supported
Basis: reported finding
- supportsGraceBench: trajectory-level GUI-agent evaluationprimary disclosure
Section 4.3, successful trajectories with deviations or loops
CLM-GRACEBENCH-TRAJECTORY-RECOVERY-02The evaluation covers 169 tasks, ten applications and eleven models, with 3,718 trajectories across instruction conditions.supported
Basis: reported finding
- supportsGraceBench: trajectory-level GUI-agent evaluationprimary disclosure
Benchmark construction and experimental setup
CLM-GRACEBENCH-TRAJECTORY-RECOVERY-03GraceBench represents ten application interaction spaces as UI Transition Graphs with 2,909 canonical GUI states and 3,116 observed transitions for step-level diagnosis.supported
Basis: reported finding
- supportsGraceBench: trajectory-level GUI-agent evaluationprimary disclosure
Paper abstract, description of UI Transition Graphs and benchmark construction
Implications
When an agent completes a task, reviewing only the final outcome can conceal detours that consumed time or changed unrelated state. Process metrics are therefore useful diagnostic signals, but they require external authorization rules and effect logs before being interpreted as security outcomes.
Controls and mitigations
- Keep separate metrics for final completion, unnecessary interactions, recovery and permission violations.
- Compare identified deviations with allowed alternative routes and independently observed filesystem, network or transaction effects.
- Require explicit approval for irreversible actions regardless of a model's apparent progress or confidence.
Unknowns and contradictions
- The rate of unauthorized external effects among trajectories marked as deviating has not been established.
- Alternative valid GUI routes may be penalized by reference-graph alignment despite achieving the intended task.
- Reproduction on other applications, operating systems and long-lived production sessions remains necessary.
Sources and citation
Material revision history
- Oct 16, 2026 · Canonical change recorded · new in release · revision 160