Veritas found agent benchmarks awarding credit to materially wrong outcomes

A benchmark score can be internally repeatable without checking that the task was accomplished. Veritas tested existing grading code by constructing executable counterexamples rather than relying on impressions from model explanations. Several widely used agent evaluations accepted defective actions or missed correct equivalents.

The audit found 27 of 82 audited Agents’ Last Exam tasks and 57 of 146 audited tau2-bench tasks with discriminating failures. In one coding benchmark, 36 of 106 eligible tasks gave full credit after an entire patch hunk was removed. Its second contribution generates task-specific executable graders, but improvements vary by model and harness. Sample denominators are audited subsets, not the whole benchmark population; no global leaderboard reversal has been proved.

An agent evaluation that cannot distinguish wrong behavior from completed work may reward unsafe automation.

A separate contribution in the same Veritas paper compiles task-specific executable graders from allowed evidence and procedural rules. The authors report improvements in five of eight selected harness comparisons, but a stronger GPT-5.6 Sol baseline beat the generated-verifier harness in three comparisons. That is not evidence that Veritas always wins or that agreement with an existing flawed benchmark grader proves ground truth.

The denominators describe audited or eligible subsets, not all tasks in each benchmark. A defective verifier can invalidate a specific success claim without proving a complete leaderboard inversion. The follow-up should rerun archived agent outputs against human-certified effects and near-miss counterexamples, rather than rely on another model's optimistic verdict.