The report
A benchmark can score a task as completed while ignoring the action that would make completion real. The Veritas study audited agent evaluations by submitting executable near-miss outputs, missing patch hunks and altered answers to official graders. These are counterexamples to particular scoring rules, not allegations that a participant secretly manipulated a leaderboard.
What the counterexamples established
Among 82 audited headless tasks from Agents' Last Exam, 27 had at least one confirmed defect under the authors' execution-based checks. For the tau2-bench subset, 57 of 146 tasks failed meaningful discrimination under their probes. In DeepSWE, removal of an entire patch hunk still received full credit for 36 of 106 eligible tasks. An MCP-Atlas judge changed its verdict on identical input for 72 of 500 tasks and rejected correct paraphrases on 47. Each denominator belongs to its own audit procedure; none is a population-wide benchmark error rate.
A particularly revealing failure checked source-code strings rather than running a scientific script: a required line could be present in dead code while the produced survival data were wrong. Another accepted apparently correct database state without verifying the necessary conversational action. Both illustrate why outcome, process and authorization cannot be inferred from a broad task-pass indicator. See the Veritas Case.
Replacement graders are not automatically correct
Veritas also generated executable task-specific evaluators and reported improvements in some tests. Its results varied with model and harness, including stronger-model comparisons where an ordinary baseline performed better. A proposed replacement still needs independent labels, negative examples and validation of what tools were permitted to do. The strongest finding is the demonstrated inability of selected official graders to distinguish completed work from plausible counterfeits; global ranking consequences remain unmeasured.