Evidence · DiggingBeagle record
Veritas executes counterexamples against six popular agent benchmark graders and documents systematic false passes
Original Veritas paper constructs executable counterexamples against task graders and audits selected subsets of Agents' Last Exam, tau2-bench, DeepSWE and other benchmarks. It also tests generated replacement verifiers, with model- and harness-dependent results; sample findings are not whole-leaderboard error rates.
- Source role
- primary disclosure
Evidence record
Original Veritas paper constructs executable counterexamples against task graders and audits selected subsets of Agents' Last Exam, tau2-bench, DeepSWE and other benchmarks. It also tests generated replacement verifiers, with model- and harness-dependent results; sample findings are not whole-leaderboard error rates.
Claim-level citations (2)
- supportsVeritas found agent benchmarks awarding credit to materially wrong outcomes: Veritas reported grading failures in 27/82 audited Agents’ Last Exam tasks, 57/146 tau2-bench tasks and 36/106 eligible DeepSWE tasks under executed counterexamples.
Section 5, benchmark-grader audit and counterexamples
- supportsVeritas found agent benchmarks awarding credit to materially wrong outcomes: The same Veritas study reports variable gains from generated executable graders, including three stronger-model tests where its baseline outperformed the generated-verifier setup.
Generated Evaluator; results across coding harnesses and eight comparisons