The reported comparison challenges the use of a better benchmark score as evidence that a model has become safer. Benchmark decontamination still left high attack success in the tested settings. The finding concerns controlled training modifications, not proof that a named public model is compromised. Mechanism, preconditions and effects. BenchDoor attacks…
Inspect the ClaimsCase · DiggingBeagle record
BenchDoor shows benchmark improvement can coexist with training backdoors
Scope
Author-designed backdoor experiments; model, benchmark and trigger choices bound the result.
Assessment profile
Assessment method- Setting
- evaluation
- Exploitation
- demonstrated
- Evidence
- Grade B
- Remediation
- not applicable
- AI role
- AGAINST AI
Basis and provenance
CLM-BENCHDOOR reports: BenchDoor reports benchmark gains in all 30 tested configurations while poisoned training produced attack success of 82.33% to 100% at 2% to 10% poisoning. Source SRC-2026-RADAR-BENCHDOOR is located at 'Abstract; tested configurations, poisoning rates and decontamination result'. The distinctive mechanism begins: BenchDoor attacks the incentive to incorporate data that improves public evaluation scores. Classification: setting=evaluation, exploit_status=demonstrated, evidence_grade=B on the bounded original investigator/vendor or paper evidence, without independent dynamic replication. remediation_status=not_applicable; scoped control or limit: Evaluate training-data provenance, hidden-trigger behavior and holdout attacks independently of public benchmark gains. Do not treat a higher evaluation score as evidence that the supplied dataset is trustworthy. The null impact band avoids an ungrounded cross-Case severity rating; observed damage, targeted population and controlled test outcomes cannot be exchanged as the same quantity. Where a patch is described, its availability is separate from independently tested effectiveness.
Assessed Oct 23, 2026 using diggingbeagle.assessment/1.
Evidence basis: CLM-BENCHDOOR · BenchDoor preprint, arXiv 2610.27533
At a glance
Independent reproduction and transfer to unrelated training pipelines remain unestablished.
Limits and uncertaintyFull account
The reported comparison challenges the use of a better benchmark score as evidence that a model has become safer. Benchmark decontamination still left high attack success in the tested settings. The finding concerns controlled training modifications, not proof that a named public model is compromised.
Mechanism, preconditions and effects. BenchDoor attacks the incentive to incorporate data that improves public evaluation scores. An adversary can supply poisoned training material or manipulate benchmark content so that a model appears to improve on the target test while learning a trigger-controlled behavior. Across 30 reported controlled configurations using Qwen and Gemma families, authors reported positive benchmark gains alongside held-out backdoor success between 82.33% and 100% at 2% to 10% poisoning.
Comparison and evidence limits. High measured trigger success is conditioned on the authors' chosen setting and does not establish compromise of deployed model checkpoints. The paper reports success even after selected similarity-based decontamination, so superficial duplicate removal is not demonstrated as sufficient, but the experiments do not prove every independent contamination defense fails.
Timeline
- Oct 20, 2026disclosure
Public disclosure
Claims & evidence
CLM-BENCHDOORBenchDoor reports benchmark gains in all 30 tested configurations while poisoned training produced attack success of 82.33% to 100% at 2% to 10% poisoning.supported
Basis: reported finding
- supportsBenchDoor preprint, arXiv 2610.27533primary disclosure
Abstract; tested configurations, poisoning rates and decontamination result
Controls and mitigations
- Evaluate training-data provenance, hidden-trigger behavior and holdout attacks independently of public benchmark gains. Do not treat a higher evaluation score as evidence that the supplied dataset is trustworthy.
Unknowns and contradictions
- Independent reproduction and transfer to unrelated training pipelines remain unestablished.
Sources and citation
Material revision history
- Oct 23, 2026 · Canonical change recorded · new in release · revision 183