Published poison collections prepend the target question or have unnatural length distributions. A classifier can learn those regularities and score highly without recognizing a realistically worded instruction or misleading evidence. The authors report RAGuard AUROC declining from 0.953 to 0.758 after deleting a question-prefix cue from their comparison…
Inspect the ClaimsCase · DiggingBeagle record
RAG poison filters lost detection after attacker-text construction cues were removed
An evaluation shows that some near-perfect RAG defenses detect benchmark formatting artifacts instead of robustly identifying malicious retrieved content.
Scope
The measurements evaluate candidate filters and curated corpora. They do not prove every filter is bypassable or quantify real production poisoning.
Assessment profile
Assessment method- Setting
- evaluation
- Exploitation
- demonstrated
- Evidence
- Grade B
- Remediation
- not applicable
- AI role
- AGAINST AI
Basis and provenance
The authors remove an artificial target-question prefix while retaining target retrieval, then measure detector AUROC and operating-point misses. The finding concerns benchmark construction and threshold behavior, not a confirmed attack on a deployed RAG service. Impact is left ungraded because a calibrated 0-to-4 impact basis for actual affected targets is unavailable from these Claims.
Assessed Oct 28, 2026 using diggingbeagle.assessment/1.
Evidence basis: CLM-CP22-015-A · CLM-CP22-015-B · CLM-ENRICH-20261028-013 · Ranking Is Not Deciding: Benchmark Shortcuts in RAG Poisoning Defenses
At a glance
Behavior under adaptive payloads with verified human labels and fixed false-positive budgets is not established.
Limits and uncertaintyFull account
Published poison collections prepend the target question or have unnatural length distributions. A classifier can learn those regularities and score highly without recognizing a realistically worded instruction or misleading evidence.
The authors report RAGuard AUROC declining from 0.953 to 0.758 after deleting a question-prefix cue from their comparison material. The same controlled change raises RAGuard miss rate from 0.191 to 0.763 while target retrieval remains high at approximately 0.93. Primary evidence.
The measurements evaluate candidate filters and curated corpora. They do not prove every filter is bypassable or quantify real production poisoning.
Test defenses against cue-free, length-matched poison sets and measure missed attacks at an actual deployment threshold rather than AUROC alone.
Claims & evidence
CLM-CP22-015-AThe authors report RAGuard AUROC declining from 0.953 to 0.758 after deleting a question-prefix cue from their comparison material.supported
Basis: reported finding
- supportsRanking Is Not Deciding: Benchmark Shortcuts in RAG Poisoning Defensesprimary disclosure
Abstract; question-prefix removal experiment
CLM-CP22-015-BThe same controlled change raises RAGuard miss rate from 0.191 to 0.763 while target retrieval remains high at approximately 0.93.supported
Basis: reported finding
- supportsRanking Is Not Deciding: Benchmark Shortcuts in RAG Poisoning Defensesprimary disclosure
Abstract; fixed operating point and retrieval comparison
CLM-ENRICH-20261028-013Of 21 published defense evaluations surveyed, the authors verified that eight used poison with a target-question prefix, five substituted a length cue, and 12 omitted the false-positive rate at the deployed threshold.supported
Basis: reported finding
- supportsRanking Is Not Deciding: Benchmark Shortcuts in RAG Poisoning Defensesprimary disclosure
Abstract, construction-cue audit
Controls and mitigations
- Test defenses against cue-free, length-matched poison sets and measure missed attacks at an actual deployment threshold rather than AUROC alone.
Unknowns and contradictions
- Behavior under adaptive payloads with verified human labels and fixed false-positive budgets is not established.
Sources and citation
Material revision history
- Oct 28, 2026 · Canonical change recorded · new in release · revision 199