nokast-secureRAG loses detection on external factual poisoning despite strong internal scores

CLM-NOKAST-DISTRIBUTION-GAP reports: The nokast-secureRAG paper reports 0.994 recall and 0.026 false positives on an internal test, but near-zero recall on external PoisonedRAG. Source SRC-2026-RADAR-NOKAST-DISTRIBUTION-GAP is located at 'Abstract; held-out in-distribution results, ASR proxy and external PoisonedRAG evaluation'. The distinctive mechanism begins: The nokast-secureRAG authors trained a small detector for malicious instructions that conflict with the task and retrieval context, reporting 99.4% recall and 2.6% false positives on their own instruction-injection distribution. Classification: setting=evaluation, exploit_status=demonstrated, evidence_grade=B on the bounded original investigator/vendor or paper evidence, without independent dynamic replication. remediation_status=not_applicable; scoped control or limit: State detector scope and evaluate factual poisoning with its own benchmarks or verification layer. A very high recall on injected instructions cannot be transferred to retrieval-time misinformation without testing. The null impact band avoids an ungrounded cross-Case severity rating; observed damage, targeted population and controlled test outcomes cannot be exchanged as the same quantity. Where a patch is described, its availability is separate from independently tested effectiveness.

Assessed Oct 23, 2026 using diggingbeagle.assessment/1.

Evidence basis: CLM-NOKAST-DISTRIBUTION-GAP · nokast-secureRAG preprint, arXiv 2610.28364