A plausible AI answer can carry information from the wrong source

A model output can be well written while carrying information from the wrong source. October research describes three very different routes: another patient's speech entering a clinical note, retrieval context changing which memorized training fragments are extractable, and a compromised inference server encoding protected weights through otherwise plausible token choices. Only the first addresses accidental cross-encounter content in generated documents; the other two concern distinct disclosure threat models.

These mechanisms demand different forensic questions. The clinical case asks who said the words. The memorization work asks which training continuation the model may reproduce under this context. The covert-channel work asks whether the server generating the tokens is honest. None can be resolved by a single generic content classifier.

In 576 simulated patient-clinician dialogues, the clinical documentation study found incidental remarks incorporated into about 35% of generated notes, with clinically relevant misattribution in 3.7%. The separate 57-recording mock-audio experiment showed a wider gap between transcription and note contamination: 48.2% versus 5.3% at the tested background volume. A transcript is not the final clinical record, but downstream summarization does not reliably establish speaker identity. The danger does not require any malicious instruction. The study does not measure real-world patient harm.

The retrieval-context investigation fixed training-text prefixes and scored corresponding suffix probabilities across three open-weight instruction-tuned models. Adding relevant or irrelevant retrieved documents left a robust core of memorization and shifted other cases near an extraction threshold in both directions. A lower aggregate extraction score could therefore hide newly extractable individual examples. These were probability measurements of known training continuations, not evidence of actual private files emitted by a production retrieval system. See the underlying paper.

The provenance question here is subtler than whether a cited retrieved document contains the sentence. Some text may come from learned training continuations, and its extraction likelihood can depend on neighboring retrieved material. Auditing only isolated prompts would miss that conditional behavior.

The model-weight channel paper assumes an attacker already controls part of the inference-serving path. It proposes a statistical detector that compares output behavior against calibrated benign variation and accumulates weak deviations across responses. Numerical nondeterminism makes isolated token mismatches insufficient proof; the stated false-alarm bounds require a calibration-transfer assumption. The study uses four models and constructed seed-aware and seed-blind attackers, not a documented real-world weight theft. Its source is therefore evidence of a possible channel and a tested detection method, not an observed campaign.

The practical distinction is to bind claims to the layer at which provenance can be established: speaker and encounter for clinical capture, retrieved-versus-memorized content for language generation, and server integrity for token emissions. A convincing sentence by itself proves none of those origins.