If an attacker already controls an LLM inference server, the generated text stream can become a covert output channel for protected model-weight information. The study considers attackers encoding bits through alternative but plausible token choices. Replaying the same prompt on trusted infrastructure can reveal deviations, yet numerical nondeterminism and…
Inspect the ClaimsCase · DiggingBeagle record
Output-token choices can conceal weight exfiltration in compromised model serving
A compromised inference server could encode protected model information in otherwise plausible output-token choices. A sequential statistical detector accumulates weak deviations across responses under a calibration assumption. The paper tests constructed attacks, not confirmed production weight theft.
Scope
Controlled finding or reported disclosure as of 2026-10-08; Constructed lab threat, not documented production weight theft.
Assessment profile
Assessment method- Setting
- simulation
- Exploitation
- demonstrated
- Evidence
- Grade B
- Remediation
- not applicable
- AI role
- AGAINST AI
Basis and provenance
Desk review of scoped canonical Claims, the pinned SRC-2026-CP3-WEIGHT-OUTPUT-COVERT-CHANNEL Source, and the publicly accessible primary disclosure. The classification applies to the documented setting and prerequisites only. Impact remains unbanded because real victim consequences are not established, and no independent reproduction is claimed. Details: No verified in-the-wild model-weight exfiltration through this channel was described.
Assessed Oct 13, 2026 using diggingbeagle.assessment/1.
Evidence basis: CLM-CP3-WEIGHT-OUTPUT-COVERT-CHANNEL-1 · CLM-CP3-WEIGHT-OUTPUT-COVERT-CHANNEL-2 · CLM-CP3-WEIGHT-OUTPUT-COVERT-CHANNEL-3 · LLM output token choices become covert channel for stolen model weights
At a glance
Protect model-serving integrity and monitor output distributions rather than relying on content filters that read only the semantic answer. A statistically calibrated detector can accumulate weak evidence, but it cannot prove that an unmonitored compromised server remained trustworthy.
Read the implicationsNo verified in-the-wild model-weight exfiltration through this channel was described. Detection depends on representative benign calibration and serving conditions that may change across hardware, seeds or decoding paths.
Limits and uncertaintyFull account
If an attacker already controls an LLM inference server, the generated text stream can become a covert output channel for protected model-weight information. The study considers attackers encoding bits through alternative but plausible token choices. Replaying the same prompt on trusted infrastructure can reveal deviations, yet numerical nondeterminism and near-tie token probabilities also produce benign mismatches. Observing a different token is therefore not by itself proof of exfiltration.
The researchers use a prompt-level sequential e-process to combine whole-response mismatch evidence across outputs, calibrating against trusted benign traffic. Their analysis tests four models and distinguishes a seed-blind attack from a stronger seed-aware variant. False-alarm control depends on a calibration-transfer assumption; no production weight theft or deployment-wide detection reliability was established. This mechanism is different from asking an uncompromised model to print its weights through a prompt.
Mechanism and trust boundary
- 01
Compromised serving control
An attacker has the ability to change token selection in the inference runtime while preserving broadly plausible responses.
Boundary: An attacker has the ability to change token selection in the inference runtime while preserving broadly plausible responses.
- 02
Covert encoding
The server biases near-equivalent output tokens to carry protected information across requests.
Boundary: The server biases near-equivalent output tokens to carry protected information across requests.
- 03
Detection boundary
Trusted replay and sequential mismatch testing aggregate deviations while accounting for benign sampling and numerical variation.
Boundary: Trusted replay and sequential mismatch testing aggregate deviations while accounting for benign sampling and numerical variation.
Timeline
- Oct 8, 2026disclosure
Public disclosure
Claims & evidence
CLM-CP3-WEIGHT-OUTPUT-COVERT-CHANNEL-1The researchers model an adversary who can perturb generated token choices to encode model-weight information in otherwise plausible responses.supported
Basis: reported finding
- supportsLLM output token choices become covert channel for stolen model weightsprimary disclosure
Abstract, threat model and exfiltration mechanism
CLM-CP3-WEIGHT-OUTPUT-COVERT-CHANNEL-2The study evaluates anytime-valid statistical tests to distinguish suspicious output changes from routine sampling variation.supported
Basis: reported finding
- supportsLLM output token choices become covert channel for stolen model weightsprimary disclosure
Abstract, anytime-valid detection analysis
CLM-CP3-WEIGHT-OUTPUT-COVERT-CHANNEL-3The proposed e-process controls false-alarm probabilities over repeated monitoring only under its stated calibration-transfer assumption; the evaluated attacks are constructed experiments.supported
Basis: reported finding
- supportsLLM output token choices become covert channel for stolen model weightsprimary disclosure
Abstract, calibration-transfer assumption and four-model attacks
Implications
Protect model-serving integrity and monitor output distributions rather than relying on content filters that read only the semantic answer. A statistically calibrated detector can accumulate weak evidence, but it cannot prove that an unmonitored compromised server remained trustworthy.
Controls and mitigations
- Restrict and attest inference-server code paths that influence token selection.
- Compare matched prompts with a trusted serving path and calibrate mismatch baselines for the actual deployment.
- Keep numerical variability and calibration drift explicit when interpreting covert-channel alerts.
Unknowns and contradictions
- No verified in-the-wild model-weight exfiltration through this channel was described.
- Detection depends on representative benign calibration and serving conditions that may change across hardware, seeds or decoding paths.
Sources and citation
Material revision history
- Oct 13, 2026 · Published version · first publication · revision 137