A model can reveal information without placing it in its answer

The visible response is only one possible information channel in an AI system. Other parties may see historical model checkpoints, intermediate feature tensors or the pattern of how an agent repairs malformed tool calls. Three studies examine privacy or attribution through these less obvious artifacts, with different attacker prerequisites.

Mengying Zhang and colleagues describe TRACE, a method that compares model weights from before and after approximate unlearning. The attacker is assumed to possess both checkpoints, not just a normal public inference endpoint. The method uses differences in learned representations and output probability behavior to reconstruct selected material the procedure was supposed to remove. The evidence comes from controlled unlearning benchmarks and selected recoveries, not a reported leak of a named person's private records.

The design question is whether releasing a sequence of checkpoints creates a new disclosure surface beyond the final model. A deletion claim should identify which historical artifacts are retained and who can compare them.

Junguang Yao and coauthors study split multimodal models in which edge and cloud components exchange intermediate activations. A server that sees those activations may infer sensitive attributes despite safeguards applied to the final answer. In one setup, with 20 labeled examples per attribute class, the attack recovered roughly 75 to 85 percent of the leakage seen in the unprotected reference. That is a share of baseline leakage, not an absolute 75 to 85 percent success rate for identifying people in production.

The boundary is the transmitted tensor. Filtering the final text cannot revoke information already present in an intermediate representation.

Gejian Zhao and colleagues introduce AgentTrace, which intentionally probes malformed tool schemas and observes the sequence of errors and repairs. Their selected model-identity spoofing tests report 96 percent attribution. Such a behavioral fingerprint is not cryptographic model attestation and may change with wrappers, settings or new versions. It reveals a service characteristic, not necessarily private training content.

These are not interchangeable attacks. An API user does not automatically see internal weights or split tensors, and a tool-call fingerprint does not by itself recover training data. The practical review is to inventory which artifact each system exports, to which principals, and for how long. A privacy boundary cannot be assessed by reading only the model's final answer.

Evidence: TRACE paper, split-representation study and AgentTrace study.