News · DiggingBeagle record

Cammarata challenges the superposition explanation for hard-to-read neurons

A Goodfire essay argues that many network neurons have coherent roles that prior tools missed. Its language-model evidence remains preliminary, and readable circuits alone are not a guarantee of safe behavior.

Published
Oct 17, 2026

The report

A neuron that responds to both a cat's face and a car hood may not be mixing unrelated concepts. Nick Cammarata's October 15 Goodfire essay argues that both responses can arise from a more general V-shaped visual feature. The example is part of a broader hypothesis: apparently tangled neural representations may sometimes be coherent at a level humans did not initially recognize.

A challenge to the standard explanation

In Neural Networks Are Already Clean, Cammarata revisits influential evidence for polysemanticity in vision models and questions whether superposition explains every hard-to-read neuron. The essay also explores native neurons and attribution circuits in language models, including Qwen3-32B. The author proposes that understanding features directly, rather than always first extracting them through a sparse autoencoder, could be productive at scale. Goodfire explicitly describes the work as an informal working hypothesis, not a complete proof that networks contain no superposition.

The language-model material includes selected circuit tracing and ablations. Changing an activation and observing a shift in an answer's probability can support a causal explanation for that chosen task. It does not establish that a neuron retains the same meaning across contexts, model revisions or different operational tasks. It also does not prove that all model behavior can be read reliably from a list of labeled neurons.

What this would mean for security analysis

Digging Beagle's earlier ContractLens research probes internal activation patterns for detecting potentially risky smart-contract capabilities. That is a different claim from Cammarata's: a selected feature may improve a trained classifier even when no stable, human-readable semantic account of its role has been established. Conversely, an attractive natural-language description of a neuron does not demonstrate that disabling it prevents malicious code execution or unauthorized tool use.

A useful follow-up would therefore separate three tests: whether independent analysts can describe the same activation consistently, whether interventions cause predicted effects on previously unseen prompts, and whether any operational safeguard prevents a separately verified harmful action. Goodfire's essay motivates these experiments. It does not substitute for an external permission gate, a deployment audit or an independent safety evaluation.

Research behind this