Topic · DiggingBeagle record
Neuron-level probing for malicious code detection
Studying hidden language-model activations to identify internal signals predictive of malicious source-code behavior, with targeted interventions to test whether those signals contribute to a detector's output.
- Topic kind
- pattern
Definition & limits
Neuron-level security probing asks which intermediate activations of a pretrained model carry information useful for detecting unsafe code. A linear readout fitted to activations can predict known labels even when the underlying network is frozen; this demonstrates decodability, not that a unit has a unique or stable semantic label. To test whether it is using programming behavior rather than file format or identifiers, researchers can compare a sample with a variant that removes the unsafe logic and a separately modified variant that preserves behavior, then evaluate selected units against random or activation-matched controls.
The important intervention boundary is the measured readout. Masking selected features may hurt a classifier more than masking controls, supporting their task-specific functional contribution, but the magnitude depends on the chosen decision rule and metric. In ContractLens, accuracy decreases by 8.48 points while AUROC falls by only 0.85 point on one Qwen3-8B masking experiment; it is therefore inappropriate to say that masking eliminated the model's knowledge of malicious contracts. An observed label is also a security taxonomy decision: an owner-only mint or reflection function may be legitimate in some contexts, whereas the same privilege can support abuse in others.
When a study fixes previously selected neurons and then trains a small probe using few labels, this is few-label decoder training, not proof that neuron localization, dataset curation and calibration are all few-shot. ContractLens evaluates code-intent labels on BSC source and the earlier Dao et al. study probes PyPI packages, so neither result automatically transfers to the other domain, model checkpoint or deployment setting. Full deployment requires evaluation of unseen code families, prevalence-dependent false positives, execution reachability and reproducible model artifacts. This technique can support review triage, but does not alone establish exploitation or replace deterministic program checks.
Examples
- The July 2026 Dao et al. PyPI experiment identifies neurons associated with malicious package recognition and intervenes on them across three instruction-tuned models; its domain is packages, not EVM token logic.
- ContractLens selects neuron activations from frozen code models for BSC intent classification; the controlled Qwen3-8B masking test reports a larger decline in accuracy than AUROC.
- A fixed-set ContractLens decoder reports 85.03% AUROC across seven intents with 32 balanced labels, but those 32 examples are not the entire neuron-discovery budget.