The report
A peer-reviewed study published October 9 evaluated blockchain auditing tools on 106 Solidity contracts, half labelled vulnerable. In a GPTScan configuration adapted to a current language-model API, the static prefilter routed just eight contracts to the model. Only two of those eight were among the 53 vulnerable examples. GPTScan consequently found zero benchmark-relevant vulnerabilities.
When the researchers bypassed the preprocessing and queried the same model directly on all contracts, they recorded 24 correct detections and four false alarms. The observed gain is a change in evaluated input coverage, not a demonstration that model reasoning is uniformly superior to static analysis.
The critical qualification is GPTScan's intended coverage. Its rules focus on ten classes of logic flaw and intentionally exclude several categories that dominate this test suite. The result is best described as a mismatch between the evaluated pipeline and the benchmark. See the case dossier for the ablation design and unresolved replication questions.
The missing 98 contracts expose a coverage property of the adapted GPTScan workflow, not an unseen failure of the language model on those contracts. Direct model prompts and specialized static filtering answer different questions; comparing their success percentages without accounting for which weaknesses each was designed to handle would be misleading.
The denominator belongs in the headline
Of the 106 benchmark contracts, 98 never entered the adapted GPTScan model-analysis path. The number zero therefore describes detection by the configured end-to-end pipeline on the 53 vulnerable examples, not a 53-case independent failure by the underlying model. The direct 24/53 experiment changed input admission as well as the analysis method. The four false positives in that second experiment are a reminder that broader coverage and higher precision are separate requirements.
An evaluation intended to generalize would record the intersection between the detector's supported ten logic categories and the benchmark labels, the number of candidates admitted by its static stage, and the disposition of every excluded file. The study's technical dossier preserves this narrower interpretation. No live exploit or contract loss was measured.