The paper compares static, symbolic, formal and AI-assisted smart-contract analysis. GPTScan was not designed as a general detector for reentrancy, arithmetic, authorization or input-validation errors, which dominate this benchmark. In the adapted pipeline, a Falcon static filter sent only eight contracts to the LLM; two carried annotated benchmark…
Inspect the ClaimsCase · DiggingBeagle record
GPTScan preprocessing filtered most benchmark contracts before model review
On 106 Solidity contracts, the adapted GPTScan pipeline reported no benchmark-relevant findings; its static filter sent only eight contracts to the language model. Direct use of the same model found 24 of 53 vulnerabilities.
Scope
106-contract curated benchmark; adapted GPTScan and gpt-5.6-terra, not field exploit discovery or production detection rate.
Assessment profile
Assessment method- Setting
- evaluation
- Exploitation
- not established
- Evidence
- Grade B
- Remediation
- not applicable
- AI role
- AGAINST AI
Basis and provenance
This is a pipeline-coverage benchmark, not an intrusion. The study supplies contract denominators and compares differently scoped analysis workflows, so no exploit impact score is defensible. Assessment uses the pinned Claims and source identities; public primary pages were not independently retrievable during this run. Impact remains unrated because field consequence or a comparable scoring rubric is not established.
Assessed Oct 15, 2026 using diggingbeagle.assessment/1.
Evidence basis: CLM-CP7-GPTSCAN-ZERO · CLM-CP7-GPTSCAN-DIRECT · CLM-CP7-GPTSCAN-FILTER · Raza et al., Blockchain Security: An Empirical Evaluation of High-Impact Exploits and Detection Tools
At a glance
Original GPTScan release and original benchmark were not reproduced unchanged. The study’s direct prompt and static pipeline target different weakness taxonomies. Independent reproduction of the 8/106 routing and 24/53 outcomes remains open.
Limits and uncertaintyFull account
The paper compares static, symbolic, formal and AI-assisted smart-contract analysis. GPTScan was not designed as a general detector for reentrancy, arithmetic, authorization or input-validation errors, which dominate this benchmark. In the adapted pipeline, a Falcon static filter sent only eight contracts to the LLM; two carried annotated benchmark vulnerabilities. Direct LLM prompting changed the coverage and revealed 24 of the 53 positive cases. The comparison exposes a pipeline-coverage blind spot, not a universal ranking of products or proof that skipping static analysis is always better. The authors explicitly keep different tool paradigms separate.
The 0/53 outcome is conditional on the adapted prefilter and the selected weakness labels. It should not be used to rate the model alone, because 98 contracts never entered its GPTScan model path.
Claims & evidence
CLM-CP7-GPTSCAN-ZEROAdapted GPTScan yielded zero benchmark-relevant detections across 53 vulnerable contracts, and the authors attribute much of the mismatch to GPTScan’s specialized logic-vulnerability scope.supported
Basis: reported finding
- supportsRaza et al., Blockchain Security: An Empirical Evaluation of High-Impact Exploits and Detection Toolsprimary disclosure
Table 7 and Section 8.6.2
CLM-CP7-GPTSCAN-DIRECTWith the same model directly analyzing all 106 contracts in three medium-effort passes, the study recorded 24 true positives, four false positives and 29 false negatives.supported
Basis: reported finding
- supportsRaza et al., Blockchain Security: An Empirical Evaluation of High-Impact Exploits and Detection Toolsprimary disclosure
Table 7 and Section 8.6.3
CLM-CP7-GPTSCAN-FILTERThe Falcon prefilter forwarded eight of 106 contracts to GPT-5.6-terra, of which only two were vulnerable under the benchmark labels.supported
Basis: reported finding
- supportsRaza et al., Blockchain Security: An Empirical Evaluation of High-Impact Exploits and Detection Toolsprimary disclosure
Section 8.6.2, GPTScan pipeline and Falcon prefilter
Unknowns and contradictions
- Original GPTScan release and original benchmark were not reproduced unchanged.
- The study’s direct prompt and static pipeline target different weakness taxonomies.
- Independent reproduction of the 8/106 routing and 24/53 outcomes remains open.
Sources and citation
Material revision history
- Oct 15, 2026 · Canonical change recorded · new in release · revision 150