The Deception Delta: Adversarial Evaluation of LLM-Based Smart Contract Bytecode Forensics, corrected arXiv v2

Evidence record

Full-text primary technical paper by Timo Schefold, arXiv:2609.14098v2, revised October 6, 2026. It evaluates 22 models on nine synthetic deceptive contracts and four controls, distinguishes mechanism identification from correct risk assessment, discloses calibration and EVM-ground-truth limitations, and limits conclusions to single-shot raw-bytecode-only triage. The results have been read in the published HTML version; no independent rerun was performed.

Read the original source ↗

Claim-level citations (18)

• supportsThe Deception Delta: synthetic fund-drain contracts mislead AI bytecode auditors: The authors used an LLM-as-judge workflow calibrated against 50 cases labeled by one domain expert, and reported two separate judge-independent checks of model responses. Selector mention dropped by 11.4 percentage points and ratings of at least 3/5 dropped by 6.5 points between controls and deceptive vectors, corroborating direction but not equaling the main detection metric.

  Sections 4.3 Human calibration and decision rule, Validation layers and limitations; Section 5.2 Judge-independent checks

• supportsThe Deception Delta: synthetic fund-drain contracts mislead AI bytecode auditors: In the authors' calibrated evaluation, mechanism detection fell from 41.0% on the two drain-bearing controls C1 and C3 to 21.0% across the nine deceptive drain vectors: a 20.0-percentage-point reduction, with a reported 95% contract-stratified run-level bootstrap interval of 17.2 to 22.8 percentage points for the decrease.

  Section 5.1 The Deception Effect, first paragraph and confidence-interval method; Section 4.3 verdict mapping

• supportsThe Deception Delta: synthetic fund-drain contracts mislead AI bytecode auditors: Adding warnings to ignore instructions embedded in the bytecode had no consistent aggregate advantage in the tested prompts: the investigation pair changed by -0.1 percentage point and the neutral pair by +1.2 points on deceptive vectors; prompt changes moved some individual models in opposing directions.

  Section 5.6 Guard instructions offer no aggregate benefit; Framing effects are weak

• supportsThe Deception Delta: synthetic fund-drain contracts mislead AI bytecode auditors: Schefold evaluated 22 language models from seven providers on 13 synthetic contracts, comprising nine adversarial-deception vectors and four controls, across six prompt strategies and five planned repetitions per model-contract-prompt cell. Of 8,580 planned runs, 8,528 were analyzable non-refusals; 36 were pending and 16 were runtime-labeled refusals.

  Sections 3.2-3.3 and 4.1 Experimental Design, paragraphs stating 22 models, 13 contracts, six prompts, 8,580 planned and 8,528 analyzable

• supportsThe Deception Delta: synthetic fund-drain contracts mislead AI bytecode auditors: On synthetic vault HV-3b, models sometimes correctly describe the concealed fund-transfer mechanism and then call it benign after accepting fake audit or accounting explanations. The paper reports 49.4% mechanism-detection under its coding, but 62.9% of those counted as detected are rationalized rather than correct harmfulness warnings.

  Sections 4.3 Inputs and outcome taxonomy and 5.3 Table 3 HV-3b row; Section 5.4 Rationalization paragraphs

• supportsThe Deception Delta: synthetic fund-drain contracts mislead AI bytecode auditors: A separate exploratory matched source-code subset found 100/240 (41.7%) drain detections from raw bytecode versus 235/240 (97.9%) from source across eight models, eight contracts and one prompt. This subset is not part of the principal 13-contract bytecode-only matrix and does not test complete decompiler-assisted or agent toolchains.

  Section 5.2 Input and construction boundary checks, source-code subset; Section 6 Limitations

• supportsThe Deception Delta: synthetic fund-drain contracts mislead AI bytecode auditors: The attacker in the paper controls the content and structure of a synthetic unverified EVM contract that a defender model receives as a hex-encoded runtime in a one-shot risk assessment without source, chain history, storage state, tools or dialogue. The demonstrated failure is misleading triage output, not an observed model-tool privilege escalation or a real financial incident. The paper discusses suppression of escalation or mistaken investigative attribution as possible downstream consequences, not measured outcomes.

  Sections 3.1 Threat Model, paragraphs defining adversary, analyst inputs and possible attribution errors; 4.1 Experimental Design; 6 Implications for LLM-based bytecode forensics

• supportsThe Deception Delta: synthetic fund-drain contracts mislead AI bytecode auditors: The study contracts were purpose-built for controlled research and were never deployed on a public blockchain; the results do not demonstrate a live victim, theft, compromised deployed contract, or prevalence of this mechanism among real-world contracts.

  Introduction final paragraph; Section 6 Limitations; Appendix B Ethical Considerations

• supportsThe Deception Delta: synthetic fund-drain contracts mislead AI bytecode auditors: The authors report 71 passing local Foundry tests and 13 selector checks, but acknowledge coverage limits: seven tests address predecessor contracts outside the 13-contract matrix; HV-1 uses a nonstandard token-call selector that cannot drain a standard ERC-20; and the two prompt-injection vectors have no dedicated EVM tests, although their bytecode selectors were checked.

  Section 3.4 Ground Truth Verification, paragraphs on HV-1, predecessor tests, prompt-injection vectors and selector checks

• supportsThe Deception Delta: synthetic fund-drain contracts mislead AI bytecode auditors: Appendix A of the paper links a Zenodo research artifact said to include synthetic contract code and bytecode, judge and model outputs, labels, and analysis scripts; this research did not independently download or execute that archive.

  Appendix A Open Science, Artifacts and Access subsections, Zenodo DOI

• supportsThe Deception Delta: synthetic fund-drain contracts mislead AI bytecode auditors: The reported detection rate on deceptive contracts varied from 0% to 75.6% across evaluated models. Five model configurations from two providers exceeded 50%; Claude Opus with thinking reached 75.6% and GPT-5.2 improved from 30.7% to 51.5% with thinking, whereas the Anthropic within-family reasoning differences were small or negative. The 0% floor model often failed to engage with bytecode and should not be interpreted as evidence of deception alone.

  Section 5.5 Model Performance, Table 4, paragraphs on floor-tier model and matched reasoning comparisons

• supportsThe Deception Delta: synthetic fund-drain contracts mislead AI bytecode auditors: The paper was first submitted to arXiv on September 12, 2026, revised October 6 as v2 to match the published ACM version, and is listed as accepted for ACM CCS 2026; the conference is scheduled for November 15-19, 2026.

  arXiv v2 HTML header and arXiv abstract page submission history; CCS proceedings banner

• supportsThe Deception Delta: synthetic fund-drain contracts mislead AI bytecode auditors: The paper reports 11.1% aggregate detection for structural deception, compared with 37.7% for string-based deception and 29.3% for embedded-instruction deception. Detection on the hardest structural contracts HV-4b and HV-3 is 2.3% and 3.5%, respectively, under the evaluated raw-bytecode protocol.

  Sections 5.1 and 5.3, Table 3 and paragraphs describing structural, string and instruction-injection vectors

• supportsThe Deception Delta: synthetic fund-drain contracts mislead AI bytecode auditors: The headline detection rate explicitly counts rationalized responses as detected because the mechanism was recognized, even though those responses are unsafe as forensic recommendations. Recoding rationalized outputs as missed increases the reported aggregate deception effect to a 23.9-percentage-point detection gap.

  Section 4.3 binary verdict mapping; Section 5.2 Verdict and judge sensitivity, rationalized-to-fn sensitivity result

• supportsThe Deception Delta: synthetic fund-drain contracts mislead AI bytecode auditors: In the HV-PI2 synthetic test, a conversational prompt-injection string asks the auditor to reconsider the input while benign GreenHope C2 runtime is embedded as inert bytes returned by an unused getReferenceContract() function. That embedded code is not a second deployed contract; it is attacker-controlled decoy data inside the contract under analysis.

  Section 3.3 Deception Vectors, Prompt injection subsection, HV-PI2 (Social Engineering) paragraph

• supportsThe Deception Delta: synthetic fund-drain contracts mislead AI bytecode auditors: Across 656 false-negative responses on the two embedded-instruction vectors, the study finds 292 (44.5%) that nevertheless mention or quote the instruction injection. Recognizing hostile instructions therefore did not imply identifying the independent malicious drain in those model responses.

  Section 5.6 Failure modes on prompt-injection vectors, sentence reporting 292 of 656 false negatives

• supportsThe Deception Delta: synthetic fund-drain contracts mislead AI bytecode auditors: A separate study by Scott Thornton tested eight models on 100 vulnerable source-code samples across Python, JavaScript and Java, obtaining 9,366 primary evaluations. It reported small, statistically non-significant effects from adversarial comments (McNemar exact p > 0.21). This is a contrast, not an independent replication of Deception Delta: Thornton evaluates comments over visible conventional vulnerabilities, whereas Schefold evaluates end-to-end deceptive EVM contract bytecode without source or execution tools.

  Sections 2.3 Adversarial Attacks on LLMs (comparison to Thornton), 3.1 Threat Model, and 5.2 Input and construction boundary checks

• supportsThe Deception Delta: synthetic fund-drain contracts mislead AI bytecode auditors: The structured Prompt F produced 32 primary-judge true positives and 45 hallucinations on HV-3b, whereas the other five prompt variants produced 2 to 9 hallucinations each on that vector. The authors describe a tradeoff: requiring concrete mechanisms can reduce vague rationalization while increasing invented mechanisms, so checklist prompting is not by itself a validated fix.

  Section 5.4 Rationalization, paragraph beginning "Prompt design interacts strongly with rationalization"; Section 5.6 Prompt F tradeoff