The Deception Delta: synthetic fund-drain contracts mislead AI bytecode auditors

A malicious contract does not need to break EVM semantics to mislead an auditor. In a controlled study, Timo Schefold placed owner-controlled fund-transfer paths inside purpose-built vault contracts, then modified the bytecode and surrounding strings to frustrate language models performing first-pass security analysis. The attacker-controlled surface was the contract runtime itself: hex-encoded bytecode, including human-readable strings and function structure, was submitted to models as data for interpretation. A bad outcome for the simulated analyst was a missed extraction path or a reassuring safety assessment despite the ability to describe the path. No public blockchain or actual funds were involved.

The September 12 arXiv submission was corrected on October 6, 2026 to match the accepted ACM CCS 2026 version. The primary matrix contains 22 models, nine deceptive drains, four controls, six prompts, five repetitions per cell and 8,528 analyzable responses. Models received only a single hex-encoded EVM runtime without source, on-chain transactions, storage contents, execution tools or follow-up questions. The researchers built the contracts adaptively with Claude Opus as a generation assistant and attack target, then froze bytecode and ran local checks. This synthetic benchmark measures robustness against designed camouflage, not the base rate of scams in deployed DeFi applications.

The reported detection result is 41.0% on two non-deceptive drain controls versus 21.0% on the nine deceptive vectors, a gap of 20.0 percentage points with a 95% bootstrap interval for the magnitude of 17.2 to 22.8 points. Structural techniques were most disruptive: a five-function settlement chain, selector XOR masking combined with indirect recipient storage, and dynamically loaded transfer parameters all forced the analyst to reconstruct privileged fund flows rather than rely on descriptive names. The two hardest individual vectors were detected in 2.3% and 3.5% of scored outputs, but the strength of this conclusion is bounded by the specific models, prompts, contracts and labeling scheme.

The distinctive result is rationalization. In synthetic vault HV-3b, an embedded fake audit explanation describes a drain as normal settlement accounting. Some models trace the transfer, note the balance-clearing operation, and still judge it benign. Thus the paper codes 49.4% of HV-3b outputs as mechanism detections, but 62.9% of those detections are rationalizations rather than correct alarm verdicts. Counting a rationalized answer as successful detection is defensible for measuring recognition but unsafe for measuring whether an investigator would be reassured. The authors explicitly report that recoding rationalized outputs as missed increases the aggregate gap to 23.9 points. The phenomenon is not simply obeying an injected system command: an artifact can bias interpretation even when the model identifies the technical behavior.

Two contracts also carried instructions embedded as bytecode strings. In the reported data, 292 of 656 false-negative analyses mention those instructions while overlooking the actual drain. The defensive instruction to ignore embedded commands did not systematically help under the six tested prompts. That matters for security-auditor workflows: recognizing hostile text and reconstructing unauthorized token transfers are separate tasks, and a successful first check cannot substitute for the second.

Measurement is not free of caveats. The authors used an LLM judging pipeline with one expert labeling 50 calibration responses; a subset of models served both as analysis subjects and judges, although selector-match and risk-rating checks corroborated the aggregate trend without using judges. The 71 local tests also require exact interpretation: seven involve predecessor variants, the HV-1 token-call selector is nonstandard and cannot drain a normal ERC-20, and HV-PI/HV-PI2 have no standalone EVM tests. The claimed ground truth is therefore documented and qualified, not an independently rerun formal proof of every route. The linked Zenodo archive could not be inspected in this session.

A separate source-code pilot changes the conclusion materially: across 240 drain-bearing runs within a smaller eight-model, eight-contract subset, detection improved from 41.7% with bytecode to 97.9% with source. This is evidence that representation and independently recoverable execution semantics matter, not proof that source-aware tools are universally accurate. The paper reports that ordinary Slither and Mythril runs separately did not identify its composed drains, while also not testing comprehensive trace-based, source-assisted or human-in-the-loop pipelines. The practical boundary is to treat one-shot model explanation as triage, then demand a separately reconstructed privileged fund-flow path before accepting a benign verdict.