The smart-contract auditor found the drain, then explained it away

In a controlled smart-contract experiment, an AI security reviewer described a function that cleared a vault's positions and transferred its balance, then treated the same operation as ordinary accounting. The model had found the mechanism, but its judgment favored an audit story planted inside the contract. That distinction is the central result of Timo Schefold's corrected October 6, 2026 paper, The Deception Delta. The contracts were synthetic and never deployed to steal funds.

The paper contrasts two answers about a test vault called HV-3b. One answer identifies an owner-triggered settlement path that clears accounting positions and sends the vault balance to a designated distributor, then flags the resulting extraction risk. Another describes the settlement and the balance transfer, but accepts the embedded explanation that the operation is a normal conversion and refill cycle. It gives a low risk rating and treats a purported audit reference as evidence of legitimacy. That reference was part of the adversarial material, not an independently checked approval.

These outputs are useful because the defect is not simply that a model failed to decode opaque EVM bytecode. The second model reported meaningful execution behavior and still reached an unsafe conclusion. The paper calls this forensic rationalization, a post-recognition error in which attacker-controlled context changes the interpretation of facts already recovered.

In the author's scoring, 49.4% of HV-3b responses recognized the drain mechanism, yet 62.9% of those counted as detections were rationalized rather than correct alarms. The distinction is crucial: a benchmark can credit recognition even when the resulting report would reassure an investigator about a suspicious contract. These are proportions from one deliberately constructed vector, not measured rates of failure in deployed audits. See the underlying Claim and coding distinction.

Schefold evaluated 22 models from seven providers on nine deceptive contracts and four controls, under six prompt variants. The main dataset contains 8,528 analyzable non-refusal runs. In the author's calibrated scoring, mechanism detection on two drain-bearing controls was 41.0%, compared with 21.0% across deceptive drain vectors, a decline of 20.0 percentage points with a reported 95% interval of 17.2 to 22.8 points for the magnitude. The estimate is tied to a single-shot protocol: the model received raw hex-encoded runtime bytecode, without source code, on-chain transactions, storage state, execution tools or a follow-up conversation.

There is an easily missed accounting choice. The primary statistic treats a rationalized response as detected because the mechanism was recognized. When the authors instead code rationalization as a failure of the usable verdict, the reported aggregate gap becomes 23.9 percentage points. A reader therefore needs both a mechanism-recognition measure and a decision-quality measure. Neither can be substituted for the other, and neither number is an observed rate of financial loss. The Case's measurement Claims retain the denominators and scoring caveats.

The contracts use more than one way to manipulate the analyst. Structural vectors hide extraction across helper calls, selectors assembled through XOR operations, or dynamically loaded recipients and amounts. Their aggregate detection in the paper was 11.1%, compared with 37.7% for string-based deception and 29.3% for embedded-instruction vectors. These are category-specific results from the fixed benchmark, not population estimates for real contracts. The structural examples show how an analyst can fail before it has enough facts to judge. HV-3b shows a different failure: helpful-looking documentation can make a correctly noticed mechanism sound acceptable.

Two further vectors embedded prompt-like instructions directly in the bytes under review. Among 656 responses coded as false negatives for those vectors, 292 still mentioned the injection text or recognized it as suspicious. The model had noticed the attempt to influence its instructions but missed the separate fund-transfer issue. Detecting indirect instruction injection and auditing contract semantics are two independent tasks. An instruction to ignore hostile text does not reconstruct a missing execution path.

A smaller matched pilot in the same paper supplied source code rather than raw bytecode to eight models across eight contracts and one prompt. On its 240 drain-bearing runs, recognition rose from 100/240 (41.7%) with bytecode to 235/240 (97.9%) with source. The pilot is not the primary thirteen-contract benchmark and does not prove that all source-aware audits work, but it makes a broad claim that AI auditors are equally vulnerable on every representation indefensible.

A separate study by Scott Thornton tested adversarial comments in source-code reviews of Python, JavaScript and Java. Across 9,366 primary evaluations, it found small, statistically non-significant detection changes (McNemar exact p greater than 0.21). The experiments are not replications: Thornton overlays manipulative comments on visible vulnerable code, while Schefold designs the contract and its compiled representation to mislead the reviewer as a whole. Both are relevant to deciding what data an auditor needs, but their percentages should not be placed on one common effectiveness scale.

That contrast also cautions against a simplistic defense based on selecting a longer prompt or asking the model to reason harder. In Deception Delta, adding a warning to ignore embedded instructions had no reliable aggregate benefit under the tested prompts. A structured checklist improved some detections, but on HV-3b it produced 32 true positives and 45 hallucinated mechanisms according to the primary judge. More specific answers can be more useful to verify while also being confidently invented. Neither additional words nor an unverified explanation constitutes execution evidence.

The Deception Delta dossier and the contrasting source-review result suggest an audit record should preserve four distinct things: the privileged fund-flow path the analyst could actually reconstruct; the identities and conditions controlling that path; the provenance of any safety, audit or governance assertion; and the ultimate risk decision, including unresolved premises. This is an editorial control proposal derived from the observed failure, not a mitigation proven by either experiment.

For example, an LLM report that names a recipient but has not established who can change that recipient should say the control condition is unresolved. A report that finds an owner-controlled transfer should distinguish that capability from an independently demonstrated unauthorized extraction; ordinary administration alone does not establish malicious intent. And a claimed third-party audit, when discovered inside attacker-controlled bytecode, should be treated as a claim requiring a separate source, not as corroboration of itself. A trace, source inspection or purpose-built fund-flow check could help evaluate those questions, although the study did not validate any complete combined review pipeline.

The limits of the evidence are substantial. The main detection labels came from an LLM judging system calibrated with 50 examples labeled by one expert. Independent selector and risk-rating signals supported the direction of the aggregate result but are not full reproductions of the verdict labels. The reported local EVM checks also have per-vector qualifications, including a nonstandard selector in HV-1 and missing dedicated EVM tests for two injection vectors. The linked research artifact archive was not independently rerun here. The ACM CCS 2026 accepted-paper listing confirms venue acceptance, not validation of each experimental outcome.

This was a study of deliberately constructed inputs and model responses, not a case of stolen funds or a deployed deceptive contract. Its practical finding is narrower and more consequential than a general warning about bad AI reasoning: when a reviewer identifies a privileged transfer and then accepts the artifact's own justification, the evidence supports an unresolved security question, not a safe verdict.