CONSTRAINTDIAL mutates smart contract tasks to detect copied exploit solutions

Smart-contract agents may appear competent because exploit calldata and postmortem solutions are searchable in public on-chain histories. CONSTRAINTDIAL changes constraints and inputs such that past payloads fail while a current exploit path should remain valid, using actual execution to check preservation of the vulnerability. This targets benchmark contamination rather than an active exploit of a particular protocol.

The paper describes reconstruction plus execution-verified protocol mutations producing 3,045 detection, exploitation and patching tasks from 203 public smart-contract incidents, and reports a 10.9-point mean decrease in success using public reference answers. The procedure requires that mutations preserve semantics correctly; the paper evaluation does not guarantee every produced task is uncontaminated or that real auditors will perform similarly.

Retest agents on unseen, execution-verified variations and independently audit the mutation invariants before accepting a new benchmark score as evidence of original security reasoning. A public exploit trace should not be a complete answer key to a supposedly novel security-assessment task. Primary evidence.

Additional evidence and trust boundary

Protocol mutation targets a known benchmark failure: a model can retrieve calldata from the original exploit and pass without understanding the causal vulnerability. A valid mutation must break that old calldata while preserving both normal functionality and a reachable exploit under the revised inputs. This requires executable equivalence checks, not just synthetic paraphrases of exploit descriptions. A result on the constructed corpus cannot rule out answer memorization for every task. Supporting record.