Generated code can pass compilation while failing its actual contract

A smart contract can compile successfully and still reject the wrong withdrawal, change a balance incorrectly or allow an operation in the wrong state. The compiler checks whether code is valid Solidity. It does not compare the economic behavior of that code with the implementation a developer intended to reproduce.

October research on Solidity generation, secure code repair and safety-monitor integration points to one practical rule: a security verdict must identify the actual behavior it tested. A clean compile, a scanner warning and an observed denial are different types of evidence.

SolBehave regenerated function bodies from 250 source-verifiable Ethereum contracts using seven open-weight models. Compilation success ranged from 26.4% to 92.7%, but only 31.4% of pooled compiling function bodies showed no divergence against originals under the researchers' differential-fuzzing tests. This is bounded empirical agreement, not a formal proof of semantic equivalence.

The experiment combined three oracles. Each exposes a different blind spot:

The researchers report that synthetic sequences exercised 246 to 311 state-changing functions not reached by historical transactions, depending on the graph extractor. For six of seven models, 13% to 31% of compiling contracts diverged under historical replay, while 13% to 44% diverged under synthetic replay. The test cohort was filtered to relatively small contracts with 1 to 20 nontrivial state-changing functions, so those rates should not be projected onto complex DeFi protocols and bridges.

FALCON studies a complementary problem: how to give a code-repairing model useful feedback without trusting an unverified model-written security test. Its workflow attempts an executable counterexample, checks the actual trace, and uses that observed result as feedback. In a selected five-round CWEval comparison, the authors report improving held-out functional-security success from 61.34% to 86.55%. A direct LLM judge improved its own proxy metric without comparable improvement on held-out tests. The measured benefit is specific to the study's tasks, repair budget and grading contract.

This makes two independent verifications necessary. The counterexample must genuinely exercise the claimed flaw, and the repair must preserve functionality while closing it. A confident explanation from either the attacker model or the repair model is not a substitute for the trace.

Monitor-Validity documents 18 selected parent-and-fixed regression cases across 11 framework families where detection evidence and enforcement could become separated. A safety callback can identify a hazard while an adapter drops the relevant signal before the final tool decision. These cases were selected for study and are not a measured industry-wide exploit rate.

For an agent that changes deployed code, the verification chain should be explicit: pin the intended version and initial state; record the exact executable oracle; reproduce the offending input or sequence; check the observed output and side effects; then prove that the final deployment gate acts on the verdict. An absent fuzzer finding is not proof of safety, and a monitor log is not proof that a dangerous write was denied.

That distinction also defines the limit of AI-assisted testing. The model can propose candidate inputs and repairs, but the accepted evidence remains the behavior of a specific artifact under a specified, reproducible test.