MolLean represents molecular reasoning as explicit state transitions checked by a Lean-based kernel. Edits and assertions are accepted only when structural conditions hold; an invalid proposal leaves the previous molecular state unchanged. Its typed witness design keeps calculations that require external measurements separate from facts the kernel can check…
Inspect the ClaimsCase · DiggingBeagle record
Lean-checked molecular actions can remain wrong for the user's objective
Scope
Controlled molecular-graph generation, editing and reaction benchmarks, not a laboratory finding or proof that formally checked operations preserve user intent in arbitrary applications.
Assessment profile
Assessment method- Setting
- evaluation
- Exploitation
- not established
- Evidence
- Grade C
- Remediation
- not applicable
- AI role
- AGAINST AI
Basis and provenance
The original paper abstract supports typed molecular verification, unchanged state on rejected actions and measured evaluation breadth. Retained detailed Claims have section locators, but the proof code, witnesses and full results were not independently audited here. Impact remains ungraded because the available evidence does not establish realized harm or a defensible general severity for the assessed deployment.
Assessed Oct 16, 2026 using diggingbeagle.assessment/1.
Evidence basis: CLM-MOLLEAN-VERIFIED-WRONG-OBJECTIVE-01 · CLM-MOLLEAN-VERIFIED-WRONG-OBJECTIVE-02 · CLM-MOLLEAN-VERIFIED-WRONG-OBJECTIVE-03 · MolLean: executable verification for molecular reasoning
At a glance
Deterministic verification can reject impossible graph transitions, but a separate specification must express which molecular result the user actually requested. Verified steps are evidence of allowed state transformations, not evidence that the final scientific or operational objective was correct.
Read the implicationsVerification of native molecular state does not automatically authenticate external measurements or witnesses. The recorded process-soundness counts cannot be equated with correct task intent or independently validated laboratory outcomes. Performance differences across one-pass, refinement and interactive MCP modes require accounting for their distinct…
Limits and uncertaintyFull account
MolLean represents molecular reasoning as explicit state transitions checked by a Lean-based kernel. Edits and assertions are accepted only when structural conditions hold; an invalid proposal leaves the previous molecular state unchanged. Its typed witness design keeps calculations that require external measurements separate from facts the kernel can check directly, while a stateful MCP interface lets a model receive feedback and revise its next action.
Those guarantees apply to the encoded chemical state and asserted properties, not to the user's complete objective. A trace can be locally valid and still select the wrong target or direction of optimization. The study separately evaluates process soundness, fidelity to the task and final answer correctness; the existing ChEBI20 audit reports many structurally sound but incorrect results. Across 8,800 matched examples and twelve tasks, the paper reports gains from whole-trace refinement over direct prompting, but such gains should not be interpreted as formal proof of successful drug design, chemical safety or externally authorized execution.
Timeline
- Oct 14, 2026disclosure
Public disclosure
Claims & evidence
CLM-MOLLEAN-VERIFIED-WRONG-OBJECTIVE-01MolLean commits accepted molecular state transitions and leaves state unchanged after rejected operations, using Lean-checked representations and assertions.supported
Basis: reported finding
- supportsMolLean: executable verification for molecular reasoningprimary disclosure
Sections 2.1 through 2.5, executable state and verification design
CLM-MOLLEAN-VERIFIED-WRONG-OBJECTIVE-02The authors distinguish process soundness, task fidelity and answer correctness. In their 3,300-example ChEBI20 evaluation, 1,533 outputs were sound but incorrect.supported
Basis: reported finding
- supportsMolLean: executable verification for molecular reasoningprimary disclosure
Section 2.4, verification boundaries; Section 3.3, ChEBI20 analysis
CLM-MOLLEAN-VERIFIED-WRONG-OBJECTIVE-03MolLean evaluates one-pass execution, whole-trace refinement and interactive MolLean-MCP reasoning on 8,800 matched examples spanning twelve tasks, reporting that whole-trace refinement improves on direct chain-of-thought across those tasks for Qwen3.6-27B.supported
Basis: reported finding
- supportsMolLean: executable verification for molecular reasoningprimary disclosure
Paper abstract, benchmark scale and comparison of one-pass, refinement and MolLean-MCP modes
Implications
Deterministic verification can reject impossible graph transitions, but a separate specification must express which molecular result the user actually requested. Verified steps are evidence of allowed state transformations, not evidence that the final scientific or operational objective was correct.
Controls and mitigations
- Specify desired objectives and constraints independently of the model-generated steps, then check both state validity and goal satisfaction.
- Treat externally measured properties as attributed witnesses, not as theorems established by the native Lean kernel.
- Retain rejected transitions and accepted state histories to distinguish syntactic/process faults from intent or final-answer errors.
Unknowns and contradictions
- Verification of native molecular state does not automatically authenticate external measurements or witnesses.
- The recorded process-soundness counts cannot be equated with correct task intent or independently validated laboratory outcomes.
- Performance differences across one-pass, refinement and interactive MCP modes require accounting for their distinct iteration budgets.
Sources and citation
Material revision history
- Oct 16, 2026 · Canonical change recorded · new in release · revision 160