Analysis · DiggingBeagle record
Three strong AI results still leave the action permission unanswered
Kimi exploit pass rates, PointerBench target accuracy and selective-auditing error guarantees measure different objects. None is an execution authorization test or an observed incident frequency.
- Published
- Oct 25, 2026
Overview
A model can solve 81.0% of selected exploitation tasks, land on 83.2% of benchmark interface targets, or be monitored by a statistical auditing scheme with an aggregate missed-error guarantee. Those figures cannot be added, averaged or used as interchangeable estimates of agent risk. Each has a different unit of observation, and none directly asks whether an irreversible action was authorized for the correct user, resource and time.
Compare the measured event, not just the percentage
| Study | Reported quantity | Unit or test contract | Boundary it does not measure |
|---|---|---|---|
| Kimi K3.1 Pro cyber ablation | 90.5% unrestricted, 81.0% after risk reduction, 33.3% earlier K3 | Internal xdev-bench-hard pass@3 with exploit-development inputs and controlled proof targets |
Unauthorized real-world compromise or success against unprovided targets |
| PointerBench | 83.2% overall and 69.6% on text targets for the leading reported model | 1,500 static screenshot tasks with a 1024-by-768 coordinate protocol | Whether the clicked control was permitted or caused the intended side effect |
| Selective Auditing | A tunable bound on the fraction of model errors left unaudited under the paper's design and guarantees | Confidence-scored decision streams sampled for expert review | A deterministic veto of one specifically dangerous payment, publication or deletion |
These quantities originate in the Kimi technical report, PointerBench paper and selective-auditing paper. The Kimi results are vendor-controlled comparisons, the GUI scores come from annotated visual grounding rather than live account operations, and the auditing claims concern aggregate control, not every individual outcome.
Where the benchmark stops
Moonshot removed selected cybersecurity training material and filtered certain reinforcement-learning tasks. The vendor-reported nine-and-a-half-percentage-point fall from 90.5% to 81.0% is meaningful within the matched experiment, but the result is still much higher than the prior K3 score. It is an intervention on capability development, not a policy enforcement step at an endpoint. The announced open-weight release was planned for November 4, 2026; as of the October publication, that was a plan rather than an observed distribution event.
PointerBench isolates a different failure: a graphical agent may know broadly what the user wants yet choose the wrong character, icon or control. Some tested models' very low raw scores also depended on output-coordinate conventions, and normalization changed their reported accuracy substantially. A fair benchmark must preserve the same frame and geometry contract. Even a perfect hit, however, cannot answer whether the agent had the account authority to click that target.
Selective Auditing reduces expert-review demand by sampling confidence-scored model decisions and statistically accounting for unreviewed mistakes. That provides a useful control objective over sequences, including adversarial ones under the paper's framework, but a reviewer deciding whether a particular transfer should execute still needs a hard authorization policy. Aggregate error regulation and mandatory approval at an irreversible boundary solve different questions.
The practical comparison is a four-part record
For a deployed agent, record the proposed decision, the exact target, the authority granted for that target, and the resulting external effect. Capability benchmarks can inform the first two; selective auditing can allocate review attention; neither substitutes for the third. The fourth requires an observed transaction or verifiable receipt. A deployer can use these studies together without turning their percentages into a synthetic safety score that none of the original researchers claimed.