Kimi K3.1 Pro: training-data ablation cuts a reported exploit benchmark from 90.5% to 81.0%

Moonshot published a matched training comparison for Kimi K3.1 Pro as part of its October 2026 release decision. Its internal xdev-bench-hard evaluation provides exploit-development tasks with source or binaries, short vulnerability hints and a controlled secret-retrieval target. The authors report pass@3 results of 90.5% for a checkpoint trained without the targeted cyber reduction, 81.0% after excluding certain cyber material and reinforcement-learning environments, and 33.3% for the earlier K3 model. The reported 9.5 percentage-point reduction compares checkpoints within Moonshot's own pipeline, not two independent vendors or public incident rates. Original experiment.

The procedure materially limits its interpretation. The benchmark offers an exploit starting point and does not apply rigid timeouts or output-token caps, while pass@3 allows up to three trials per task. These conditions differ from an unknown production target without hints, but they also deliberately test whether the model can complete technically involved exploit chains. An 81.0% score after filtering does not demonstrate harmlessness; the risk-reduced model still substantially exceeds the earlier K3 on this vendor-defined workload.

This intervention acts during model training, before a downstream agent receives a tool capability, credential or operating-system permission. A customer still needs to reason about what an attached harness may execute, and a provider cannot infer from the score that a particular transaction or exploit request will always be refused. The model's hosted interfaces were introduced in October, whereas Moonshot stated that full weights were planned for November 4, 2026. Planned open-weight availability must not be backdated as a completed October release.

The unresolved question is how the two matched checkpoints compare on independently run offensive tasks, benign code security work and equivalent tool budgets. Until then, the disclosed result supports a benchmark-specific reduction in measured capability, with no verified denominator for effects outside the study.