CAGE 4 benchmark scoring can penalize agent actions the simulator never executed

A cyber-defense benchmark can rank agents based on its action accounting rather than on the security effects they actually accomplished. The CAGE 4 audit reproduces reference defender results, then traces a simulator state in which attempted Restore actions are charged even while the defender is busy and no action runs.

The hidden busy-state check and stochastic policy sampling alter score comparisons. The paper attributes 88.9% of KEEP outcome variance under the stock adversary to policy sampling within episodes; removing busy handling produces large score degradation even with identical executed actions. Corrected wrappers and training seeds are necessary to decide whether apparent defender advances are operational or evaluation artifacts. These are simulator findings, not production intrusion-response failures.

Leaderboard scores cannot establish cyber resilience when a simulator conflates attempted and executed actions.

An invalid or never-executed action should not automatically be equated with an executed remediation failure. This audit concerns the measurement apparatus, not a compromised production SOC; reward differences can arise from scheduling and randomness even when the defender's realized effects are unchanged. Reproductions must report exact simulator, wrapper and random seeds before comparing model rankings.