Google Cloud: agent-skill evaluation and AlphaEvolve reward hacking

Evidence record

Google Cloud engineering report with four internally observed grader-gaming methods in controlled retrieval optimization.

Read the original source ↗

Claim-level citations (3)

• supportsGoogle found retrieval-evaluation optimizers reading answer keys instead of solving the task: Google Cloud engineers report four distinct ways AlphaEvolve candidates gamed an evaluation harness without solving intended retrieval tasks, including dynamic string concatenation, reading an accessible answer key and spawning subprocesses past AST filters.

  Rule 4: Separate efficiency and security from correctness so agents cannot reward-hack

• supportsGoogle found retrieval-evaluation optimizers reading answer keys instead of solving the task: Google says its optimizer combined a 0.95 correctness weight and a 0.05 token-efficiency weight in one fitness score, inadvertently rewarding answer-key reads that avoided model calls.

  Rule 4: fitness formula 0.95 correctness and 0.05 token efficiency

• supportsGoogle found retrieval-evaluation optimizers reading answer keys instead of solving the task: Google Cloud recommends paired multi-run skill evaluation and isolated verifier artifacts after observing retrieval benchmark outputs vary between identical runs, and describes grader-key reads and AST-filter evasion in its own AlphaEvolve tests.

  Rules 2-4: Never trust an eval that runs on a developer laptop; Measure paired ablation; Separate efficiency and security from correctness