Evidence · DiggingBeagle record
RoboDojo: Grader mismatches and apparent self-improvement
A self-improving robot agent trained against grader feedback that contained hidden conditions absent from the task text. Exposure to those conditions improved scores against the flawed grader without corresponding user-specification performance. On 16 mismatched simulated tasks, the reported 51.6% versus 27.8% apparent success narrowed to 45.0% versus 47.2% when the same trajectories were judged by a specification-faithful grader. Research/disclosure claims are specific to the documented settings; do not infer unreported victims or additional compromise.
- Source role
- primary disclosure
Evidence record
A self-improving robot agent trained against grader feedback that contained hidden conditions absent from the task text. Exposure to those conditions improved scores against the flawed grader without corresponding user-specification performance. On 16 mismatched simulated tasks, the reported 51.6% versus 27.8% apparent success narrowed to 45.0% versus 47.2% when the same trajectories were judged by a specification-faithful grader. Research/disclosure claims are specific to the documented settings; do not infer unreported victims or additional compromise.
Claim-level citations (2)
- supportsRoboDojo grader exposure inflated robot-agent scores without faithful task gains: On 16 mismatched simulated tasks, the reported 51.6% versus 27.8% apparent success narrowed to 45.0% versus 47.2% when the same trajectories were judged by a specification-faithful grader.
RoboDojo: 16 rubric-mismatched tasks; regrading same episodes
- supportsRoboDojo grader exposure inflated robot-agent scores without faithful task gains: A self-improving robot agent trained against grader feedback that contained hidden conditions absent from the task text. Exposure to those conditions improved scores against the flawed grader without corresponding user-specification performance.
RoboDojo: 16 rubric-mismatched tasks; regrading same episodes