Enterprise deployment agents rarely achieved full stakeholder and outcome closure

FDE-Gym evaluates agents that must deliver useful changes for a simulated enterprise customer, not just pass a code test. Each task places an agent among stakeholders, editable systems, partial or hidden requirements and costs for unnecessary messages, delays or incidents. The decisive problem is often finding the right requirement before modifying anything: success depends on the customer's actual business outcome rather than the visible task description alone.

The benchmark contains 95 tasks inspired by 20 projects across 18 sectors. Its highest reported mean score was approximately 0.56, yet that configuration satisfied the study's stricter resolved condition on only 14 of 95 tasks. The threshold combines a reference-relative uplift of at least 0.8 with no incident. Even where the leading agent discovered all essential facts on 23 tasks, only 10 reached the uplift threshold. A different tested model performed worse than leaving the customer's system unchanged on 79 of 95 tasks. These are separate measures and should not be presented as one rate.

The cases are designed and graded simulations, not reports that named corporate customers were damaged by agents. The study provides an especially useful distinction between question-answer accuracy, stakeholder discovery, an implemented technical artifact and measurable end-state improvement. A production replication would require consented real customer work, independent outcome measurements and an accounting of review time and mistakes, not simply another benchmark score.