CTXGAP: What Coding Agents Leave Unsaid

Evidence record

Original research paper comparing coding-agent reports, full execution context and terminal scrollback across selected tasks and task-state probes.

Read the original source ↗

Claim-level citations (5)

• supportsCoding-agent progress reports omit consequential execution state: Across 37 selected tasks and 4,799 valid probes, report-only readers recovered 52.0% of task state compared with 92.7% using full recorded execution context; the authors report a 40.7-point gap (95% CI 37.3 to 43.8 percentage points).

  Abstract and Table 1, main measurement (37 tasks; 4,799 probes)

• supportsCoding-agent progress reports omit consequential execution state: Among final checkpoints for 30 hidden-test-verified fixes, the reported context versus report gap remained 34.8 percentage points.

  Table 1, Fixed at k=4 and Finding 1

• supportsCoding-agent progress reports omit consequential execution state: In matched-total-length reporting experiments, context-informed natural and targeted rewrites improved recoverability by 8.3 and 8.5 percentage points over original reports, without a detectable advantage for the targeted rewrite.

  Introduction, Reporting and robustness checks; Section 5.1, comparing reporting designs

• supportsCoding-agent progress reports omit consequential execution state: Two independent annotators estimated that details mentioned without sufficient supporting evidence accounted for 55.4% and 59.2% of the weighted unrecovered-state gap mass in a 195-atom sample.

  Section 4.3, mention, support and recovery; Table 2, 195-atom independent human annotation

• supportsCoding-agent progress reports omit consequential execution state: Full terminal scrollback improved recoverability to 73.6% but still trailed full context at 92.7% on the same probes.

  Table 1, Scrollback V; Finding 3