The paper compares the same execution-grounded questions under several reading views. A supervisor using only routine agent reports can miss failed tests, workarounds and constraints that are recoverable from full traces. The agent may answer a question about its work if prompted, but a missing incident in the report can prevent a supervisor from knowing…
Inspect the ClaimsCase · DiggingBeagle record
Coding-agent progress reports omit consequential execution state
CTXGAP measures a 40.7 percentage-point gap between report-only and full-context task-state recoverability on its selected sample.
Scope
Controlled report-recoverability measurement over 37 nonrandomly selected coding tasks and 4,799 valid probes. Not a deployed breach, deception rate or claim that a model intentionally conceals information.
Assessment profile
Assessment method- Setting
- evaluation
- Exploitation
- not established
- Evidence
- Grade B
- Remediation
- not applicable
- AI role
- AGAINST AI
Basis and provenance
The directly inspected original study provides recorded coding-agent execution and report comparisons with quantified uncertainty and auditing. This is an observability measurement rather than exploitation, and no production damages or deployed remediation are established.
Assessed Oct 16, 2026 using diggingbeagle.assessment/1.
Evidence basis: CLM-CTXGAP-407 · CLM-CTXGAP-FIXED · CLM-CTXGAP-SCROLLBACK · CLM-CTXGAP-ANNOTATION · CLM-CTXGAP-REWRITES · CTXGAP: What Coding Agents Leave Unsaid
At a glance
Successful test outcomes and confident progress reports do not establish that a supervisor received the information necessary to assess changes. Immutable trace receipts and targeted evidence questions would need independent usability evaluation.
Read the implicationsThe main 37-task cohort was selected nonrandomly and the probes were partly constructed or judged by models. Probe recoverability is not equivalent to actual human supervision outcomes or intentional deception. The study measures whether test questions can be answered, not whether a human supervisor would have noticed and correctly acted on each omission.
Limits and uncertaintyFull account
The paper compares the same execution-grounded questions under several reading views. A supervisor using only routine agent reports can miss failed tests, workarounds and constraints that are recoverable from full traces. The agent may answer a question about its work if prompted, but a missing incident in the report can prevent a supervisor from knowing what to ask. CTXGAP tests this difference separately from benchmark task completion. The study compares identical questions against routine progress reports, terminal scrollback and full execution traces. Its headline is a recoverability difference, not a frequency of deliberate hiding. In a human-annotated sample, many missed details were mentioned but lacked enough supporting context to answer a concrete question. Matched-length rewrites based on execution evidence improved recovery by roughly eight percentage points, indicating that adding relevant evidence can help without simply making reports longer. The nonrandom 37-task sample and model-assisted probe construction limit extrapolation to ordinary teams or all coding agents.
Mechanism and trust boundary
- 01
Agent executes a coding task
The agent performs tests, revisions and decisions inside its execution trace.
Boundary: agent execution state
- 02
Agent summarizes progress
Routine report messages omit or under-specify some evidence, including failures and risks.
Boundary: full execution trace to report channel
- 03
Supervisor assesses the result
A reader relying on reports has less recoverable task state than a reader with the full context.
Boundary: report channel to oversight decisions
Timeline
- Oct 14, 2026disclosure
Public disclosure
Claims & evidence
CLM-CTXGAP-407Across 37 selected tasks and 4,799 valid probes, report-only readers recovered 52.0% of task state compared with 92.7% using full recorded execution context; the authors report a 40.7-point gap (95% CI 37.3 to 43.8 percentage points).supported
Basis: reported finding
- supportsCTXGAP: What Coding Agents Leave Unsaidprimary disclosure
Abstract and Table 1, main measurement (37 tasks; 4,799 probes)
CLM-CTXGAP-FIXEDAmong final checkpoints for 30 hidden-test-verified fixes, the reported context versus report gap remained 34.8 percentage points.supported
Basis: reported finding
- supportsCTXGAP: What Coding Agents Leave Unsaidprimary disclosure
Table 1, Fixed at k=4 and Finding 1
CLM-CTXGAP-REWRITESIn matched-total-length reporting experiments, context-informed natural and targeted rewrites improved recoverability by 8.3 and 8.5 percentage points over original reports, without a detectable advantage for the targeted rewrite.supported
Basis: reported finding
- supportsCTXGAP: What Coding Agents Leave Unsaidprimary disclosure
Introduction, Reporting and robustness checks; Section 5.1, comparing reporting designs
CLM-CTXGAP-ANNOTATIONTwo independent annotators estimated that details mentioned without sufficient supporting evidence accounted for 55.4% and 59.2% of the weighted unrecovered-state gap mass in a 195-atom sample.supported
Basis: reported finding
- supportsCTXGAP: What Coding Agents Leave Unsaidprimary disclosure
Section 4.3, mention, support and recovery; Table 2, 195-atom independent human annotation
CLM-CTXGAP-SCROLLBACKFull terminal scrollback improved recoverability to 73.6% but still trailed full context at 92.7% on the same probes.supported
Basis: reported finding
- supportsCTXGAP: What Coding Agents Leave Unsaidprimary disclosure
Table 1, Scrollback V; Finding 3
Implications
Successful test outcomes and confident progress reports do not establish that a supervisor received the information necessary to assess changes. Immutable trace receipts and targeted evidence questions would need independent usability evaluation.
Controls and mitigations
- Preserve an immutable, accessible execution trace and attach concise evidence receipts for tests, failures, deviations and unresolved risks to agent status reports. The study does not validate any particular logging product or guarantee that humans will inspect these receipts.
- Evaluate reports with task-state questions grounded in the underlying trace. Check whether a reviewer can recover why a change was made and which tests failed, rather than grading tone, report length or stated completion alone.
Unknowns and contradictions
- The main 37-task cohort was selected nonrandomly and the probes were partly constructed or judged by models.
- Probe recoverability is not equivalent to actual human supervision outcomes or intentional deception.
- The study measures whether test questions can be answered, not whether a human supervisor would have noticed and correctly acted on each omission.
Sources and citation
Material revision history
- Oct 16, 2026 · Canonical change recorded · new in release · revision 157