An answer can contain a number that appears verbatim in a real paper yet describe a different dataset, model, experimental arm or metric. The study checks source context around claims produced by research agents rather than treating a citation as proof. It found a substantial wrong-experiment category in its curated sample, and stricter provenance screening…
Inspect the ClaimsCase · DiggingBeagle record
AI research agents cite correct numbers from the wrong experiment
An audit of AI-generated research summaries finds many apparently real numbers attached to the wrong experiment or denominator.
Scope
The research uses a curated set of agent-generated claims and annotated judgments, not all published AI research.
Assessment profile
Assessment method- Setting
- evaluation
- Exploitation
- demonstrated
- Evidence
- Grade C
- Remediation
- not applicable
- AI role
- WITH AI
Basis and provenance
The pinned Claim and its primary disclosure support the described controlled result, but do not establish a population-wide rate or calibrated impact. The source is primary to the described analysis, not independent replication of every outcome. Impact remains unbanded because no calibrated impact rubric for this particular evidence is established.
Assessed Nov 2, 2026 using diggingbeagle.assessment/1.
Evidence basis: CLM-2026-NOV02-RESEARCH-AGENT-MISATTRIBUTED-NUMBERS-A · Beyond fabrication: Misattributed numbers in papers written by AI research agents
At a glance
Claim verification must preserve the experiment, population, numerator, denominator and observation dates, not only a URL or numeric value.
Read the implicationsThe research uses a curated set of agent-generated claims and annotated judgments, not all published AI research. The assessed sample is curated research-agent output, not a random survey of all scientific articles or an estimate of field-wide falsification.
Limits and uncertaintyFull account
An answer can contain a number that appears verbatim in a real paper yet describe a different dataset, model, experimental arm or metric. The study checks source context around claims produced by research agents rather than treating a citation as proof. It found a substantial wrong-experiment category in its curated sample, and stricter provenance screening improved precision while losing recall. This speaks directly to auditability of evidence and metrics rather than generic hallucination alone.
The failure is subtler than inventing a statistic. The agent can reproduce a real number from a paper but attach it to the wrong trial, dataset slice, denominator or model configuration. The study's audit distinguishes supported observations, wrong-source attributions and missing evidence; its comparison with naive number matching shows why numeric coincidence is a poor provenance test. A verifier should require an exact table row, evaluation population, aggregation and source version. Where any of those do not match, the number must be reported as unverified even when the digits appear in a cited publication.
Mechanism and trust boundary
- 01
Agent extracts a plausible metric
A statistic is copied from a cited publication.
Boundary: Primary evidence into synthesized claim
- 02
Attach the wrong denominator or experiment
The response uses real digits but changes the evaluated population or run.
Boundary: Source context into inaccurate assertion
- 03
Naive verifier accepts matching digits
A search for the number finds it without validating the actual experimental referent.
Boundary: Cited numeric string into unearned certainty
Claims & evidence
CLM-2026-NOV02-RESEARCH-AGENT-MISATTRIBUTED-NUMBERS-AThe researchers classify 47 of 256 reviewed numeric claims as sourced to the wrong experiment, while six lack a source; a simple number-presence check often misses this attribution error.supported
Basis: reported finding
- supportsBeyond fabrication: Misattributed numbers in papers written by AI research agentsprimary disclosure
Benchmark and data, attribution error categories and provenance checking results
Implications
Claim verification must preserve the experiment, population, numerator, denominator and observation dates, not only a URL or numeric value.
Controls and mitigations
- Match every metric to its exact table, condition, run and aggregation context before publication; keep uncertain attributions out of summary statistics.
- Record a source table or artifact locator and population/denominator with each numeric Claim; refuse a matched value from a different experimental condition.
Unknowns and contradictions
- The research uses a curated set of agent-generated claims and annotated judgments, not all published AI research.
- The assessed sample is curated research-agent output, not a random survey of all scientific articles or an estimate of field-wide falsification.
Sources and citation
Material revision history
- Nov 2, 2026 · Canonical change recorded · new in release · revision 214