Researchers found an apparent breast-imaging diagnostic AUC near 0.99 on a dataset with strong acquisition-year and label confounding. The test cohort was small and acquisition years correlated with labels; year alone approached the model’s reported discrimination. In a separate benign-versus-malignant evaluation, the tested image representations performed…
Inspect the ClaimsCase · DiggingBeagle record
Breast-imaging AI scores tracked acquisition year as well as disease labels
Researchers found an apparent breast-imaging diagnostic AUC near 0.99 on a dataset with strong acquisition-year and label confounding. In a separate benign-versus-malignant evaluation, the tested image representations performed much closer to chance.
Scope
The test cohort was small and acquisition years correlated with labels; year alone approached the model’s reported discrimination. The result identifies an evaluation shortcut, not proof that all breast-imaging AI models fail clinical diagnosis or that any patient was harmed.
Assessment profile
Assessment method- Setting
- evaluation
- Exploitation
- demonstrated
- Evidence
- Grade B
- Remediation
- not applicable
- AI role
- AGAINST AI
Basis and provenance
The cited research paper supports the stated mechanism within its reported scope. The assessment does not infer a production breach, a victim count or a comparable impact score from a laboratory result. The measured sample and evaluation conditions do not establish field prevalence. Boundary: The model is not being independently tested for medical disease recognition when the image collection itself makes acquisition year a near-label proxy.
Assessed Oct 29, 2026 using diggingbeagle.assessment/1.
Evidence basis: CLM-CP23-BREAST-IMAGING-YEAR-BIAS-1 · CLM-ENRICH-20261029-08 · Federated Breast Imaging Models: Acquisition-Period Confounding
At a glance
In a separate benign-versus-malignant evaluation, the tested image representations performed much closer to chance.
Read the implicationsThe result identifies an evaluation shortcut, not proof that all breast-imaging AI models fail clinical diagnosis or that any patient was harmed.
Limits and uncertaintyFull account
Researchers found an apparent breast-imaging diagnostic AUC near 0.99 on a dataset with strong acquisition-year and label confounding.
The test cohort was small and acquisition years correlated with labels; year alone approached the model’s reported discrimination. In a separate benign-versus-malignant evaluation, the tested image representations performed much closer to chance.
The result identifies an evaluation shortcut, not proof that all breast-imaging AI models fail clinical diagnosis or that any patient was harmed. Evaluate external year- and device-matched cohorts and compare all models with an acquisition-period-only baseline.
Evidence interpretation and control boundary. The model is not being independently tested for medical disease recognition when the image collection itself makes acquisition year a near-label proxy. A year-only baseline reaching a similar AUC is evidence of confounding, not a medical alternative. The benign-versus-malignant results and limited cohort size prevent a safe extrapolation of the high headline AUC to clinical diagnostic benefit.
Mechanism and trust boundary
- 01
Step
- 02
Step
- 03
Step
Timeline
- Oct 27, 2026disclosure
Public disclosure
Claims & evidence
CLM-ENRICH-20261029-08The breast-imaging study reports a year-only baseline that almost matches the high AUC of image models on the confounded BCD evaluation and much weaker benign-versus-malignant discrimination on a separate task.supported
Basis: reported finding
- supportsFederated Breast Imaging Models: Acquisition-Period Confoundingprimary disclosure
Breast imaging paper: Sections 4.2-4.4, Acquisition-Period Confounding, dataset limits
CLM-CP23-BREAST-IMAGING-YEAR-BIAS-1Researchers found an apparent breast-imaging diagnostic AUC near 0.99 on a dataset with strong acquisition-year and label confounding. The test cohort was small and acquisition years correlated with labels; year alone approached the model’s reported discrimination. In a separate benign-versus-malignant evaluation, the tested image representations performed much closer to chance.supported
Basis: reported finding
- supportsFederated Breast Imaging Models: Acquisition-Period Confoundingprimary disclosure
Breast imaging paper: Sections 4.2-4.4, Acquisition-Period Confounding, dataset limits
Implications
In a separate benign-versus-malignant evaluation, the tested image representations performed much closer to chance.
Controls and mitigations
- Evaluate external year- and device-matched cohorts and compare all models with an acquisition-period-only baseline.
Unknowns and contradictions
- The result identifies an evaluation shortcut, not proof that all breast-imaging AI models fail clinical diagnosis or that any patient was harmed.
Sources and citation
Material revision history
- Oct 29, 2026 · Canonical change recorded · new in release · revision 202