Selective auditing can bound aggregate AI errors escaping review, not a specific harmful approval

The Selective Auditing paper studies when a system may defer individual assessments to a human reviewer rather than assuming every AI decision should receive equal scrutiny. The method randomizes review probabilities and weights the observed mistakes to estimate and control the share of errors not caught. Experiments use specific model safety-evaluation datasets; some finite-sample settings exceed desired targets.

The distinction is operational: a bounded expected or asymptotic error fraction cannot substitute for a mandatory signed approval when a particular action would transfer funds, publish a release or alter production. The paper supports study of attention allocation, not replacing authorization enforcement. Statistical method.

The model's confidence score determines a probability of referral, so the auditor observes a biased sample rather than every mistake. Importance weighting corrects for this sampling when estimating the unattended error rate. The objective therefore has two independent parts: spend fewer human reviews and keep the weighted error fraction under a specified threshold. Neither part proves that an unaudited high-cost decision is safe. An actual payment or production-change system would still need a mandatory effect-specific approval gate; an adaptive review allocator could prioritize lower-consequence work around that gate. The preprint evaluates harmful-request screening, qualitative annotation and labeling, not production financial loss or reviewer fatigue.