The Paired Fallback Robustness test asks a compact decision model to choose an alternative when its initially correct class is not available. The study separates two operations that an application might wrongly assume are equivalent: recognizing the input and applying a fallback policy. On the recorded matched TREC50 subset, Intern kept ordinary…
Inspect the ClaimsCase · DiggingBeagle record
Typed decision models misapply fallback rules without malicious input
Scope
Experimental typed-decision classification with three named benchmark families, not a production authorization incident. Incorrect fallback choice can affect downstream routing only where a separate executor trusts the model-selected option.
Assessment profile
Assessment method- Setting
- evaluation
- Exploitation
- not established
- Evidence
- Grade C
- Remediation
- not applicable
- AI role
- AGAINST AI
Basis and provenance
Primary paper metadata and abstract confirm the existence and type of the fallback-policy experiments. Prior pinned Claims give a specific matched subset and exact section locators, but the full paper tables could not be retrieved or independently rerun during this review. Impact remains ungraded because the available evidence does not establish realized harm or a defensible general severity for the assessed deployment.
Assessed Oct 16, 2026 using diggingbeagle.assessment/1.
Evidence basis: CLM-DECISION-FALLBACK-POLICY-01 · CLM-DECISION-FALLBACK-POLICY-02 · CLM-DECISION-FALLBACK-POLICY-03 · Lost in the Fallback: policy selection in typed decision models
At a glance
A typed choice should be interpreted as a recommendation unless an independent component verifies that the chosen branch satisfies the current policy. The experiment does not imply that all typed decision models or production fallbacks fail at the reported TREC50 rate.
Read the implicationsThe exact effect of fallback errors on deployed authorization or model-routing systems has not been measured. Equivalent-policy robustness across changed candidate sets, repeated calls and untested model sizes remains unresolved. A parsed typed response does not establish conformance with an independently defined policy.
Limits and uncertaintyFull account
The Paired Fallback Robustness test asks a compact decision model to choose an alternative when its initially correct class is not available. The study separates two operations that an application might wrongly assume are equivalent: recognizing the input and applying a fallback policy. On the recorded matched TREC50 subset, Intern kept ordinary classification accuracy near 98%, yet its fallback selections were correct in fewer than one quarter of the trials. Supplying the correct original class did not remove the rule-application problem.
The failure requires no attacker-controlled prompt. A typed response can satisfy a parser while choosing the wrong permitted branch, and equivalent descriptions of the same fallback rule can change the model decision. This is a semantic routing error, not evidence that the model bypassed an enforced deny rule in production. Applications that use the selected option as authority need a separate reference implementation for allow, deny, priority and deferral semantics.
Timeline
- Oct 14, 2026disclosure
Public disclosure
Claims & evidence
CLM-DECISION-FALLBACK-POLICY-01On a matched subset of 329 TREC50 inputs that Intern recognized correctly, ordinary assignment accuracy was 97.9% while fallback assignment accuracy was 23.1%.supported
Basis: reported finding
- supportsLost in the Fallback: policy selection in typed decision modelsprimary disclosure
Section 3.2, matched correctly recognized TREC50 subset
CLM-DECISION-FALLBACK-POLICY-02Further experiments supplied the original category explicitly and still found substantial fallback-policy errors.supported
Basis: reported finding
- supportsLost in the Fallback: policy selection in typed decision modelsprimary disclosure
Section 3.3, gold-category and category-only experiments
CLM-DECISION-FALLBACK-POLICY-03The authors report inconsistent decisions under logically equivalent definitions of the same fallback rule, separately from errors in recognizing the original category.supported
Basis: reported finding
- supportsLost in the Fallback: policy selection in typed decision modelsprimary disclosure
Paper abstract, Paired Fallback Robustness test and equivalent fallback rule formulations
Implications
A typed choice should be interpreted as a recommendation unless an independent component verifies that the chosen branch satisfies the current policy. The experiment does not imply that all typed decision models or production fallbacks fail at the reported TREC50 rate.
Controls and mitigations
- For consequential workflows, implement fallback priority, denial and default semantics in a deterministic policy component rather than inferring authorization from valid typed output.
- Test semantically equivalent policy formulations and cases where the recognized class has been removed, while holding the required reference decision fixed.
- Treat deferral to another model or human as an explicit state transition with externally enforced routing, not a model-invented replacement category.
Unknowns and contradictions
- The exact effect of fallback errors on deployed authorization or model-routing systems has not been measured.
- Equivalent-policy robustness across changed candidate sets, repeated calls and untested model sizes remains unresolved.
- A parsed typed response does not establish conformance with an independently defined policy.
Sources and citation
Material revision history
- Oct 16, 2026 · Canonical change recorded · new in release · revision 160