A downstream service relies on a model-generated predicate, choice or score. An attacker alters the input evidence or argument order to move the model toward an unauthorized choice while preserving the expected output type. JevAdvBench evaluates black-box perturbations against typed models that return probabilities, choices or ordered levels, with…
Inspect the ClaimsCase · DiggingBeagle record
Typed AI decision outputs remained vulnerable to persuasive adversarial input
A structured answer format reduces free-text prompt injection but does not ensure that an attacker-controlled input leaves an automated risk decision intact.
Scope
The benchmark tests model decisions and one service-style case study; valid output formatting does not show an actual unauthorized production payment or access grant.
Assessment profile
Assessment method- Setting
- evaluation
- Exploitation
- demonstrated
- Evidence
- Grade C
- Remediation
- not applicable
- AI role
- AGAINST AI
Basis and provenance
The benchmark controls typed decision labels and adversarial prompt inputs and records decision deviations in a model/routing setup. Well-formed typed outputs do not guarantee faithful decisions, but no production payment or authorization bypass is independently shown. Impact is left ungraded because a calibrated 0-to-4 impact basis for actual affected targets is unavailable from these Claims.
Assessed Oct 28, 2026 using diggingbeagle.assessment/1.
Evidence basis: CLM-CP22-031-A · CLM-CP22-031-B · JevAdvBench: Black-Box Attacks on Typed Decision Models
At a glance
Threshold calibration, source provenance and exact operational effect under deployment-specific human review remain uncertain.
Limits and uncertaintyFull account
A downstream service relies on a model-generated predicate, choice or score. An attacker alters the input evidence or argument order to move the model toward an unauthorized choice while preserving the expected output type.
JevAdvBench evaluates black-box perturbations against typed models that return probabilities, choices or ordered levels, with downstream services routing automatically above decision thresholds. The paper’s black-box suite tests argument-driven and evidence-order inputs, not only imperative instructions, and tracks whether answers remain valid in the required typed format. Primary evidence.
The benchmark tests model decisions and one service-style case study; valid output formatting does not show an actual unauthorized production payment or access grant.
Treat typed model output as evidence for a decision, and separately enforce application-owned access, payment and consequence policy.
Claims & evidence
CLM-CP22-031-AJevAdvBench evaluates black-box perturbations against typed models that return probabilities, choices or ordered levels, with downstream services routing automatically above decision thresholds.supported
Basis: reported finding
- supportsJevAdvBench: Black-Box Attacks on Typed Decision Modelsprimary disclosure
Abstract; threat model and downstream service
CLM-CP22-031-BThe paper’s black-box suite tests argument-driven and evidence-order inputs, not only imperative instructions, and tracks whether answers remain valid in the required typed format.supported
Basis: reported finding
- supportsJevAdvBench: Black-Box Attacks on Typed Decision Modelsprimary disclosure
Section 3 benchmark attack suite; Section 4 attack results
Controls and mitigations
- Treat typed model output as evidence for a decision, and separately enforce application-owned access, payment and consequence policy.
Unknowns and contradictions
- Threshold calibration, source provenance and exact operational effect under deployment-specific human review remain uncertain.
Sources and citation
Material revision history
- Oct 28, 2026 · Canonical change recorded · new in release · revision 199