LinguaGuardBench tests whether a guard makes the same security decision when a legitimate user request is translated while the proposed tool action and its security label stay fixed. The input language should not redefine the authority granted to an agent, yet a probabilistic classifier can treat different phrasing as a reason to accept or block the…
Inspect the ClaimsCase · DiggingBeagle record
Language-only changes flipped security guard decisions on the same agent state
A multilingual paired benchmark found that some tool-use security monitors changed attack/benign verdicts when the legitimate request was translated but actions and labels were unchanged.
Scope
arXiv:2610.22244, LinguaGuardBench multilingual AgentDojo-derived evaluations; identical tool states.
Assessment profile
Assessment method- Setting
- evaluation
- Exploitation
- not established
- Evidence
- Grade B
- Remediation
- not applicable
- AI role
- AGAINST AI
Basis and provenance
The inspected arXiv study provides 1,220 held-constant agent states, multilingual perturbations and 19 guard measurements. Its documented verdict instability is reproducible as an evaluation effect, but the sampled guards were not shown authorizing completed malicious transactions in real systems. Remediation is methodological rather than a product patch.
Assessed Oct 18, 2026 using diggingbeagle.assessment/1.
Evidence basis: CLM-CP13-LANG-PAIRS · CLM-CP13-LANG-SHIFT · CLM-ENRICH-LANG-GUARD-DELTA · LinguaGuardBench: language-only changes to security monitor decisions
At a glance
A security policy expressed as actor, resource and effect should not grant different permissions simply because a benign user sentence is in Amharic rather than French. An LLM guard can nevertheless treat language as an implicit feature. Deterministic effect authorization remains necessary even when a multilingual model is used for contextual warnings.
Read the implicationsTranslation semantic equivalence is approximate; no actual unauthorized operation is proved by verdict flips alone. Production multilingual traffic composition and monitoring rates unknown.
Limits and uncertaintyFull account
LinguaGuardBench tests whether a guard makes the same security decision when a legitimate user request is translated while the proposed tool action and its security label stay fixed. The input language should not redefine the authority granted to an agent, yet a probabilistic classifier can treat different phrasing as a reason to accept or block the identical operation.
Researchers used 1,220 base agent states, translations and paraphrases across five additional languages, with repeated and placebo runs across multiple monitors. In smaller guards, selected language conditions reduced attack recall by up to about 7.4 percentage points. Among guards below 13 billion parameters, placebo-adjusted decision flips rose from 1.5 percentage points for French to 4.4 for Amharic in the reported comparisons. Larger models sometimes overblocked legitimate actions instead. The direction and size of the effect depend on model and language; translations may not always preserve every nuance perfectly.
These are measured changes in guard verdicts, not documented breaches of an external system. A defensible control test keeps the actual destination, resource, tool arguments and authorized outcome constant while varying only the language of the surrounding request. Where the intended effect can be checked against a stable permission record, that executor-side decision should not inherit a translated request's incidental linguistic differences.
Claims & evidence
CLM-CP13-LANG-PAIRSThe benchmark holds tool state and outcome label fixed while varying legitimate request language over five tested languages and multiple paraphrases.supported
Basis: reported finding
- supportsLinguaGuardBench: language-only changes to security monitor decisionsprimary disclosure
Benchmark construction and equivalent request pairs
CLM-CP13-LANG-SHIFTSome sub-10B-parameter monitors exhibited attack-recall changes up to approximately 7.4 percentage points for controlled translations, with model-specific benign overblocks.supported
Basis: reported finding
- supportsLinguaGuardBench: language-only changes to security monitor decisionsprimary disclosure
Monitor recall, FPR and language effect tables
CLM-ENRICH-LANG-GUARD-DELTAAmong the nine evaluated guards smaller than 13B parameters, placebo-adjusted verdict flip rates rose from 1.5 percentage points for French to 4.4 points for Amharic in this held-constant task corpus.supported
Basis: reported finding
- supportsLinguaGuardBench: language-only changes to security monitor decisionsprimary disclosure
Abstract and main findings, nine small guards and placebo-adjusted language comparisons
Implications
A security policy expressed as actor, resource and effect should not grant different permissions simply because a benign user sentence is in Amharic rather than French. An LLM guard can nevertheless treat language as an implicit feature. Deterministic effect authorization remains necessary even when a multilingual model is used for contextual warnings.
Controls and mitigations
- Test equivalent requests across languages while holding exact tool calls, permissions and labels constant; measure false positives and false negatives separately.
- Use policy checks keyed to authenticated identity and operation arguments rather than relying exclusively on a language model's interpretation of the surrounding request.
Unknowns and contradictions
- Translation semantic equivalence is approximate; no actual unauthorized operation is proved by verdict flips alone.
- Production multilingual traffic composition and monitoring rates unknown.
Sources and citation
Material revision history
- Oct 18, 2026 · Canonical change recorded · new in release · revision 171