Models changed refusal behavior when a safety test was declared in different languages

An evaluation label is information the model receives, not an external guarantee that the answer will be safer. In a study spanning four models and six languages, explicitly describing a request as a safety test increased harmful-request refusals overall but did not produce an identical effect in English and the sampled Indic languages. Benign prompts also experienced additional refusals in some conditions.

The distinction matters because a dashboard can appear safer under declared evaluation than under ordinary user framing. The controlled measurements identify response-level differences under specific prompts and translations; they do not establish that a model strategically recognized surveillance or deliberately concealed future actions. Translation quality, sampled tasks and the choice of “harmful” labels all affect comparison.

To verify a deployed guard, matched language pairs need independent native-language review and downstream action checks, not only a refusal text label. An application that permits a transaction in one language but denies an equivalent authorized request in another has a consistency failure even if neither path leads to an exploit. The observed study exposes a candidate test dimension, not a production attack count.