A model safety score may change when the agent harness changes the task

Safety measurements obtained from direct text prompts do not automatically transfer to an agent that can use scientific tools, inspect images, persist a dialogue state and make repeated attempts. Recent controlled research shows that the surrounding task structure can change which harmful requests are answered and which ordinary requests are rejected. The evidence should be read as harness-dependent evaluation, not a single universal model safety score.

Michael Yew Chuan Lee and colleagues compare several scientific agent harnesses around multiple model variants on 50 harmful chemistry questions from their chosen evaluation. The study finds substantially different unsafe-answer rates between direct model interaction and specific scientific workflows. These are model/harness/test combinations; the work does not verify real chemical production, quantify unique end users harmed, or prove that every scientific tool lowers safety.

Mechanically, a harness can add decomposition, retrieval and specialist tool contexts that were not present in the base-model refusal test. Safety enforcement must therefore observe the proposed tool action, not only the original user text.

Omni-UBench aggregates more than 183,000 samples from over 35 sources across text, image, audio, video and combinations. The authors examine fifteen multimodal models or guards and report failures when problematic meaning depends on the relationship between image and text, alongside unnecessary refusals of benign inputs. Results from one modality should not be substituted for another's recall, and an overblocking system should not be declared superior by measuring only blocked harmful samples.

M3 frames jailbreak exploration as a stateful search process over several turns rather than one independent adversarial prompt. The authors report up to 53 percent improvement in an ASR@5 comparison against selected baselines. This is a relative benchmark-specific improvement; it is not a measured probability that any random five-message user conversation becomes harmful.

What transfers across these findings is a method, not a percentage. A production evaluation should pin the actual deployed harness, tool permissions, modal converters, model version and attack budget. It should report harmful effect, refusal, successful authorized task and false alarm separately. For connected tools, a real effect check is essential: an unsafe-looking text answer and an executed dangerous action are not the same event.

The research supports revisiting safety tests whenever a harness or modality changes. It does not establish that a model is universally safe or unsafe independent of its application.