DIB-Bench showed why plausible AI outputs cannot attest the deployed stack

The DIB-Bench authors tested whether safety and correctness judgments over output text could detect that the system producing that output had been changed. Controlled modifications targeted the model, system instructions, tools, retrieval sources, dependencies and postprocessing. Seven thousand five hundred sixty scored benchmark examples were evaluated across these manipulation categories.

Under this deliberately tampered population, a rule-based judge accepted 88.6% of manipulated outputs, Qwen3-8B accepted 89.3%, and GPT-5-nano accepted 94.1%. A response can look right when a different tool was invoked, a retrieval index was swapped, or a postprocessor changed what was displayed. Component hashes and signed manifests answer an identity question that behavioral scoring cannot answer.

A secure runtime needs explicit assertions about effective tool versions, retrieval components, executable dependencies and model weights at the point of invocation. Hashes alone only identify what actually ran: they do not guarantee that signed software is benign or that an authorization check is correct.

The relevant failure occurs before anyone asks whether an answer is useful: model weights, instructions, retrieval data, tool implementations, dependencies or the postprocessor can be substituted while leaving the visible response plausible. In the authors' constructed evaluation, component identity distinguished all changed configurations; it was not inferred from the wording of an answer. That does not prove an arbitrary production manifest is complete. A package hash can miss mutable remote tools, runtime-selected plugins or a retriever index fetched after the measurement. Integrity attestation and transaction authorization should be tested as separate controls.

The associated execution-evidence pattern distinguishes what produced an answer from whether the answer appears plausible. DIB-Bench changes deployment components while leaving outputs sufficiently credible to fool selected judges. This is a component-identity test, not proof of what an underlying host executed and not evidence that a satisfying answer was authorized. A complete review would compare the tested manifest with tool-call traces and independently observed effects instead of promoting any one channel into a system-wide safety verdict.