Evidence · DiggingBeagle record
Testing Conversation Preservation Across LLM APIs: isloss
Independent evaluation of role, system instruction, tool and error semantic loss during cross-provider API translation.
- Published
- Nov 11, 2026
- Source role
- primary disclosure
Evidence record
Independent evaluation of role, system instruction, tool and error semantic loss during cross-provider API translation.
Claim-level citations (2)
- supportsAPI gateways sometimes returned success after silently changing the authority or error state of an agent conversation: The study normalized source and destination message structures to compare system, user, assistant and tool roles, tool-error flags, arguments and sequence boundaries. Some translations remained valid JSON and passed HTTP checks while dropping the information that determines how a downstream model should interpret or respond to a tool result. Across 685 completed source-valid translation cells, 127 met the researchers' stricter avoidable, silent role/prompt/tool-loss definition, about 18.5 percent. A separate set of 558 HTTP-200 response streams included 203 outputs with SDK exceptions or silent corruption. These are different denominators and should not be added.
isloss paper, Table of 685 cells, 127 primary findings and 203/558 response stream errors
- supportsAPI gateways sometimes returned success after silently changing the authority or error state of an agent conversation: This does not prove every production gateway exhibits the same failure rate or that these specific translations caused external unauthorized writes. The demonstrated control weakness is that schema compatibility is not semantic preservation. The evidence is bounded by the cited source and test setting. Broader incidence, other environments, and actual production consequences are not established merely by these observations.
Primary disclosure: results, scope and limitations