What an AI security score actually counts

A model can score 51.2% Macro-F1 without detecting 51.2% of a dataset's individual vulnerabilities. That distinction matters in the new cross-chain bridge study: its headline combines a dataset of 830 audit findings with an evaluation that predicts vulnerability patterns for 209 report-level code packages. The two numbers describe different units.

Similar distinctions appear in ToFu's tool-fuzzing results and Microsoft FORGE's validation costs. Read together, the studies show how to evaluate a security result without turning an inventory into execution coverage or a successful subset into an end-to-end cost. This is a comparison of measurement design, not a ranking of the systems.

The bridge experiment defines 11 patterns and asks each model to make multi-label predictions over audit-scoped source packages. Some patterns are common and others rare. A constant predictor that reports all patterns for every report therefore receives a nonzero F1 score, even though it does no code analysis. Macro-averaging its pattern scores gives 29.3%.

Composer-2.5 reaches 51.2% and GPT-5.5 33.8%, improvements of 21.9 and 4.5 percentage points over that reference. DeepSeek-V4-Pro and Opus-4.8 fall below it. The baseline does not become a useful production auditor merely by beating a model in this calculation. Its purpose is to expose how much a result can gain from the label distribution alone.

The selected reports already contain retained verification findings. Pattern-level precision can penalize predicting a category absent from a report, but the experiment does not establish an operational false-alarm rate for entirely clean bridge projects. Nor does its Macro-F1 score express the chance that a particular transfer path is safe. Those require different populations and evidence.

ToFu starts from a larger repository inventory and narrows it to software the authors could prepare and exercise. Its 273 cleanly executable repositories contain 3,405 extracted tools, but only a subset of those tools was actually exercised. The authors then manually verify and deduplicate candidate findings, retaining 253 bugs affecting 343 tools across 118 repositories. Neither the extracted inventory nor the bug count is a count of compromised production deployments.

FORGE's $3.61 and 21.5-minute averages have a different filter: they cover all attempts associated with 182 successful crash cases. Microsoft excludes initial screening, failed or filtered candidates, human investigation and patch preparation. The measured automation can be useful while the total cost per accepted and serviced vulnerability remains unknown.

The editorial inference is practical: a security result should travel with its unit, selection rule, baseline and stopping point. For the bridge study, the stopping point is pattern detection under supplied context. For ToFu it is reproduced implementation behavior in the tested configuration. For the FORGE cost figure it is successful crash validation, before the full human and servicing process.

These boundaries do not erase the findings. They identify which question each finding can answer and which experiment would be needed next. A model leaderboard, a tool inventory and an automation cost can all be accurate while supporting very different decisions. Keeping their denominators visible is what makes the evidence usable.