AI SOC triage methods cover different alerts, but transfer across tenants breaks

A November 3 retrospective study compares a PPO reinforcement-learning ranking policy with an autoencoder and Isolation Forest. The analysis includes 559,792 training events and 74,909 analyst-labeled test alerts, divided into 503 feature-pattern clusters. At the constrained budget of 32 selected clusters, the authors report a cluster-level benign-escalation fraction of 0.000 for PPO, 0.007 for the autoencoder and 0.041 for Isolation Forest. This score answers how many selected clusters had benign content under the experiment's definition; it is not the overall false-positive rate for all held-out alerts.

The three methods chose different areas of the test population. PPO and Isolation Forest had approximately 0.007 Jaccard overlap at that cluster budget, so two filters with useful scores can still examine nearly disjoint event groups. Removing PPO's benign-escalation reward increased its reported cluster metric to roughly 0.079, illustrating the dependence on the training objective and evaluation contract.

The paper's zero-shot transfer experiment measures Isolation Forest and the autoencoder on a second tenant, not a demonstrated live PPO deployment. Their reported AUC values fell from 0.7126 to 0.4681 and from 0.7682 to 0.4730, respectively, after differences in SIEM feature semantics and representation. Their sub-0.5 results are an out-of-distribution warning, not evidence that a deployed SOC was compromised or that PPO has been evaluated across those tenants. An earlier draft's Isolation Forest score direction was corrected by the authors.

Operational evaluation would have to freeze the feature mapping, report both cluster- and event-level denominators, distinguish selected benign clusters from missed threats, and test a second tenant after explicit schema normalization. The evidence-tier field note explains why the attractive 0.000 cannot be promoted to a system-wide safety guarantee.