Evidence · DiggingBeagle record
We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs
A peer-reviewed USENIX Security 2025 study of 576,000 Python and JavaScript code samples from 16 models. It reports average hallucinated-package rates of at least 5.2% for commercial models and 21.7% for open-source models, identifies 205,474 unique nonexistent package names, and analyzes persistence across repeated prompts. The paper frames repeated hallucinated names as a package-confusion attack surface because an attacker can register a name before a later user follows the same recommendation.
- Published
- 2025-08 (month precision)
- Publisher
- USENIX Association
Evidence record
A peer-reviewed USENIX Security 2025 study of 576,000 Python and JavaScript code samples from 16 models. It reports average hallucinated-package rates of at least 5.2% for commercial models and 21.7% for open-source models, identifies 205,474 unique nonexistent package names, and analyzes persistence across repeated prompts. The paper frames repeated hallucinated names as a package-confusion attack surface because an attacker can register a name before a later user follows the same recommendation.