The Devaluation of Vulnerability Reports in the Age of LLMs

The Shift from Insight to Triage

Large Language Models (LLMs) have fundamentally changed the nature of vulnerability reporting by automating the discovery of "shallow" bugs. Previously, a confidential vulnerability report was viewed as a "special" contribution—a gift from a researcher that warranted a high level of responsiveness and care from maintainers. Today, the ability for anyone to run an LLM to find potential issues has shifted the bottleneck from the discovery of vulnerabilities to the assessment of their validity.

As a result, the signal-to-noise ratio in security inboxes has collapsed. Maintainers are now inundated with a volume of reports that often lack real-world impact or are outright hallucinations, making it increasingly difficult to identify and prioritize critical security flaws.

The "Vulnpocalypse": AI Slop and Maintainer Burnout

The influx of LLM-generated reports has created a systemic crisis for open-source maintainers and security engineers. The primary challenge is no longer the scarcity of insight, but the overwhelming volume of "slop"—reports that may be technically correct in a vacuum but are not exploitable or relevant to the user.

The Triage Burden

Security teams are reporting a drastic increase in the manual labor required to filter reports. One security engineer noted that tasks that previously took one person a day now require multiple people working full-time just to sift through the garbage. This leads to a dangerous paradox: critical, verified vulnerabilities (including those with Proof-of-Concepts and patches) are being ignored or ghosted because they are buried under a mountain of low-quality AI reports.

The Rise of Automated Triage

To combat this, some organizations are turning to the same technology causing the problem. Companies are implementing LLM-based pre-triage tools to assess the legitimacy of a report before it ever reaches a human engineer, effectively using AI to filter AI.

Divergent Perspectives on the Value of Research

There is significant debate among practitioners regarding whether vulnerability reports were ever truly "special" and what the future holds for the profession.

The "Always Noise" Argument

Some argue that vulnerability reports have always been plagued by low-quality submissions and "beg-bounties" from script-kiddies. From this perspective, LLMs have not changed the nature of the problem, only the scale and the professional appearance of the spam.

The "Human Insight" Argument

Other experts contend that true security research is not about finding a bug, but about demonstrating impact and communicating a solution.

"Vulnerability reports aren’t special. The demonstration of security impact through vulnerability reports was special. The automation of ‘demonstration of impact’ with AI isn’t that at all. The last mile is human and always was."

This view suggests that while LLMs can find the "what," they struggle with the "so what"—the actual adversarial thinking required to prove a vulnerability is a critical risk.

Future Outlook: Toward a New Equilibrium

Industry participants are speculating on how the security ecosystem will evolve as LLMs continue to iterate.

The Exhaustion of Low-Hanging Fruit

One theory is that the current surge is temporary. As LLMs find and fix the majority of shallow bugs, the stream of trivial reports will dry up, and the value of deep, human-led research will return to the forefront. This would lead to a world where only truly complex vulnerabilities remain.

The Shift to Exploit Development

As simple bug discovery becomes commoditized, the career path for security researchers may shift toward high-level exploit development. The ability to prove real-world impact will become the primary differentiator for researchers seeking to maintain their value.

Engineering-Level Solutions

Some argue that the solution is not better models, but better engineering. This includes a broader adoption of memory-safe languages (like Rust) and the development of new CS research to eliminate entire classes of problems, such as TOCTOU (time-of-check to time-of-use) issues and authentication failures, rather than relying on AI to catch bugs after they are written.