Anthropic OSS Scanner

Anthropic has launched OSS Scanner, a free opt-in service that uses its strongest language models to identify vulnerabilities in critical open-source software. This initiative aims to provide maintainers with a defensive advantage by delivering high-quality, raw model-generated reports and patches faster than human-led triage processes can allow.

Automated Vulnerability Discovery at Scale

OSS Scanner provides periodic security scans for eligible open-source projects at no cost. Unlike Anthropic's general-access Claude Security product for enterprises, OSS Scanner is specifically designed for the open-source ecosystem to accelerate the discovery and remediation of security flaws.

Key technical characteristics of the service include:

  • Model-Driven Analysis: Scans are performed by Anthropic's most capable models, including Claude Mythos.
  • Raw Model Output: To maximize speed and frequency, reports are generated entirely by models without human review or triage. This means reports may occasionally be incorrect or invalid.
  • Comprehensive Reporting: Each report includes a self-contained reproducer, an explanation of the vulnerability (including a bisection to determine when the bug was introduced, where possible), and a candidate patch when available.

Performance and Validation

Anthropic reports a significant increase in the efficacy of LLMs for vulnerability discovery. On the CyberGym benchmark, LLM performance in finding vulnerabilities rose from under 20% at the start of last year to over 85% this year.

To validate the OSS Scanner pipeline, Anthropic conducted a test involving expert penetration testers who reviewed 97 critical and high-severity vulnerabilities across 48 projects. The results showed:

  • High Precision: 85 of the 97 vulnerabilities (88%) met the bar for Anthropic's coordinated vulnerability disclosure (CVD) process.
  • Low False Positive Rate: Only one of the 97 findings was found to be completely invalid.
  • Real-World Impact: Initial disclosures to early testers included multiple vulnerabilities that could be chained to achieve unauthenticated remote code execution.

Maintainer Feedback and Integration

Early adopters from major projects have indicated that the raw model outputs are highly actionable.

"An unusually high fraction of OSS Scanner’s findings uncovered PostgreSQL defects. Several reports came with fixes we can use nearly as-is, and fast-track access let us address the newest issues before they reached a GA release."’ — Noah Misch, PostgreSQL

"Early AI reports about 18 months ago, before Project Glasswing, were appalling. The reports we received from Anthropic, raw model output included, were as good and sometimes better than what we get from people."— Anton Arapov, OpenSSL Corporation

"We found the signal from these reports high: of the 74 reports we received, all but two were valid, and five became CVEs."— Todd Ouska, wolfSSL

While maintainers have noted that severity ratings can sometimes be inflated or that the scanner may misunderstand a project's specific threat model, the overall signal-to-noise ratio is reported as high.

Eligibility and Enrollment

OSS Scanner is available to core maintainers of projects that have a "critical impact on infrastructure and user security," following a criteria set similar to Google's OSS-Fuzz. Enrollment is handled via a pull request to the official OSS Scanner GitHub repository.

Anthropic continues to maintain its coordinated vulnerability disclosure (CVD) process for human-verified reports, particularly for projects that lack the resources to perform their own triage.

Sources