Finding bugs with Claude and property-based testing

Anthropic has developed an AI agent capable of efficiently identifying bugs in large software projects by autonomously inferring general code properties and applying property-based testing (PBT). By utilizing a technique similar to fuzz testing, the agent discovered bugs in prominent Python packages including NumPy, SciPy, and Pandas.

Property-Based Testing vs. Example-Based Testing

Property-based testing focuses on verifying whether a general invariant or property of the code holds true across a wide range of inputs, rather than verifying specific, manually defined examples.

• Example-based tests: A developer defines a specific input (e.g., [2, 10, 5, 4]) and verifies the output matches an expected result (e.g., [2, 4, 5, 10]). This approach often misses edge cases the developer failed to anticipate.
• Property-based tests: A developer specifies a general property (e.g., "JSON deserialization is the inverse of serialization") and the input domain. A framework then automatically generates a vast array of valid inputs to search for a counterexample that breaks the property.

Anthropic's agent automates this process by reading type annotations, docstrings, function names, and comments to infer these properties and then writing tests using the Hypothesis library.

The Property-Based Testing Agent Workflow

The agent is implemented as a custom Claude Code command. It accepts a target—such as a Python file, module, or specific function—and follows a five-step iterative process:

1. Analysis: Reads the code and documentation to understand the target and its relationship to the codebase.
2. Proposal: Proposes properties grounded in the analysis.
3. Implementation: Writes property-based tests using Hypothesis.
4. Reflection: Runs the tests and evaluates the results. If a test fails, the agent determines if a genuine bug was found or if the test itself needs adjustment. If it succeeds, the agent assesses if the test was trivial or meaningful.
5. Reporting: If confident in a bug's validity, the agent generates a formatted bug report.

To manage long-range reasoning, the agent utilizes a to-do list. Researchers noted that self-reflection capabilities improved significantly with Opus 4.1 and Sonnet 4.5 compared to Sonnet 4.

Real-World Performance and Validation

Researchers tested the agent on over 100 popular PyPI packages. The evaluation was conducted in two phases:

Phase 1: Initial Evaluation (Opus 4.1)

Of 984 bug reports generated by Claude Opus 4.1, a manual review of 50 reports found that 56% were valid bugs, and 32% were valid and reportable. To improve precision, the team developed a 15-point rubric to rank bugs. When applying this rubric, 86% of the top-scoring reports were valid, and 81% were both valid and reportable.

Phase 2: Refined Evaluation (Sonnet 4.5)

The team ran the agent on 10 key packages multiple times using Sonnet 4.5 and employed a more sophisticated evaluation agent to check correctness and severity before final human expert review.

Case Studies: Identified Bugs

Several bugs found by the agent have been reported and patched in major libraries:

• NumPy: The agent discovered that numpy.random.wald sometimes returned negative numbers, violating a property of the Wald distribution. The fix involved addressing a catastrophic cancellation to create a more numerically stable formulation, reducing relative error by nearly ten orders of magnitude.
• aws-lambda-powertools: The agent found that slice_dictionary() returned the first chunk repeatedly because the iterator was not incremented. This was identified by testing the property that slicing and reconstructing a dictionary should return the original.
• cloudformation-cli-java-plugin: The agent identified that item_hash() produced the same hash for all lists because it used the in-place .sort() method, which returns None. This was caught by testing that different inputs should produce different hashes.
• tokenizers: The agent found a missing closing parenthesis in EncodingVisualizer.calculate_label_colors(), resulting in invalid HSL CSS, by testing the output against a regex for HSL color codes.

One reported issue in python-dateutil regarding the Julian calendar was marked as invalid by maintainers, highlighting a limitation: the agent struggles with code containing subtle or implicit assumptions that only maintainers can define.

Future Directions

Anthropic views agentic PBT as a critical complement to human testing, particularly as LLMs improve at identifying properties from context. The researchers suggest that the next logical step is the automatic generation of patches. If correctness properties can be fully specified, correcting bugs becomes simpler, potentially allowing LLMs to propose high-quality patches for maintainer review.