Anthropic and CMU Research on Cyber Toolkits for LLMs

Anthropic and Carnegie Mellon University's CyLab have demonstrated that Large Language Models (LLMs) not specifically fine-tuned for cybersecurity can successfully execute multi-stage cyber attacks on networks containing dozens of hosts when equipped with a specialized toolkit. This research highlights a pathway where LLMs could reduce the barriers to entry for complex cyber attacks while simultaneously providing a framework for automating cyber defensive workflows.

The Incalmo Toolkit and LLM Performance

Incalmo is a cyber toolkit that acts as a translator, converting an LLM's high-level attack planning into the specific computer commands required to execute an attack. The research found that LLMs using Incalmo achieved significantly higher success rates in compromising simulated networks compared to those without the toolkit.

Success Rates: LLMs equipped with Incalmo successfully fully compromised 5 out of 10 test networks and partially compromised 4 others. In contrast, LLMs without the toolkit experienced almost complete failure.
Attack Complexity: Successful attacks involved orchestrating complex sequences, including gaining initial network access, lateral movement between systems, and data exfiltration across networks consisting of 25-50 hosts.
Baseline Performance: Without Incalmo, no tested LLM realized an end-to-end multi-stage attack in any of the ten environments. Only Claude Sonnet 3.5 was able to exfiltrate a single file in a 4-layer chain environment.

Simulation Scenarios and Results

Researchers tested six LLMs across ten simulated networks, including high-fidelity simulations of historical breaches.

Equifax Breach Simulation: All tested models achieved at least partial success in simulating the Equifax data breach when using Incalmo.
Colonial Pipeline Simulation: All but one LLM demonstrated either complete or partial success in simulating the Colonial Pipeline attack.
Operational Parameters: Attacks were carried out autonomously by the LLMs with minimal manual intervention; prompting was limited to introducing the scenario, the goal, and the Incalmo toolkit.

Technical Limitations and Constraints

While the results demonstrate a capability gain, the researchers noted several critical limitations to the current state of of the research:

Known Vulnerabilities: The attacks relied on known vulnerabilities rather than the discovery of novel (zero-day) vulnerabilities.
Scenario-Specific Tooling: Some tools within Incalmo were developed specifically for the research scenarios, meaning new tools would be required to threaten real-world networks.
Lack of Active Defenses: The simulated networks lacked active defenses, making them easier to compromise than real-world environments.

Implications for Cyber Attack and Defense

The ability of LLMs to automate complex attack chains suggests that the cost and skill required to conduct multi-stage attacks may decrease as models scale and toolkits improve.

Offensive Risks

General scaling of LLMs improves these capabilities. For example, Claude Sonnet 3.5 outperformed the smaller Claude Haiku 3.5 in scenarios where neither had access to Incalmo. The researchers warn that as the cost of using LLMs falls and capabilities improve, malicious actors may find it easier to execute complex attacks.

Defensive Opportunities

The ability of LLMs to emulate human cyber attack profiles provides opportunities for automated penetration testing. This could significantly accelerate the identification and remediation of network vulnerabilities.

Future Research Directions

Further research is required to understand the performance gains possible through specific cyber fine-tuning and the efficacy of LLM-based attackers against networks with active defenses.