Anthropic Claude 3.5 Sonnet Computer Use Capability

Anthropic has released a public beta enabling the latest version of Claude 3.5 Sonnet to use computers. The model can follow user commands to move a cursor, click on screen locations, and input text via a virtual keyboard, allowing it to interact with software as a human would rather than relying on bespoke tools.

Technical Implementation and Research

Claude 3.5 Sonnet achieves computer use by combining multimodality and reasoning. The model interprets screenshots of the user's screen and calculates the exact pixel coordinates required to move the cursor and click the correct location.

Key technical insights from the development process include:

• Pixel-Level Accuracy: Training the model to count pixels accurately was essential for reliable mouse commands, addressing a common LLM struggle with precise counting.
• Generalization: The model demonstrated rapid generalization, moving from training on simple software like calculators and text editors to executing complex sequences of logical steps based on written prompts.
• Self-Correction: During tasks, the model was observed to self-correct and retry actions when encountering obstacles.
• Performance Benchmarks: On the OSWorld evaluation, Claude 3.5 Sonnet achieved a score of 14.9%, significantly higher than the next-best AI model's 7.7%, though still well below human-level performance (70-75%).

Safety Framework and Risk Mitigation

Anthropic has categorized the computer use capability as a way of lowering the barrier to existing cognitive skills rather than increasing those skills. Consequently, the updated Claude 3.5 Sonnet remains at AI Safety Level 2 according to the company's Responsible Scaling Policy.

Primary Safety Concerns

• Prompt Injection: Because Claude interprets screenshots from internet-connected computers, it is vulnerable to prompt injection attacks where malicious content on a screen could override user instructions.
• Intentional Misuse: To prevent abuse, Anthropic has implemented classifiers to flag misuse. Specifically, the company is monitoring for election-related activity and has implemented systems to nudge the model away from registering web domains, interacting with government websites, or posting to social media.

Anthropic argues that introducing computer use at AI Safety Level 2 allows the company to address safety issues before the model reaches higher safety levels (Level 3 or 4) where catastrophic risks are more prevalent.

Current Limitations and Future Direction

Anthropic describes the current state of computer use as a shift from "making tools fit the model" to "making the model fit the tools." However, several limitations persist:

• Input Constraints: The model currently cannot perform actions such as dragging or zooming.
• Observation Gaps: Claude views the screen as a "flipbook" of screenshots rather than a continuous video stream, which can lead to missing short-lived notifications or actions.
• Reliability: The capability is currently slow and error-prone, with reported instances of the model accidentally stopping screen recordings or deviating from tasks to browse unrelated content.

Anthropic intends to improve the speed, reliability, and ease of implementation for non-developers as the capability evolves.