Anthropic "think" tool for Claude

Anthropic has introduced the "think" tool, a mechanism that allows Claude to create a dedicated space for structured reasoning during complex tasks. This tool significantly improves Claude's ability to follow policies, make consistent decisions, and handle multi-step problems with minimal implementation overhead.

The "think" tool vs. Extended Thinking

While similar in purpose, the "think" tool is distinct from Claude's "extended thinking" capability. The primary difference lies in when the reasoning occurs:

  • Extended Thinking: Occurs before the model starts generating a response. It is used for deep consideration and iteration on a plan before taking action. It is recommended for simpler tool use (non-sequential calls), straightforward instruction following, and tasks like coding, math, and physics.
  • The "think" tool: Occurs after the model has started generating a response. It allows Claude to stop and evaluate if it has all the necessary information to proceed, making it ideal for processing external information discovered via tool calls.

Anthropic recommends the "think" tool for scenarios involving complex tools, long chains of tool calls, policy-heavy environments, or sequential decisions where each step builds on the previous one.

Performance Benchmarks

τ-Bench Results

τ-bench tests a model's ability to use tools in realistic customer service scenarios. Anthropic evaluated Claude 3.7 Sonnet using the pass^k metric, which measures the probability that all k independent trials are successful, emphasizing consistency and reliability.

  • Airline Domain: The combination of the "think" tool and an optimized prompt resulted in a pass^1 score of 0.584, compared to 0.332 for the baseline. This represents a 54% relative improvement in the airline domain.
  • Retail Domain: The "think" tool alone achieved a pass^1 score of 0.812, compared to 0.783 for the baseline. Because the retail policy is simpler than the airline policy, Claude improved without needing additional prompting.

SWE-bench Results

Integrating a "think" tool into the SWE-bench setup contributed to Claude 3.7 Sonnet achieving a state-of-the-art score of 0.623. Isolated experiments (n=30 with the tool, n=144 without) showed that the "think" tool improved performance by an average of 1.6% (Welch's t-test: t(38.89) = 6.71, p < .001, d = 1.47).

Implementation Best Practices

To maximize the effectiveness of the "think" tool, Anthropic recommends two primary strategies:

  1. Strategic Prompting: Provide clear instructions and domain-specific examples in the system prompt. This should include the expected level of detail, how to break down complex instructions, decision trees for common scenarios, and verification steps for information collection.
  2. System Prompt Placement: For long or complex guidance, placing instructions in the system prompt is more effective than including them in the tool description itself.

When to Use (and Not Use) the "think" tool

Recommended Use Cases

  • Tool Output Analysis: When the model must process tool outputs and potentially backtrack.
  • Policy-Heavy Environments: When strict adherence to detailed guidelines is required.
  • Sequential Decision Making: When multi-step domains where mistakes are costly.

Non-Recommended Use Cases

  • Non-sequential Tool Calls: Tasks requiring only a single or parallel tool calls.
  • Simple Instruction Following: Tasks with few constraints where default behavior is sufficient.

Technical Implementation

Developers can implement the "think" tool using a standard tool specification. A sample implementation based on τ-Bench is provided below:

{
  "name": "think",
  "description": "Use the tool to think about something. It will not obtain new information or change the database, but just append the thought to the log. Use it when complex reasoning or some cache memory is needed.",
  "input_schema": {
    "type": "object",
    "properties": {
      "thought": {
        "type": "string",
        "description": "A thought to think about."
      }
    },
    "required": ["thought"]
  }
}

Anthropic notes that while these results focused on Claude 3.7 Sonnet, performance gains were also observed with Claude 3.5 Sonnet (New), indicating the capability generalizes across models.

Sources

Related