Hugging Face CodeAgents + Structure: Improving Agent Reliability via Structured Generation

Hugging Face has introduced a method to enhance AI agent reliability by combining the expressiveness of code-based actions with the reliability of structured generation. By forcing CodeAgents to generate both their internal reasoning (thoughts) and their executable Python code within a structured JSON blob, agents achieve significantly higher success rates across multiple benchmarks compared to traditional tool-calling or standard code-generation approaches.

The Evolution of Agent Actions

AI agents typically execute actions through one of three primary paradigms:

  • Traditional JSON Agents: These agents select from predefined tools and generate JSON-formatted calls. While reliable, they lack composability and are limited to a rigid set of predefined actions, making it struggle with tasks requiring intermediate state management across multiple calls.
  • Code Agents: These agents write executable Python code directly, allowing them to call tools within loops, functions, and conditional statements. This provides unlimited flexibility and the ability to maintain state, but parsing the resulting code from markdown blocks can be error-prone.
  • Structured CodeAgents: This approach enforces the generation of a JSON object containing a thoughts field for reasoning and a code field for execution. This combines the reliability of structured parsing with the full expressiveness of Python code.

Benchmark Results and Performance Gains

Testing across benchmarks including GAIA, MATH, SimpleQA, and Frames shows that code actions combined with structured generation consistently improve performance for capable models. On average, the structured approach outperformed regular CodeAgents by 2 to 7 percentage points.

Key observations by model family include:

  • OpenAI models: Demonstrated the largest improvements, particularly on reasoning-heavy tasks.
  • Claude models: Showed strong results, with Claude 3.7 Sonnet performing especially well.
  • Qwen models: Generally improved, though smaller models began to experience a "structure tax."

Why Structured Generation Improves Success Rates

Eliminating the Parsing Problem

Parsing errors in markdown-based code extraction frequently lead to agent failure. An analysis of 15,724 agent traces revealed that 2.4% of traces had parsing errors in their first call. The impact of these errors is severe:

  • Traces without parsing errors: 51.3% success rate.
  • Traces with parsing errors: 42.3% success rate.

Agent traces without parsing errors succeed 21.3% more often than those with parsing errors. Furthermore, parsing errors increase the average number of steps taken to solve a problem from 3.18 to 4.63.

Enforced Reasoning Process

By requiring a thoughts field, the agent is forced to articulate its reasoning before executing code. This leads to more systematic planning and allows the agent to catch logical errors early in the process.

The "Structure Tax" and Model Capability

Structured generation is not universally beneficial. There is a capability threshold where models must have sufficient instruction-following and JSON pre-training to handle the cognitive load.

Smaller models (such as mistralai/Mistral-7B-Instruct-v0.3) may suffer from a "structure tax," where the overhead of maintaining JSON syntax while simultaneously writing Python code leads to syntactically broken code (e.g., malformed strings or extra commas), resulting in immediate SyntaxError failures.

Implementation and Usage

For users of the smolagents library, structured output can be enabled by setting use_structured_outputs_internally=True when initializing a CodeAgent.

Recommended Use Cases

Use Structured CodeAgents when:

  • Using capable models (frontier models or those with 32B+ parameters).
  • Executing tasks that require complex reasoning and code execution.
  • Reliable parsing of agent outputs is required.

Consider alternatives when:

  • Using smaller models that struggle with structured generation.
  • Simple, predefined workflows are sufficient.

Implementation Tips

  1. Clear Prompting: Prompts must explicitly specify the expected JSON structure.
  2. Model Selection: Choose models with strong structured generation capabilities.
  3. Provider Support: Use API providers (such as OpenAI or Anthropic) that support structured generation natively to ensure maximum reliability.

Sources