Hugging Face Transformers Code Agent GAIA Benchmark Results

Hugging Face has developed a Code Agent using the transformers.agents library (now evolved into the smolagents library) that achieved a top ranking on the GAIA benchmark, one of the most challenging benchmarks for AI agents. The system demonstrates that allowing agents to express actions through Python code rather than JSON blobs significantly improves efficiency, reduces token costs, and enhances the agent's ability to handle complex, multi-step trajectories.

GAIA Benchmark Challenges

GAIA is designed to test agents on tasks that require high-level planning and rigorous execution. A typical GAIA question may require multimodal capabilities (reading images), gathering disparate pieces of information from the web, and adhering to strict output constraints. These tasks often require chained steps where later information depends on previous findings, highlighting the common struggle LLMs face with planning and execution.

The Advantage of Code-Based Actions

Rather than using dictionary-like outputs such as JSON, the Hugging Face agent uses a "Code Agent" approach where actions are formulated and executed as Python code. This method provides several technical advantages:

  • Conciseness and Efficiency: Code is a more optimized way to express complex sequences. For example, parallel streams of actions can be handled in a single code step rather than multiple JSON blobs. Research indicates that code actions can require 30% fewer steps than JSON, reducing generated tokens and lowering operational costs.
  • Intuitive Variable Management: Code allows the agent to store tool outputs as named variables (e.g., rock_image = image_generation_tool("A picture of a rock")), making it easier for the LLM to reference these outputs in subsequent steps compared to the complex naming gymnastics required in JSON.
  • Model Fluency: Because LLMs are trained on vast amounts of code, they are often more fluent in writing code than in writing JSON.

Secure Code Execution Implementation

To mitigate the risks of executing LLM-generated code, Hugging Face implemented a secure Python interpreter from the ground up using the ast (Abstract Syntax Tree) module. Instead of a "blacklist" approach (forbidding certain actions), the interpreter uses a "whitelist" approach:

  • AST-Based Execution: The interpreter executes tree nodes one by one and stops at any operation not explicitly authorized.
  • Authorized Imports: Only imports explicitly listed in the authorized_imports list are executed; standard functions like print and range are included, while dangerous functions like open are forbidden by default.
  • Safety Guardrails: The system caps the number of operations to prevent infinite loops and limits the number of lines in print outputs to prevent the LLM's context window from being flooded with junk data.

Multi-Agent Orchestration and Planning

The system utilizes a multi-agent architecture to manage context and reduce noise, particularly during web browsing:

Orchestration Structure

  • Manager Agent: A ReactCodeAgent that handles high-level task solving and has access to a file_inspector, a visualizer, and a search_agent.
  • Search Agent: A JSON-based agent wrapped as a tool for the Manager Agent. It handles sequential web browsing tasks (using tools like informational_web_search, page_down, and find_in_page) and returns only the relevant information to the manager to avoid cluttering the context.

Planning Workflow

The agent employs a "plan-ahead" workflow where, every N steps, it generates a summary of known and needed facts and a step-by-step plan.

  • Tuning: The manager agent updates its plan every 2 steps (N=2), while the search agent updates every 5 steps (N=5).
  • Context Optimization: Hugging Face discovered that omitting the previous version of the plan from the prompt improves scores, as it prevents the LLM from being biased toward outdated plans and encourages re-evaluation of the approach.

Performance Results

Using GPT-4o without fine-tuning, the agent achieved the following results on the GAIA benchmark:

  • Validation Set: 44.2%, ranking #1 overall.
  • Test Set: 33.3%, ranking #2 overall, outperforming Microsoft Autogen's submission.
  • Level 3 Questions: The agent achieved the best average score on the most difficult "hardcore" Level 3 questions.

Future Improvements

Hugging Face identified several paths for further optimization:

  • LLM Engine: Exploring fine-tuned open-source models to reduce parsing errors.
  • Orchestration: Moving toward more seamless multi-agent orchestration.
  • Web Tools: Integrating the selenium package to handle JavaScript and cookie banners.
  • Planning: Testing alternative planning strategies from current literature.

Sources