Hugging Face Open-source DeepResearch

Hugging Face has developed and open-sourced an agentic framework designed to reproduce the capabilities of OpenAI's Deep Research. By utilizing a code-native agent approach, the system achieved a 55.15% score on the GAIA benchmark validation set, significantly improving upon previous open-source framework benchmarks.

The Role of Agent Frameworks in LLM Performance

An agent framework is a layer integrated on top of a Large Language Model (LLM) that enables the model to execute actions—such as browsing the web or reading PDF documents—and organize these operations into a structured series of steps.

Integrating LLMs into agentic systems provides substantial performance gains over standalone models. Hugging Face reports that using the smolagents library can increase performance by up to 60 points in certain benchmarks. This mirrors findings from OpenAI, which noted that Deep Research performed significantly better than standalone LLMs on the "Humanity’s Last Exam" benchmark.

Benchmarking with GAIA

The General AI Assistants (GAIA) benchmark is used to evaluate these systems because it requires high-level planning and rigorous execution. GAIA questions typically involve:

  • Constrained formatting: Requiring answers in a specific structure.
  • Multimodal capabilities: Extracting information from images.
  • Information gathering: Finding multiple interdependent pieces of data.
  • Complex chaining: Executing a problem-solving trajectory in the correct order.

While standalone GPT-4 scores below 7% on the GAIA validation set, OpenAI's Deep Research reached 67.36%. The Hugging Face open-source implementation currently reaches 55.15%.

Technical Implementation: CodeAgents and Tooling

The Advantage of Code-Native Agents

Hugging Face utilizes a "CodeAgent," where the agent expresses its actions in code rather than JSON. This approach offers several technical advantages:

  • Conciseness: Code can express complex sequences of actions more efficiently. For example, running parallel streams of actions requires significantly fewer steps in code than in JSON blobs.
  • Efficiency: Code actions require approximately 30% fewer steps than JSON, reducing token generation and lowering operational costs.
  • State Management: Code allows for better handling of state, particularly for multimodal tasks, as images or audio can be assigned to variables and reused across multiple steps.
  • Performance: LLMs generally perform better with code due to extensive exposure to code during training.

Tool Integration

The current proof-of-concept utilizes two primary tools adapted from Microsoft Research's Magentic-One:

  1. Text-based Web Browser: A simple browser for initial web interaction.
  2. Text Inspector: A tool capable of reading various text file formats.

Results and Comparison

In a 24-hour reproduction sprint, Hugging Face improved the open-source state-of-the-art on GAIA from approximately 46% (Magentic-One) to 55.15%. The importance of the code-native approach was highlighted when switching the same setup to a JSON-based agent, which caused performance to drop to 33% on the validation set.

Future Roadmap and Community Efforts

To reach full parity with proprietary systems, Hugging Face identifies the need for improved browser interaction. The next phase of development focuses on building GUI agents capable of viewing a screen and interacting directly via mouse and keyboard.

Other community implementations of Deep Research have emerged from contributors including dzhng, assafelovic (gpt-researcher), nickscamara, jina-ai, and mshumer. Hugging Face aims to further benchmark these findings using open LLMs like DeepSeek R1 and exploring the integration of vision LMs.

Sources