Transformers Agents 2.0 release notes / what's new

Hugging Face has released Transformers Agents 2.0, a modular framework that enables Large Language Models (LLMs) to use tools and iterate based on past observations to solve complex tasks. This update introduces new agent types capable of multi-step reasoning, which allows an open-source Llama-3-70B-Instruct agent to outperform several GPT-4 based agents on the GAIA Leaderboard.

Core Design Philosophy

Transformers Agents 2.0 is built on two primary design goals: clarity and modularity. The framework minimizes abstractions to ensure that error logs and attributes remain transparent and accessible for inspection. It conceptually separates the agent system (the vehicle) from the LLM engine (the power source), allowing users to create any agent type using any underlying LLM.

Key Framework Elements

  • Tool: A class used to implement specific actions. It consists of a callable forward method and attributes including name, descriptions, inputs, and output_type, which are used to dynamically generate usage manuals for the LLM's prompt.
  • Toolbox: A pre-instantiated set of tools provided to an agent to avoid the overhead of re-initializing tools during agent setup.
  • CodeAgent: A basic agent that generates actions as a single block of Python code without the ability to iterate on previous observations.
  • ReactAgent: Agents that follow a "Thought → Action → Observation" cycle. This is available in two variants: ReactCodeAgent (generates Python blobs) and ReactJsonAgent (generates JSON blobs).

Agent Workflows and Execution

An agent operates by allowing an LLM to utilize tools through a specific process: providing tool usage information in a prompt, parsing tool calls from the LLM output (via code or JSON), executing those calls, and maintaining a memory of previous tool calls and observations for iterative agents.

Self-Correcting Retrieval-Augmented Generation (RAG)

Transformers Agents 2.0 can be used to build self-correcting RAG systems. By providing an agent with a retriever tool that accepts dynamic parameters (such as specific knowledge base sources), the agent can adjust its search strategy based on observations. For example, if a restricted search returns no documents, a ReactJsonAgent can automatically decide to expand the search to all sources to find the required information.

Multi-Agent Orchestration for Web Browsing

Complex tasks can be solved by encapsulating specialized agents as tools for a higher-level agent. Hugging Face demonstrates this by creating a specialized "web surfer" agent (powered by Cohere's Command-R+) to handle the intricacies of web navigation. This web surfer is then wrapped in a SearchTool and provided to a primary task-solving agent (powered by Llama-3-70B-Instruct), allowing the high-level agent to delegate information gathering to the specialist while focusing on the final reasoning and calculation.

Performance and Benchmarking

LLM Engine Comparison

Using the agents_reasoning_benchmark (which utilizes a calculator and a basic search tool across HotpotQA, GSM8K, and GAIA datasets), Hugging Face compared several engines:

  • Llama-3-70B-Instruct: Performed on par with GPT-4 Turbo and led the open-source models. It showed particular strength when used within a ReactCodeAgent due to its strong coding capabilities.
  • Mixtral-8x7B: Performed less effectively in Code-based agents compared to JSON-based agents, as it more frequently failed to generate valid code.

GAIA Leaderboard Results

To tackle the full GAIA benchmark, Hugging Face implemented a multi-modal agent using ReactCodeAgent powered by Llama-3-70B-Instruct. The agent was equipped with a toolbox including a SearchTool (web browser), TextInspectorTool (document reader), SpeechToTextTool (using distil-whisper), and VisualQATool (using Idefics2-8b-chatty).

This configuration achieved a 4th place ranking on the GAIA Leaderboard, surpassing many GPT-4-based agents and establishing itself as a top contender in the open-source category.

Future Roadmap

Hugging Face plans to expand the framework with the following developments:

  • Implementation of pushing and loading full agents via the Hub (extending beyond current tool sharing).
  • Enhanced tools for image processing.
  • Advanced long-term memory management.
  • Improved multi-agent collaboration capabilities.

Note: transformers.agents has been upgraded to a standalone library called smolagents.

Sources