Open-source LLMs as LangChain Agents

Open-source Large Language Models (LLMs) have reached a performance threshold where they can serve as effective reasoning engines for agent workflows. Benchmarking by Hugging Face reveals that Mixtral-8x7B can surpass GPT-3.5 in general-purpose reasoning when integrated into an agentic system, suggesting that further fine-tuning for function calling could push open-source models toward GPT-4 levels of performance.

Understanding LLM Agents and the ReAct Framework

LLM agents are systems that use an LLM as a central engine to perform actions on an environment based on observations. These systems typically follow a cycle of Perception $\Rightarrow$ Reflexion $\Rightarrow$ Action to achieve a goal, often augmented by planning or knowledge management systems.

The ReAct Approach

ReAct (Reasoning and Acting) is a specific method for building agents that combines Chain-of-Thought prompting with action execution. The model is prompted to think "step by step" to plan and execute actions. The internal loop operates as follows:

  1. Thought: The LLM reflects on the current state and plans the next step.
  2. Action: The LLM calls a tool using a specific format (e.g., JSON).
  3. Observation: The system executes the tool and appends the result back to the prompt.
  4. Final Answer: Once sufficient information is gathered, the LLM provides the final response.

Core Challenges in Agent Systems

Implementing reliable agents involves overcoming three primary technical hurdles:

  • Tool Selection: Choosing the correct tool from a provided list to advance toward the goal.
  • Argument Formatting: Maintaining rigorous formatting (e.g., JSON) for tool calls to avoid misspellings or incorrect argument values.
  • Information Integration: Efficiently using gathered observations and initial context to inform subsequent reasoning steps.

Implementing Agents with LangChain and Hugging Face

Hugging Face has integrated the ChatHuggingFace wrapper into LangChain, simplifying the creation of agents using open-source models. This allows developers to bind a HuggingFaceEndpoint model to a ReAct-style prompt and a set of tools (such as SerpAPI for search or LLM-math for calculations).

By using the AgentExecutor in LangChain, the model can process complex queries—such as finding a world record holder's age and performing a mathematical operation on that age—by iteratively searching for data and using a calculator tool.

Performance Benchmark: Open-Source vs. Proprietary Models

To evaluate general-purpose reasoning, Hugging Face tested several models using a dataset combining samples from HotpotQA (internet search), GSM8K (grade-school math), and GAIA (general AI assistants).

Evaluated Models

  • Open-Source: Llama2-70b-chat, Mixtral-8x7B-Instruct-v0.1, OpenHermes-2.5-Mistral-7B, Zephyr-7b-beta, and SOLAR-10.7B-Instruct-v1.0.
  • Proprietary: GPT-3.5 and GPT-4 (using OpenAI-specific function-calling templates).

Key Findings

  • Mixtral-8x7B Superiority: Mixtral-8x7B performed exceptionally well, beating GPT-3.5 on the benchmark. This is notable because Mixtral was not specifically fine-tuned for agent workflows, whereas GPT-3.5 was.
  • Model Variance: Performance varied significantly among open-source models; while Mixtral excelled, Llama2-70b performed surprisingly poorly in this specific agentic context.
  • Impact of Tool Use: Agent workflows significantly boost model performance. For example, on a small sample of GSM8K questions, Mixtral-8x7B's zero-shot performance reached 73% when given a calculator, compared to 57.6% reported on the LLM Leaderboard with 5-shot prompting.

Potential for Improvement

Approximately 10% of Mixtral's failures on the GAIA benchmark were due to incorrectly formatted tool arguments. Hugging Face suggests that targeted fine-tuning for function calling and task planning could further increase the performance of open-source models, potentially challenging GPT-4.

Sources