SalesforceAIResearch/MCP-Universe

What it solves

MCP-Universe provides a comprehensive ecosystem for building, optimizing, and evaluating AI agents that use the Model Context Protocol (MCP). It addresses the gap in existing benchmarks by testing LLMs in real-world scenarios—such as long-horizon reasoning and interaction with large, unfamiliar tool spaces—rather than simplistic tasks.

How it works

The project is structured into several layers:

• Foundation Layer: Manages MCP servers/clients, LLM integration across multiple providers, and memory systems.
• Agent Layer: Implements various agent types, including Basic, ReAct, and FunctionCall agents.
• Orchestration Layer: Handles workflows (chains, routers) and the benchmark evaluation engine.
• Application Layer: Provides a Gradio dashboard, Web API, and Python library for interaction.

It includes specialized components like MCP+, which reduces token costs by filtering verbose tool outputs, and Wide & Deep (W&D) research agents that use parallel tool calling to improve research efficiency.

Who it’s for

• AI Agent Developers: Those building production-ready agents that need to interact with external tools via MCP.
• LLM Researchers: People needing to benchmark agent performance on real-world tasks across domains like financial analysis, 3D design (Blender), and repository management.

Highlights

• Real-world Benchmarking: Evaluates agents across diverse domains including web search, location navigation, and browser automation.
• MCP+ Context Management: A drop-in replacement for MCP clients that can reduce LLM token costs by 50-75%.
• Wide & Deep Research: Parallel tool calling capabilities that improve accuracy and reduce wall-clock time for deep research tasks.
• Custom Benchmark Creation: Support for defining custom tasks via JSON with specific evaluators for ground-truth verification.