octotools/octotools

What it solves

OctoTools is designed to handle complex reasoning tasks across diverse domains (such as math, medicine, and general knowledge) without requiring additional training. It addresses the limitation of LLMs in performing multi-step reasoning and precise tool usage by providing a structured framework for integrating and optimizing external tools.

How it works

The framework consists of three primary components:

• Tool Cards: Standardized metadata containers that encapsulate tool functionality, allowing new tools to be added or updated without changing the core agent logic.
• Planner: Manages both high-level global objectives and low-level step-by-step action refinement.
• Executor: Converts planned actions into executable commands, runs the tools, and saves structured results into the context to summarize a final answer.

Additionally, it includes a task-specific toolset optimization algorithm to identify the most effective subset of tools for a given task.

Who it’s for

It is intended for developers and researchers building AI agents that need to perform complex, multi-modal reasoning (text and image) and require a flexible, extensible way to integrate various external APIs and local models.

Highlights

• Training-Free: Integrates new tools without needing to retrain the model or refine the framework.
• Broad Model Support: Compatible with a wide array of LLM engines including OpenAI, Claude, Gemini, DeepSeek, Grok, vLLM, and Ollama.
• Extensible Architecture: Modular tool card design allows for easy addition of custom tools.
• Proven Performance: Outperforms baselines like AutoGen and LangChain on diverse benchmarks including MathVista and GAIA-Text.