camel-ai/crab

🦀️ CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agents. https://crab.camel-ai.org/

What it solves

CRAB provides a Python-centric framework for building benchmark environments for multimodal embodied language model agents. It simplifies the process of creating environments where agents can be tested across different deployment options, such as virtual machines, Docker containers, or physical machines, using a unified interface.

How it works

The framework allows developers to define environment actions using a simple @action decorator on Python functions. These actions are then integrated to form a complete environment. It also includes a benchmarking suite that lets users define tasks and evaluators in a Python-native way, featuring a novel graph evaluator for fine-grained performance metrics.

Who it’s for

It is designed for researchers and developers building and evaluating multimodal embodied AI agents that need to interact with diverse, cross-platform environments.

Highlights

  • Cross-platform support: Supports in-memory, Docker-hosted, VM, or distributed physical machines.
  • Unified interface: Agents can access multiple environments simultaneously through a single API.
  • Easy configuration: New actions are added via Python decorators.
  • Fine-grained evaluation: Includes a graph evaluator method for detailed metrics.

Related

  • Project
    evalstate/fast-agent

    A flexible framework for building multimodal AI agents and workflows with deep Model Context Protocol (MCP) integration and advanced orchestration patterns.

  • Project
    microsoft/agent-framework

    A multi-language framework for building production-grade AI agents and multi-agent workflows in .NET and Python, featuring graph-based orchestration and built-in observability.

  • Project
    harbor-framework/harbor

    Harbor is a Python framework for running and scaling benchmarks of AI agents and language models. Installable via `pip`/`uv`, it provides a CLI (`harbor run`) that lets you specify a dataset (e.g., Terminal‑Bench‑2.0), an agent (Claude‑Code, OpenHands, etc.), and a model, then executes the benchmark locally or on cloud providers (Daytona, Modal, etc.) with configurable parallelism. It also supports creating custom benchmarks and generating RL roll‑outs.

  • Project
    RobotecAI/rai

    A flexible AI agent framework for developing and deploying Embodied AI features, enabling robots to use generative AI for reasoning, multi-modal interaction, and complex task execution.

  • Project
    ServiceNow/AgentLab

    A framework for developing and evaluating web agents across various benchmarks, featuring large-scale parallel experimentation and trace visualization tools.