RobotecAI/rai
RAI is a vendor agnostic agentic framework for Physical AI robotics, utilizing ROS 2 tools to perform complex actions, defined scenarios, free interface execution, log summaries, voice interaction and more.
What it solves
RAI provides a flexible framework for developers to integrate generative AI and multi-agent architectures into robotic systems. It bridges the gap between high-level AI reasoning and physical robot execution, enabling robots to handle complex problem-solving, natural human interaction, and multi-modal sensory data.
How it works
The framework consists of a core system for multi-agent coordination and human-robot interaction, supported by a suite of specialized modules:
- Perception & Interaction: Includes speech-to-text (ASR), text-to-speech (TTS), and open-set object detection tools.
- Robot Knowledge: A "whoami" tool that synthesizes robot embodiment information from URDFs and documentation.
- Integration: Connects to simulation environments and integrates with NoMaD for navigation.
- Evaluation: A benchmarking suite to test agents, models, and simulators.
Who it’s for
Robotics engineers and AI researchers building Embodied AI features, specifically those working with ROS2 (Humble or Jazzy) and Python.
Highlights
- Multi-Modal Support: Natively handles diverse data types and simultaneous input/output modalities.
- Agentic Capabilities: Enables robots to reason through missions, such as analyzing the best course of action for obstacles in an orchard.
- Broad Hardware Compatibility: Demonstrated on various platforms including autonomous tractors, Franka Panda arms, and Husarion ROSbot XL.
- Simulation Ready: Includes dedicated packages for connecting AI agents to virtual environments for testing.