OM1: a modular AI runtime for deploying multimodal agents across physical robots and simulators

What it solves

OM1 is a modular AI runtime designed to simplify the creation and deployment of multimodal AI agents. It bridges the gap between high-level AI models (LLMs and VLMs) and physical hardware, allowing developers to deploy agents across digital environments, simulators, and physical robots like humanoids and quadrupeds without needing to rebuild the core pipeline for every different form factor.

How it works

OM1 acts as a middleware layer that connects diverse data inputs (web data, camera feeds, LIDAR) to pre-configured AI endpoints (OpenAI, Anthropic, Gemini, etc.) and then translates those outputs into physical actions. It uses a modular architecture written in Go for high performance and low latency on edge devices. The system interfaces with robot hardware abstraction layers (HAL) via protocols like ROS2, Zenoh, and CycloneDDS, allowing it to send elemental movement commands to the robot.

Who it’s for

It is primarily for robotics developers and AI engineers who want to build human-focused robots that are easy to upgrade and reconfigure. It also supports users working with simulators like Gazebo and Isaac Sim for rapid prototyping.

Highlights

• Multimodal Support: Processes visual, audio, and text inputs and responds via speech or physical motion.
• Broad Hardware Compatibility: Supports a wide range of robots (Humanoids, Quadrupeds, TurtleBot 4) and simulators (Gazebo, Isaac Sim).
• Extensible Plugin System: Allows developers to add new hardware connections and data sensors via plugins.
• Observability: Includes a built-in Prometheus and Grafana stack to monitor real-time pipeline metrics like LLM and ASR latencies.
• Performance-focused: Migrated to Go to ensure a smaller memory footprint and lower latency for edge deployment.