OpenMind/OM1

OpenMind OM1 – A Modular AI Runtime for Robots

What it is

• OM1 is a runtime (a piece of software you run) that lets you build multimodal AI agents – programs that can see, hear, speak and act – and attach them to robots or virtual environments.
• The core is written in Go for low‑latency, low‑memory operation on edge devices (Jetson, Raspberry Pi, etc.). A legacy Python version exists but is deprecated.

Key capabilities

Capability	How it works
Modular architecture	Plug‑in system for new sensors, actions and hardware adapters.
Input handling	Webcam, microphone, LIDAR, web feeds, social‑media streams, etc.
Hardware adapters	Plugins communicate with ROS 2, Zenoh, CycloneDDS, USB/serial or websockets.
LLM/VLM integration	Pre‑configured endpoints for OpenAI, Anthropic, Gemini, Ollama, DeepSeek, Meta, NearAI and visual language models.
Observability	Bundled Prometheus + Grafana stack monitors pipeline latencies (LLM, ASR, etc.).
Robotics features	Through the optional BrainPack you get SLAM, autonomous navigation, auto‑charging and face‑anonymisation.

Typical workflow

1. Install dependencies – portaudio & ffmpeg (audio/video) and the Go toolchain.
2. Get the binary (or build from source) and set your OpenMind API key.
3. Run a config – e.g. config/conversation.json5 starts a “conversation” agent that uses the webcam/mic, sends data to an LLM, and replies via TTS.
4. Watch the logs – the terminal shows state updates; optional Grafana dashboard visualises latency metrics.
5. Extend – add new inputs (sensor plugins) or actions (robotic motion, custom commands) by dropping Go files under plugins/ and referencing them in a JSON5 config.

Supported robots & simulators

• Physical: Unitree Go2, G1, Deep Robotics M20 Pro, LimX Tron, plus any robot exposing a high‑level SDK (ROS 2, Zenoh, etc.).
• Simulators: Gazebo (for rapid prototyping) and NVIDIA Isaac Sim (GPU‑accelerated physics).

Why Go?

• Faster inference pipelines, efficient concurrency, single‑binary deployment, and a small memory footprint suitable for edge compute.

Getting started quickly

# macOS example
brew install portaudio ffmpeg
curl -L -o om1.tar.gz <release‑url>
tar xf om1.tar.gz
chmod +x om1
export OM_API_KEY=YOUR_KEY
./om1 -config ./config/conversation.json5

You should see the agent start, process webcam/audio, call the chosen LLM, and speak the response.

Extensibility

• Action plugins live under plugins/actions/. Example: unitree/go2/autonomy/move.go translates a high‑level move command into ROS 2 cmd_vel messages via Zenoh.
• Input plugins can be added for new sensors (e.g., ZED depth camera) by implementing the expected interface and registering it in the config.

Documentation & community

• Full docs: https://docs.openmind.com/
• Technical paper: arXiv 2412.18588
• X (formerly Twitter) account: @openmind_agi

License

• MIT – you can freely use, modify and ship OM1‑based agents.

---

Bottom line: OM1 is a genuine, production‑grade framework for building AI‑driven robots and simulators, focusing on multimodal perception, LLM/VLM reasoning, and real‑time actuation, all packaged as a performant Go runtime.