dexmal/opendm

What it solves

OpenDM provides a framework and pre-trained models for open-world robot control. It addresses the challenge of enabling robots to follow open-ended instructions, handle long-horizon tasks, manage dynamic disturbances, and operate across different robot embodiments (multi-embodiment control).

How it works

The project centers around DM0.5, a Vision-Language-Action (VLA) model. It processes visual inputs (e.g., from a head camera and wrist cameras) and language instructions to predict robot actions. The system supports supervised fine-tuning (SFT) and LoRA training to adapt the base model to specific robot hardware and tasks. It also includes a "fast backend" using TensorRT, Triton, and PyTorch FlexAttention for accelerated inference.

Who it’s for

Researchers and developers working in embodied AI and robotics who need to train, fine-tune, and deploy VLA models for real-world or simulated robot control.

Highlights

• Multi-Embodiment Support: Designed for various robot platforms, including AgileX COBOT Magic and DOS-W1.
• VLA Architecture: Integrates vision, language, and action prediction into a single model.
• Flexible Training: Supports full SFT and LoRA fine-tuning workflows.
• High-Performance Inference: Offers a specialized fast backend for reduced latency in robot control.
• Benchmark-Ready: Includes pre-trained checkpoints and evaluation workflows for benchmarks like LIBERO, RoboTwin2.0, and VLA-Arena.