dexmal/opendm

What it solves

OpenDM provides a framework and model weights for DM0.5, a Vision-Language-Action (VLA) model designed for open-world robot control. It addresses the challenges of executing open-ended instructions, managing long-horizon tasks, handling dynamic disturbances, and controlling robots across different embodiments.

How it works

The project implements a native embodied modeling approach. It provides a suite of tools for training, fine-tuning (including LoRA SFT), and deploying the VLA model. The system supports multiple image inputs (e.g., from head and wrist cameras) and outputs action chunks to control robot joints and grippers. For high-performance deployment, it includes a "fast backend" that utilizes TensorRT, Triton, and PyTorch FlexAttention to accelerate inference.

Who it’s for

It is designed for researchers and developers working on embodied AI and robotics who need to train, evaluate, and deploy generalist robot control models.

Highlights

• High Benchmark Performance: Topped leaderboards for RoboChallenge Table30 V2, RoboColiseum, and RoboDojo-Sim.
• Multi-Embodiment Support: Capable of controlling various robot platforms, including AgileX COBOT Magic and DOS-W1.
• Detailed Training Pipeline: Includes comprehensive guides for supervised fine-tuning (SFT) and LoRA, along with a dataset registration system for custom robot data.
• Accelerated Inference: Offers a specialized fast-inference backend for reduced latency in real-world applications.