ginwind/VLA-JEPA

What it solves

VLA-JEPA improves the performance of Vision-Language-Action (VLA) models by integrating a latent world model. This helps robotic agents better understand and predict the outcomes of their actions in a latent space, enhancing their ability to perform complex tasks across different environments.

How it works

The project combines a Vision-Language Model (VLM) with a latent world model encoder. Specifically, it utilizes the Qwen3-VL-2B model and the V-JEPA2 encoder. The system is trained on a mixture of robot-specific datasets (such as Droid, LIBERO, and BridgeV2) and human video datasets (such as ssv2) to learn generalizable action policies. It supports both pre-training and post-training phases and can be adapted to custom robot datasets using the LeRobot v2.1 format.

Who it’s for

This project is designed for robotics researchers and AI engineers working on embodied intelligence and VLA models who want to improve robot control policies using world models.

Highlights

• Hybrid Training: Supports training on both robot-specific action data and general human videos.
• Broad Benchmark Support: Includes evaluation scripts for LIBERO, LIBERO-Plus, and SimplerEnv benchmarks.
• Customizable: Allows users to integrate their own robot datasets via the LeRobot v2.1 format.
• Pre-trained Components: Leverages state-of-the-art components like Qwen3-VL and V-JEPA2.