SmolVLA: Efficient Vision-Language-Action Model trained on Lerobot Community Data

Hugging Face has released SmolVLA, a compact 450M parameter open-source Vision-Language-Action (VLA) model designed for robotics. SmolVLA enables high-performance robot control on consumer-grade hardware, outperforming larger VLA models and baselines like ACT in both simulated environments (LIBERO, Meta-World) and real-world tasks using affordable hardware such as the SO100 and SO101 arms.

Model Architecture and Design

SmolVLA utilizes a modular architecture that combines a Vision-Language Model (VLM) for perception and a specialized action expert for control.

Vision-Language Model (VLM) Backbone

SmolVLA uses SmolVLM2 as its backbone, which consists of a SigLIP vision encoder and a SmolLM2 language decoder. The model processes multimodal inputs as follows:

  • Images: Extracted via the vision encoder.
  • Language Instructions: Tokenized and fed into the decoder.
  • Sensorimotor States: Projected into a single token via a linear layer to align with the language model's dimensions.

Action Expert: Flow Matching Transformer

The action expert is a compact transformer (~100M parameters) that generates sequences of future robot actions (action chunks). It is trained using a flow matching objective, which predicts a "correction vector" to guide noisy samples back to the ground truth trajectory. This approach allows for direct, non-autoregressive prediction of continuous actions, ensuring real-time control and high precision without the latency associated with autoregressive decoding.

Efficiency Optimizations

To maintain low latency and a small footprint, SmolVLA implements three specific architectural choices:

  • Visual Token Reduction: Images (e.g., 512×512) are compressed into 64 tokens instead of 1024 using PixelShuffle, significantly reducing inference time.
  • Layer Skipping: The action expert attends to VLM features only up to half of the total layers. This halves the compute cost for both the VLM and action expert with minimal performance loss.
  • Interleaved Attention: The action expert alternates between cross-attention (conditioning actions on perception and instructions) and self-attention (ensuring temporal smoothness and reducing jitter).

Asynchronous Inference Stack

SmolVLA introduces an asynchronous inference stack to decouple action execution from chunk prediction, eliminating execution lags.

In a synchronous setup, a robot pauses to compute the next action chunk after executing the current one. In the asynchronous setup, the robot sends the latest observation to a Policy Server (potentially GPU-hosted) while still executing the current chunk.

Key mechanisms include:

  • Early Trigger: New observations are sent to the server when the queue length drops below a threshold (e.g., 70%).
  • Decoupled Threads: Inference happens in parallel with the control loop.
  • Chunk Fusion: Overlapping actions from successive chunks are merged to prevent jitter.

This system results in 30% faster task completion (9.7s vs 13.75s) and 2× higher task throughput (19 vs 9 cubes completed in a fixed window) compared to synchronous inference, while maintaining a similar success rate (~78%).

Training on Community Datasets

SmolVLA is pretrained on a curated collection of 487 high-quality datasets from the LeRobot community on the Hugging Face Hub, totaling approximately 10 million frames. This is an order of magnitude smaller than typical benchmark datasets but offers greater diversity in behaviors, camera views, and embodiments.

Data Standardization

To handle the fragmented nature of community data, Hugging Face applied two standardization processes:

  1. Task Annotation Improvement: Using Qwen2.5-VL-3B-Instruct, vague labels (e.g., "Move") were rewritten into concise, action-oriented descriptions under 30 characters starting with a verb (e.g., "Pick up the cube").
  2. Camera View Standardization: Inconsistent camera labels were mapped to a standard scheme: OBS_IMAGE_1 (Top-down), OBS_IMAGE_2 (Wrist-mounted), and OBS_IMAGE_3+ (Additional views).

Impact of Pretraining

Pretraining on community data significantly boosts performance. On the SO100 arm, success rates increased from 51.7% (without pretraining) to 78.3% (after pretraining), representing a +26.6% absolute improvement.

Performance Results

SmolVLA demonstrates strong generalization and efficiency across multiple benchmarks:

  • Simulation: Outperforms larger VLAs and baselines on LIBERO and Meta-World.
  • Real-World (SO100): Matches or exceeds baselines in pick-place, stacking, and sorting tasks.
  • Generalization (SO101): Shows superior ability to generalize to new embodiments compared to the ACT baseline.
  • Hardware Accessibility: The model is small enough to run on CPUs, MacBooks, or a single consumer GPU.

Sources