xiaomi-research/recogdrive

[ICLR 2026] ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Driving

What it solves

ReCogDrive addresses the "long-tail" problem in end-to-end autonomous driving. It solves the mismatch between high-level cognitive reasoning (typically handled by Vision-Language Models) and the physical execution of driving actions, which often suffer from format errors, infeasible trajectories, and slow inference when treated as simple language tasks.

How it works

ReCogDrive integrates an autoregressive Vision-Language Model (VLM) with a diffusion planner to unify understanding and planning. It follows a three-stage process:

  1. Cognitive Foundation: A hierarchical data pipeline (generation, refinement, and quality control) instills human-like driving cognition into the VLM.
  2. Action Generation: The VLM's learned driving priors are injected into a diffusion planner to generate continuous, stable, and physically feasible trajectories.
  3. Safety Reinforcement: A Diffusion Group Relative Policy Optimization (DiffGRPO) stage reinforces the planner to improve safety and comfort, reducing collisions.

Who it’s for

Researchers and engineers working on end-to-end autonomous driving, Vision-Language-Action (VLA) models, and trajectory planning in robotics.

Highlights

  • Hybrid Architecture: Combines the reasoning power of VLMs with the precision of diffusion planners.
  • Extensive Pretraining: Pretrained on 12 open-source driving datasets and a custom automated annotation pipeline for NavSim.
  • SOTA Performance: Achieves state-of-the-art results on NAVSIM and Bench2Drive benchmarks.
  • Reinforced Safety: Uses DiffGRPO to specifically optimize for safer and more comfortable driving maneuvers.

Related

  • Project
    NVlabs/alpamayo

    Alpamayo 1 is a Vision-Language-Action model for autonomous driving that combines reasoning traces with trajectory prediction to handle complex, rare driving scenarios.

  • Project
    baidu-baige/LoongForge

    A high-performance training framework for LLMs, VLMs, diffusion, and embodied models that optimizes parallelism and kernel efficiency to reduce training costs.

  • Project
    OpenDriveLab/SimScale

    A scalable simulation pipeline and co-training strategy for end-to-end autonomous driving that synthesizes high-fidelity reactive scenarios to improve model robustness.

  • Project
    hustvl/Senna

    Senna is an autonomous driving system that integrates a Large Vision-Language Model with an end-to-end model to enhance planning safety, robustness, and generalization.

  • Dispatch
    Hugging Face Vision Language Models 2025 Update

    Hugging Face provides a comprehensive overview of the 2024-2025 evolution of Vision Language Models (VLMs), highlighting trends in any-to-any architectures, reasoning models, and the rise of Vision-Language-Action (VLA) models for robotics.