xiaomi-research/recogdrive

ReCogDrive – Quick Overview

What it is: ReCogDrive is a research‑grade, end‑to‑end autonomous‑driving system that combines a large Vision‑Language Model (VLM) with a diffusion‑based trajectory planner. The framework is built to give the model a cognitive understanding of driving scenes (via QA‑style pre‑training) and then turn that knowledge into safe, continuous vehicle trajectories.

Key ideas

1. Hierarchical data pipeline – driving data are turned into a three‑stage “generation → refinement → quality‑control” process that mimics how a human driver thinks before acting.
2. VLM + diffusion planner – the VLM learns driving priors from massive QA datasets; those priors are injected into a diffusion planner that directly outputs smooth trajectories, avoiding the “language‑action mismatch” of earlier methods that tried to write actions as text.
3. DiffGRPO – a reinforcement‑learning‑style “Diffusion Group Relative Policy Optimization” step that fine‑tunes the planner for safety and comfort.

Why it matters: Traditional end‑to‑end driving models either output discrete control commands or treat planning as a language‑generation problem, which can produce illegal or jerky motions. ReCogDrive’s diffusion‑based planner produces continuous, feasible paths while still benefiting from the rich world knowledge of a VLM.

---

Getting Started (From the README)

1. Data – Download the NAVSIM datasets following the official NAVSIM instructions.
2. Environment – Follow the docs/Installation.md guide to set up the required Python packages and CUDA dependencies.
3. Training / Evaluation – Use the scripts described in docs/Train_Eval.md to train the VLM stage, the imitation‑learning (IL) stage, and the reinforcement‑learning (RL) stage, then evaluate on NAVSIM, Bench2Drive, DriveLM, etc.

---

Available Checkpoints

Model	Size	Training stages	PDMS (NAVSIM)	Download
ReCogDrive‑Base‑VLM	2 B	Stage 1 (VLM pre‑train)	84.1	HuggingFace
ReCogDrive‑Base‑IL	2 B + 35 M	Stages 1 & 2 (VLM + Imitation)	86.5	HuggingFace
ReCogDrive‑Base‑RL	2 B + 35 M	Stages 1‑3 (VLM + Imitation + RL)	90.8	HuggingFace
ReCogDrive‑Large‑VLM	8 B	Stage 1	86.4	HuggingFace
ReCogDrive‑Large‑IL	8 B + 35 M	Stages 1 & 2	86.5	HuggingFace
ReCogDrive‑Large‑RL	8 B + 35 M	Stages 1‑3	90.4	HuggingFace

The tables also list benchmark results on Bench2Drive (CARLA) and DriveLM/DriveBench, where ReCogDrive leads in success rate, safety, and overall scores.

---

Driving Pre‑training Datasets

ReCogDrive is pre‑trained on 12 open‑source driving QA datasets (e.g., NAVSIM‑Traj, DriveLM, NuInstruct, Nuscenes‑QA, OmniDrive, Senna, LingoQA, etc.).

• The authors re‑annotated many of these with the large VLM Qwen2.5VL‑72B, filtered for quality, and released the resulting JSONL files on HuggingFace.
• For a lightweight experiment you can pick a single dataset (e.g., NAVSIM‑Traj + NAVSIM‑ReCogDrive) and still obtain competitive performance.

---

Notable Results

• NAVSIM – PDMS up to 90.8 (Base‑RL) and 90.4 (Large‑RL).
• Bench2Drive (CARLA) – Success 45.45 % (best among listed methods), high safety (DS 71.36) and strong multi‑ability scores (e.g., 69.09 % on overtaking).
• DriveLM / DriveBench – GPT‑Score 67.30, Lingo‑Judge 67.20, average DriveBench 56.71, outperforming prior baselines.

---

Where to Find More

• Paper – arXiv: 2506.08052 (ICLR 2026).
• Project page – https://xiaomi-research.github.io/recogdrive/ (contains visualizations, detailed methodology, and download links).
• Code & models – https://github.com/xiaomi-research/recogdrive (the repository you’re reading).
• Related work – The authors point to their follow‑up “UniDriveVLA” project for a unified vision‑language‑action model.

---

Who Might Use This?

• Researchers exploring VLM‑based planning, diffusion‑based trajectory generation, or reinforcement learning for autonomous driving.
• Engineers at automotive or robotics firms who need a strong baseline for end‑to‑end driving that already integrates large‑scale language‑grounded knowledge.
• Students looking for a complete pipeline (data, pre‑training, IL, RL, evaluation) to study the interaction between perception, cognition, and control.

---

Citation

@inproceedings{li2026recogdrive,
  author = {Li, Yongkang and Xiong, Kaixin and Guo, Xiangyu and Li, Fang and Yan, Sixu and Xu, Gangwei and Zhou, Lijun and Chen, Long and Sun, Haiyang and Wang, Bing and Ma, Kun and Chen, Guang and Ye, Hangjun and Liu, Wenyu and Wang, Xinggang},
  booktitle = {International Conference on Learning Representations},
  title = {ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Driving},
  year = {2026},
  pages = {157518--157556},
  url = {https://proceedings.iclr.cc/paper_files/paper/2026/file/ff7bf6014f7826da531aa50f4538ee19-Paper-Conference.pdf}
}

---

Bottom line: ReCogDrive is a full‑stack research platform that fuses large‑scale vision‑language understanding with diffusion‑based motion planning, offering state‑of‑the‑art performance on several autonomous‑driving benchmarks and providing all the data, models, and scripts needed to reproduce the results.