Thinklab-SJTU/Bench2Drive

What it solves

Bench2Drive provides a standardized dataset and closed-loop evaluation benchmark for autonomous driving (AD) systems. It addresses the need for more reliable performance assessment by moving beyond open-loop planning (which the authors argue is not a meaningful indicator) to a closed-loop environment where agents must react to dynamic scenarios.

How it works

The project utilizes the CARLA simulator to create a closed-loop testing environment. It provides several dataset splits (Mini, Base, Full, and a strictly uniform 0.0.4 set) containing diverse driving scenarios. The benchmark evaluates models based on a variety of metrics, including a general Driving Score, Success Rate, Driving Efficiency, and Driving Smoothness. It also specifically tests "multi-ability" performance across categories like overtaking, merging, emergency braking, giving way, and following traffic signs.

Who it’s for

It is designed for researchers and developers working on autonomous driving agents, specifically those building end-to-end driving models or vision-language models for autonomous driving (VLM4AD).

Highlights

• Diverse Dataset Splits: Offers multiple training sets, including a strictly uniform distribution of 44 scenarios across 1,100 routes.
• Closed-Loop Evaluation: Focuses on real-time interaction within the CARLA simulator rather than static open-loop predictions.
• Comprehensive Metrics: Tracks not just success and score, but also efficiency, comfort/smoothness, and specific driving abilities.
• Multi-Modality Support: Includes support for 3D occupancy (3D occ) and depth data.