hustvl/Senna

[IJCV 2026] Senna: Bridging Large Vision-Language Models and End-to-End Autonomous Driving

What it solves

Senna addresses the limitations of end-to-end autonomous driving systems by integrating a Large Vision-Language Model (LVLM) to improve planning safety, robustness, and generalization across different scenarios.

How it works

Senna bridges a vision-language model with an end-to-end driving model. It utilizes a 7B parameter model based on Vicuna-7b-v1.5, processing 6-view input images to generate scene descriptions and planning explanations. The system is trained in three stages: mix pre-training, driving fine-tuning, and planning fine-tuning (supporting both full-parameter and LoRA fine-tuning).

Who it’s for

Researchers and developers working on autonomous driving, embodied intelligence, and the integration of large multimodal models for vehicle planning and perception.

Highlights

  • SOTA Planning Performance: Achieves state-of-the-art results in planning accuracy.
  • Cross-Scenario Generalization: Demonstrates strong ability to transfer and generalize to new driving environments.
  • Multimodal Integration: Combines visual input from multiple camera views with language-based reasoning for driving decisions.
  • Comprehensive Tooling: Includes scripts for QA data generation, training, and visualization of predicted meta-actions.

Related

  • Project
    bytedance/SALMONN

    A suite of advanced multi-modal LLMs that provide large language models with generic hearing abilities and integrated audio-visual perception.

  • Project
    xiaomi-research/recogdrive

    A reinforced cognitive framework for end-to-end autonomous driving that combines a Vision-Language Model with a diffusion planner to generate safe, continuous trajectories.

  • Project
    EvolvingLMMs-Lab/lmms-engine

    A unified, high-performance training engine for multimodal models that integrates distributed training, kernel fusion, and advanced optimizers to enable scalable pretraining and fine-tuning.

  • Project
    Thinklab-SJTU/Bench2Drive

    A closed-loop dataset and benchmark for autonomous driving systems using the CARLA simulator to evaluate driving scores, success rates, and specific driving abilities.

  • Project
    erwinmsmith/SOMAS

    A multi-agent framework for human-machine collaborative crisis response that combines vision-language models and reinforcement learning to ensure safety and reliability.