hustvl/Senna

[IJCV 2026] Senna: Bridging Large Vision-Language Models and End-to-End Autonomous Driving

What it solves

Senna addresses the limitations of end-to-end autonomous driving systems by integrating a Large Vision-Language Model (LVLM) to improve planning safety, robustness, and generalization across different scenarios.

How it works

Senna bridges a vision-language model with an end-to-end driving model. It utilizes a 7B parameter model based on Vicuna-7b-v1.5, processing 6-view input images to generate scene descriptions and planning explanations. The system is trained in three stages: mix pre-training, driving fine-tuning, and planning fine-tuning (supporting both full-parameter and LoRA fine-tuning).

Who it’s for

Researchers and developers working on autonomous driving, embodied intelligence, and the integration of large multimodal models for vehicle planning and perception.

Highlights

  • SOTA Planning Performance: Achieves state-of-the-art results in planning accuracy.
  • Cross-Scenario Generalization: Demonstrates strong ability to transfer and generalize to new driving environments.
  • Multimodal Integration: Combines visual input from multiple camera views with language-based reasoning for driving decisions.
  • Comprehensive Tooling: Includes scripts for QA data generation, training, and visualization of predicted meta-actions.

Related

  • Project
    bytedance/SALMONN

    SALMONN is an open‑source family of multimodal large language models that jointly handle vision, speech, text and actions. The repo provides model checkpoints, inference code, training scripts, and large annotated datasets (including video‑captioning and speech‑quality data). Variants such as video‑SALMONN‑2, ELLSA, and F‑16 enable streaming, full‑duplex perception and generation, making the models suitable for interactive agents, video QA, captioning, and speech‑quality assessment.

  • Project
    xiaomi-research/recogdrive

    ReCogDrive is a research‑grade autonomous‑driving framework that combines a Vision‑Language Model with a diffusion‑based trajectory planner, reinforced by a novel DiffGRPO RL step. It ships with pre‑training QA datasets (12 open‑source driving corpora), multiple checkpoints (2 B‑8 B VLMs, IL and RL variants), and scripts for training/evaluation on NAVSIM, Bench2Drive, DriveLM, etc. State‑of‑the‑art results are reported on several benchmarks, and all code, models, and data are publicly released.

  • Project
    EvolvingLMMs-Lab/lmms-engine

    A unified, high-performance training engine for multimodal models that integrates distributed training, kernel fusion, and advanced optimizers to enable scalable pretraining and fine-tuning.

  • Project
    Thinklab-SJTU/Bench2Drive

    A closed-loop dataset and benchmark for autonomous driving systems using the CARLA simulator to evaluate driving scores, success rates, and specific driving abilities.

  • Project
    erwinmsmith/SOMAS

    SOMAS is a multi-agent reinforcement learning framework designed for safe human-machine collaboration in crisis response, featuring real-time safety guardrails and offline simulation training.