bytedance/Lance

What it solves

Lance addresses the challenge of creating a single, efficient model capable of handling multiple multimodal tasks. Instead of using separate models for different needs, it provides a unified framework for image and video understanding, generation, and editing.

How it works

Lance is a native unified multimodal model with 3 billion active parameters. It was trained from scratch using a staged multi-task recipe on a budget of up to 128 A100 GPUs. It supports a wide range of tasks including text-to-image, text-to-video, image-to-video, image/video editing, and visual understanding (captioning and question answering).

Who it’s for

This project is primarily a research artifact intended for researchers and developers studying unified multimodal modeling, specifically those looking to achieve high performance in image and video tasks within a relatively small model size and limited compute budget.

Highlights

• Unified Capabilities: Supports image/video generation, editing, and understanding in one model.
• Efficient Scale: Operates with only 3B active parameters while remaining competitive on benchmarks.
• Versatile Task Support: Handles text-to-video (T2V), image-to-video (I2V), text-to-image (T2I), and multi-turn consistency editing.
• Research-Ready: Includes fine-tuning code and is supported by vLLM-Omni.