bytedance/Bernini

What it solves

Bernini addresses the challenge of high-quality video generation and editing, specifically improving instruction following and semantic consistency. It aims to provide a unified framework that can handle complex editing instructions by planning semantic changes before the actual rendering process.

How it works

The framework consists of two primary components:

1. Semantic Planner: An MLLM-based (Multimodal Large Language Model) planner that decomposes complex instructions and plans the necessary semantic changes.
2. Renderer: A DiT-based (Diffusion Transformer) renderer that executes the visual generation.

Depending on the user's needs, the project provides two model families: the full Bernini pipeline (Planner + Renderer) for stronger instruction following, and Bernini-R (Renderer-only) for simpler setup and strong rendering consistency.

Who it’s for

Researchers and developers working in video diffusion and AI-driven video editing who require precise control over visual changes and high-quality output.

Highlights

• Unified Framework: Supports multiple tasks including text-to-video (t2v), image-to-video (i2v), video-to-video (v2v), and others.
• High Performance: Reaches the first tier among leading closed-source commercial models in video editing benchmarks.
• Flexible Model Options: Offers both a full planning pipeline and a standalone renderer (available in 1.3B and 14B variants).
• Prompt Enhancement: Includes an optional prompt enhancer via OpenAI-compatible endpoints to improve generation quality.