showlab/Kiwi-Edit

Kiwi‑Edit – Instruction‑ and Reference‑Guided Video Editing

What it is – Kiwi‑Edit is an open‑source framework that lets you edit whole videos by giving natural‑language instructions, optionally together with a reference image. It combines a multi‑modal large language model (MLLM) encoder with a video Diffusion Transformer (DiT) to understand the prompt and synthesize the edited frames.

Key capabilities

• Instruction‑only editing – e.g. “Remove the monkey.” or “Apply the dynamic aesthetic of abstract art.”
• Reference‑guided editing – supply an image that defines a new object, style, or background and ask the model to insert or replace it in the video.
• Supports a range of operations: style transfer, object replace/add/remove, background replacement, and fine‑grained local edits.

How to get started

1. Set up the environment (Python 3.10, CUDA 12.8). The repo provides two ready‑made conda scripts – install_full_env.sh for the full training stack (DeepSpeed, FlashAttention) and install_diffusers_env.sh for lightweight inference with the 🤗 Diffusers library.
2. Download the pretrained video‑DiT (Wan‑AI/Wan2.2‑TI2V‑5B) via hf download into models/Wan‑AI/.
3. Run a quick demo:

   python demo.py \
     --ckpt_path path/to/ckpt \
     --video_path ./demo_data/video/source/example.mp4 \
     --prompt "Remove the monkey." \
     --save_path ./output/demo.mp4
   

   For Diffusers users replace demo.py with diffusers_demo.py and point --model_path to one of the three published model cards (instruction‑only, reference‑only, or combined).

Training

• Data are stored as CSV files describing source video, target video, optional reference image, and the textual prompt. Demo CSVs are included for image‑only, video‑only, and reference‑video stages.
• Training scripts live in scripts/. They orchestrate a three‑stage curriculum (image → image+video → image+video+reference) using the Qwen2.5‑VL‑3B instruction model together with the Wan2.2‑TI2V‑5B video backbone.
• The repo lists hyper‑parameters (resolution, frame count, learning rate, steps) and provides links to the resulting checkpoints on Hugging Face.

Evaluation

• Benchmarks: OpenVE‑Bench (general video editing) and RefVIE‑Bench (reference‑guided editing). Download links are provided; the expected directory layout is documented.
• Inference on a benchmark is a single command (test_benchmark.py) followed by the appropriate evaluation script (eval_openve_gemini.py or eval_refvie_gemini.py).

Resources

• Project page & paper: https://showlab.github.io/Kiwi-Edit, https://arxiv.org/abs/2603.02175
• Model hub (🤗 Hugging Face): three Diffusers checkpoints for the different training regimes.
• Demo Space: https://huggingface.co/spaces/linyq/KiwiEdit
• Acknowledged datasets and tools: Ditto‑1M, OpenVE‑3M, ReCo, GPT‑Image‑Edit‑1.5M, EditScore, Qwen2.5‑VL, Wan‑Video, etc.

Who might use it – Researchers exploring video diffusion models, developers building AI‑powered video editing apps, and creators who want a programmable way to modify video content without manual frame‑by‑frame work.