jd-opensource/JoyAI-Video-Edit

[Official Repo] JoyAI-Video-Edit: Real-Time Open-Ended Video Editing with Autoregressive Diffusion

What it solves

JoyAI-Video-Edit enables real-time, instruction-guided video editing for open-ended video streams. Unlike traditional offline batch processing, it allows users to edit live camera feeds or uploaded videos as frames arrive, removing the need to wait for the entire video sequence or define a fixed video length upfront.

How it works

The system utilizes an autoregressive diffusion architecture consisting of an MLLM-based condition encoder, a causal video VAE, and a 16B-parameter multimodal diffusion transformer (MMDiT). To achieve high-throughput performance (up to 30 FPS at 720p), it employs several optimization techniques: aligned autoregressive distribution matching distillation, long-horizon optimization, bounded KV-state inference, and deployment-oriented scheduling.

Who it’s for

This tool is designed for developers and researchers interested in interactive streaming video generation, real-time visual effects, and AI-driven video editing applications.

Highlights

  • Real-time streaming: Processes frames causally as they arrive, supporting live webcam feeds.
  • Diverse control: Supports subject and local edits, background changes, style transfer, motion changes, and reference-image-guided editing (RV2V).
  • High performance: Capable of reaching 30 FPS at 720 × 1248 resolution.
  • Consumer GPU support: Optimized to run on single consumer GPUs like the RTX 5090.

Related

  • Project
    diffusionstudio/editor

    A professional video editor designed for AI agents that treats video compositions as code, allowing agents to and humans to collaboratively edit videos via a bidirectional JSX-based workflow.

  • Project
    jd-opensource/JoyAI-VL-Interaction

    An 8B-scale vision-language interaction model and system that enables AI to proactively respond to real-time video streams in under a second without waiting for user prompts.

  • Project
    SandAI-org/MAGI-1

    MAGI-1 is an autoregressive video generation model that produces high-fidelity videos chunk-by-chunk to ensure temporal consistency and physical accuracy.

  • Project
    GVCLab/PersonaLive

    A real-time, streamable diffusion framework for generating infinite-length expressive portrait animations from a single reference image and a driving video.

  • Project
    EternalEvan/Astra

    Astra is an interactive world model that uses an autoregressive diffusion transformer to generate realistic, action-conditioned long-horizon video predictions.