jd-opensource/JoyAI-Video-Edit

[Official Repo] JoyAI-Video-Edit: Real-Time Open-Ended Video Editing with Autoregressive Diffusion

What it solves

JoyAI-Video-Edit enables real-time, open-ended video editing for live camera streams or uploaded videos. Unlike traditional offline batch processing, it edits frames causally as they arrive, removing the need to wait for the full video sequence or predefine a video length.

How it works

The system uses an autoregressive diffusion architecture consisting of a multimodal diffusion transformer (MMDiT) with 16 billion parameters, a causal video VAE, and an MLLM-based condition encoder. To achieve high-throughput performance (up to 30 FPS at 720p), it employs several acceleration techniques: aligned autoregressive distribution matching distillation, long-horizon optimization, bounded KV-state inference, and deployment-oriented scheduling.

Who it’s for

This tool is designed for developers and researchers working on interactive streaming generation, real-time video effects, and AI-driven video content creation.

Highlights

  • Real-time streaming: Processes live video feeds without needing the full sequence upfront.
  • Diverse editing capabilities: Supports subject and local edits, background replacement, style transfer, motion changes, and reference-image-guided editing (RV2V).
  • High performance: Capable of reaching 30 FPS at 720 × 1248 resolution.
  • Consumer GPU support: Optimized to run on hardware like the GeForce RTX 5090.

Related

  • Project
    diffusionstudio/editor

    A video editing platform that treats projects as code, allowing AI agents to programmatically edit, transcribe, and render videos while providing a visual interface for manual refinement.

  • Project
    SandAI-org/MAGI-1

    MAGI-1 is an autoregressive video generation model that produces high-fidelity videos chunk-by-chunk to ensure temporal consistency and physical accuracy.

  • Project
    jd-opensource/JoyAI-VL-Interaction

    An 8B-scale vision-language interaction model and system that enables AI to proactively respond to real-time video streams in under a second without waiting for user prompts.

  • Project
    jamichss/Stream-DiffVSR

    Stream‑DiffVSR is a diffusion‑based video‑super‑resolution system designed for low‑latency, online use. It runs a 4‑step distilled denoiser, an auto‑regressive temporal guidance module, and a lightweight temporal decoder to upscale 720p video frames in ~0.33 s on an RTX 4090, far faster than prior diffusion VSR methods. The repo provides conda environments, CLI and Gradio UI for inference, optional TensorRT acceleration, Docker support, and full training scripts using REDS and Vimeo‑90K datasets.

  • Project
    thu-ml/Causal-Forcing

    A framework for high-quality real-time interactive video generation using Causal ODE and Consistency Distillation to optimize autoregressive diffusion distillation.