zai-org/SCAIL-2

Official Implementation of SCAIL-2: Unifying Controlled Character Animation with End-to-end In-Context Conditioning

What it solves

SCAIL-2 addresses the limitations of traditional character animation that rely on intermediate pose representations (like skeleton maps), which often struggle with complex motions, non-human driving sources (e.g., animals), and identity generalization. It aims to provide a more flexible, end-to-end system for character animation and character replacement in videos.

How it works

SCAIL-2 uses a Unified Motion Transfer Interface with specific masking channels and a dedicated RoPE design to bypass intermediate pose representations. It was trained on 60K motion pairs synthesized from other models (SCAIL-Preview, Wan-Animate, and MoCha) using a "reserve driving" approach to develop emergent capabilities. The system supports both animation mode (end-to-end or pose-driven) and replacement mode. It also incorporates Bias-Aware DPO (Direct Preference Optimization) to improve fine details like lip and eye synchronization and reduce hand distortion.

Who it’s for

This project is for developers and creators working with AI-driven video generation, character animation, and video editing who need high-fidelity motion transfer and character replacement capabilities.

Highlights

  • End-to-End Animation: Supports character animation without relying on skeleton maps, enabling animal-driving scenarios.
  • Character Replacement: Allows replacing a character in a video with a reference image, including a Relighting LoRA to ensure natural lighting and shadow blending.
  • Multi-Reference Support: Enables zero-shot multi-reference generation to provide additional visual details (e.g., different views or close-ups) of a character.
  • Bias-Aware DPO: Uses a novel mechanism to improve synchronization and detail quality.
  • ComfyUI Integration: Available for use within the ComfyUI ecosystem.

Related

  • Project
    phongdaot/MocapAnything

    An end-to-end motion capture system that converts monocular videos into 3D joint rotations for arbitrary skeletons, enabling movement transfer to any rigged character.

  • Project
    wuwukaka/ComfyUI-WanAnimatePlus

    ComfyUI‑WanAnimatePlus is a ComfyUI custom‑node pack that extends the WanVideo video‑generation pipeline. It adds multi‑reference image injection (prefix_frames), seamless video continuation (transition_video), and full support for Bernini and SCAIL‑2 models via dedicated nodes and easy‑sampler wrappers. Install by cloning into `ComfyUI/custom_nodes`, replace the original WanVideo chain with the WanAnimatePlus equivalents, and connect optional prefix/transition inputs to generate longer, more controllable videos.

  • Project
    perfanalytics/pose2sim

    A markerless 3D kinematics workflow that converts 2D video from low-cost cameras into research-grade 3D joint angles and skeletal analysis using OpenSim.

  • Dispatch
    Reducing Bias and Improving Safety in DALL·E 2

    OpenAI has implemented a system-level technique for DALL·E 2 to increase the diversity of generated people and enhanced safety filters to prevent the creation of deceptive content.

  • Dispatch
    BLIP-2: Zero-Shot Image-to-Text Generation

    BLIP-2 is a visual-language model from Salesforce Research that uses a Querying Transformer (Q-Former) to bridge frozen image encoders and frozen large language models for efficient zero-shot image-to-text generation.