sirioberati/Genjustsu-Open-Source-Workflow

Open-source AI video replacement workflow by TEIN. Combine depth maps, SAM segmentation, face meshes, and audio processing to replace characters and objects with Seedance. Includes a local web UI to prepare inputs, review masks, preview generations, and export with original audio.

What it solves

Genjustsu is a local UI and agent-assisted workflow designed to replace subjects in videos while preserving the original source audio. It streamlines the process of subject replacement by combining depth estimation, segmentation masks, and AI-driven video generation, ensuring that the final output maintains the original soundtrack rather than using generated audio.

How it works

The system uses a multi-stage pipeline to prepare a video for subject replacement:

  1. Audio Processing: It isolates vocals using Demucs and shifts their pitch by +3 semitones using Rubber Band to create a guidance track.
  2. Visual Preparation: It generates colored depth maps using depth-anything-video and creates segmentation masks via SAM 3. The system also offers a "Fast" mode using Google MediaPipe Face Landmarker for facial tracking without depth/SAM calls.
  3. Compositing: The depth subject is composited over the original background, and the pitched vocals are embedded into the video.
  4. Generation: The prepared video and character references are sent to Enhancor Seedance (using Edit or Omni modes) to generate a draft preview.
  5. Finalization: Upon user approval of the draft, the system generates a 1080p version and restores the original, unpitched source audio byte-for-byte.

Who it’s for

This tool is for creators and developers who want to replace characters in existing video scenes with new identities while maintaining high-quality audio and precise visual control through a mask review process.

Highlights

  • Audio Restoration: Automatically restores the entire original source audio to the final export.
  • Agent-Assisted Setup: Includes a detailed runbook for coding agents to handle installation and environment configuration.
  • Multi-Mode Preparation: Supports Advanced (Depth + SAM), Fast (Face Mesh), and Combined (Depth + SAM + Face Mesh) modes.
  • Draft-to-HD Workflow: Implements a mandatory review gate where users must approve a draft before committing to a high-definition 1080p generation.
  • Local Control: Runs as a local FastAPI application with a vanilla JS UI, using hosted models for heavy inference via Replicate and Enhancor.

Related

  • Project
    Francis-Rings/StableAvatar

    StableAvatar is an open‑source diffusion‑based video transformer that generates arbitrarily long, high‑quality avatar videos driven by audio. It introduces a time‑step‑aware audio adapter, audio native guidance, and a sliding‑window fusion strategy to keep lip‑sync and identity stable over unlimited duration. The repo provides pretrained 1.3 B checkpoints, inference scripts (single‑GPU, multi‑GPU, Gradio, ComfyUI), and full training/LoRA code, with detailed data‑preparation instructions.

  • Project
    UnderleveledBuilder/awesome-short-drama-skills

    A curated list of AI agent skills for short drama and manju production, covering scriptwriting, storyboards, video prompts, and more, with test results and installation instructions.

  • Project
    nyanmisaka/ffmpeg-rockchip

    A specialized version of FFmpeg that enables hardware-accelerated video decoding, encoding, and filtering on Rockchip platforms to improve performance and reduce CPU load.

  • Project
    NikoDemon80/ComfyUI-H3-Motion-Context

    Adds custom nodes to ComfyUI that let you chain MiniMax‑H3 video‑plus‑audio generations. The previous clip is passed as a latent (no decode‑encode), preserving exact pixel data and continuing the audio waveform. Includes Save/Load Latent, Trim, Chain, and optional drift‑reduction nodes, enabling seamless multi‑clip videos with consistent motion and sound.