shang-zhu/violin

Open-source Video Translation Skill

What it solves

Violin is an open-source tool for translating videos into different languages while maintaining a natural feel. It automates the complex process of transcribing original speech, translating the text, generating a native-sounding voice-over, and syncing the new audio back into the video file.

How it works

Violin uses a pluggable pipeline of AI models to process video:

  1. Audio Extraction: Uses ffmpeg to pull audio from the video.
  2. Transcription: Employs Whisper Large v3 to create word-level timestamps and sentence segments.
  3. Translation: An LLM (DeepSeek V4 Pro by default) translates the segments based on a selected style profile.
  4. Speech Synthesis: A TTS engine (Cartesia Sonic 3 by default) generates the dubbed audio.
  5. Remuxing: ffmpeg aligns the video speed to the dubbed audio and encodes the final output MP4 with optional SRT subtitles.

Who it’s for

Content creators and developers who need to localize video content for global audiences across 33 target languages without manual dubbing services.

Highlights

  • Pluggable Stack: Easily switch between providers like Together AI, OpenAI, and ElevenLabs via a single YAML configuration.
  • Style Profiles: Six experimental profiles (standard, kids, academic, casual, storyteller, news) that adjust both translation tone and TTS delivery.
  • In-video Q&A: Allows users to ask questions about specific moments in the dubbed video using sampled frames and subtitles.
  • Natural-language Voice Picker: Uses an LLM to select the best voice from the catalog based on a user's description.
  • Flexible Deployment: Available as a CLI tool, a FastAPI web app, or a Claude Code skill.

Related

  • Project
    open-gigaai/giga-models

    GigaModels is an open‑source Python toolbox that wraps dozens of pre‑trained vision, diffusion and multimodal models (e.g., Grounding DINO, Depth Anything, GigaBrain‑0, Pi0, Cosmos‑Predict2.5) behind a unified `load_pipeline` API. It supports both inference and training, offers ready‑to‑run scripts, and is installable via conda + pip.

  • Project
    InternRobotics/PointLLM

    PointLLM is a multi-modal large language model that enables LLMs to understand colored 3D point clouds for tasks like object classification and captioning.

  • Project
    microsoft/Biodiversity

    A collection of open-source AI tools for biodiversity monitoring, providing models for animal detection in camera-trap imagery, bioacoustics, aerial views, and sonar.

  • Project
    Stonesjtu/pytorch_memlab

    pytorch_memlab is a Python library that profiles CUDA memory usage line‑by‑line, lists live tensors, and can temporarily move GPU data to CPU. It integrates with IPython/Jupyter via `%mlrun` magics and helps debug out‑of‑memory errors in PyTorch models.