ASR/STT subtitle generator. Uses Qwen3-ASR, local LLM, Whisper, TEN-VAD. Noise-robust for JAV

WhisperJAV – Japanese‑adult‑video subtitle generator

What it is – WhisperJAV is a desktop (and CLI) application that turns the audio track of a Japanese Adult Video (JAV) into timed subtitle files (.srt/.vtt). It runs locally, so no media is uploaded to the cloud. The tool builds a custom pipeline around OpenAI Whisper‑family models (and newer Japanese‑tuned ASR models) to cope with the very noisy, long‑form audio typical of JAV.

Core ideas & why a special tool is needed

Noisy, low‑SNR audio – breathing, moans, and background music often look like Japanese syllables to a vanilla Whisper model.
Long‑form drift – feature‑length videos cause Whisper’s attention to “hallucinate” repeated or invented text.
Pre‑processing paradox – aggressive denoising can strip the high‑frequency detail needed for accurate phoneme discrimination.

WhisperJAV tackles these three failure points with three engineering blocks:

Scene‑based segmentation – cuts the video at natural acoustic boundaries so each chunk is acoustically homogeneous.
VAD clamping – a Voice‑Activity‑Detector tells the ASR exactly where speech occurs, preventing the model from listening to silence or non‑speech sounds.
Defensive decoding & Japanese‑aware post‑processing – confidence thresholds, hallucination filters, and language‑specific clean‑ups (particle handling, dialect patterns, removal of pure‑moan lines, timing fixes).

How the pipeline works (high‑level flow)

flowchart LR
    A[Audio extraction] --> B[Scene detection]
    B --> C[Speech enhancement (optional)]
    C --> D[Speech segmentation (VAD)]
    D --> E[ASR model]
    E --> F[Japanese‑specific post‑processing]
    F --> G[Subtitle file (.srt/.vtt)]

Audio extraction – FFmpeg pulls the audio stream from any video format.
Scene detection – either semantic clustering, energy‑based auditok, or neural Silero splits the file into scenes.
Speech enhancement – optional neural or classic denoisers (e.g., clearvoice, zipenhancer). By default it is off, because over‑cleaning hurts Whisper.
VAD – determines the exact speech intervals; these intervals become the timestamps for the final subtitles.
ASR – any of the supported recognizers (OpenAI Whisper, Faster‑Whisper, Qwen‑3‑ASR, anime‑whisper, HuggingFace models, etc.) transcribe the speech.
Post‑processing – Japanese‑specific cleanup: regroup sentences around particles, drop pure‑moan lines, remove repetitions, and repair absurdly long timings.

Main features

Feature	What it gives you
GUI & CLI	One‑click desktop app (`whisperjav-gui`) or simple command line (`whisperjav video.mp4`).
Multiple processing modes	`balanced` (default), `fidelity`, `fast`, `faster`, `qwen`, `anime‑whisper`, `transformers`, `crispasr`. Each mode selects a different ASR engine and preset preprocessing aggressiveness.
Sensitivity presets	`conservative`, `balanced`, `aggressive` – tune how eager the VAD is to label quiet speech.
Two‑pass ensemble	Run two completely different pipelines on the same file and merge the results (e.g., anime‑whisper + Qwen3‑ASR) for higher recall.
Mix‑and‑match	Every stage (scene detector, enhancer, VAD, ASR) is swappable; you can build custom pipelines without coding.
AI translation	After transcription, translate subtitles in‑place using local LLMs (Ollama) or cloud APIs (Gemini, Claude, etc.).
Local‑only operation	All processing happens on your machine; no media leaves your computer.
Cross‑platform installers	Stand‑alone Windows `.exe`, plus source‑install scripts for macOS (Apple Silicon) and Linux.
Automatic hardware detection	Detects NVIDIA GPU and installs the matching CUDA build; falls back to CPU‑only if needed.

Typical use‑cases

Personal archiving – generate accurate subtitles for a personal collection of JAV without relying on third‑party services.
Research / linguistic analysis – obtain time‑aligned Japanese dialogue for studies of colloquial speech, dialects, or vocalization patterns.
Content moderation – quickly get a transcript to scan for prohibited language while keeping the video private.
Translation workflows – produce a Japanese transcript and then feed it to a local LLM for English subtitles in a single command.

Installation quick‑start (Windows example)

Download the latest release .exe from the Releases page.
Run it – it creates a local Python environment, installs PyTorch, FFmpeg, and the required models (≈3 GB on first run).
Launch the shortcut → GUI appears. Drag‑and‑drop a video, pick a mode, and click Start.

Alternative: use the provided Colab or Kaggle notebooks (badge at the top of the README) for a zero‑install, cloud‑free run.

Example command line

# basic transcription, default balanced mode
whisperjav video.mp4

# faster mode, conservative VAD (good for very noisy clips)
whisperjav video.mp4 --mode faster --sensitivity conservative

# two‑pass ensemble with custom merge strategy
whisperjav video.mp4 \
    --ensemble \
    --pass1-pipeline anime-whisper --pass2-pipeline qwen \
    --merge-strategy smart_merge

The resulting subtitle file (video.srt) appears next to the original video.

Limitations & caveats

Domain‑specific – the pipelines and filters are tuned for JAV‑style audio; results on other Japanese content (news, podcasts) may be poorer.
Model size & VRAM – high‑accuracy modes (e.g., whisper‑large‑v2, Qwen‑3‑ASR 1.7 B) need ≥8 GB GPU memory; otherwise the tool falls back to CPU which is much slower.
Hallucination cannot be fully eliminated – even with defensive decoding, occasional invented lines or duplicated text can appear, especially on extremely low‑quality recordings.
Legal/ethical responsibility – the software merely processes media you already own; distribution of copyrighted or non‑consensual content remains the user’s responsibility.

Where to learn more

Documentation site – English: https://meizhong986.github.io/WhisperJAV/ – contains detailed guides, benchmark tables, and troubleshooting tips.
GitHub repo – https://github.com/meizhong986/WhisperJAV – source code, issue tracker, and release assets.
Community notebooks – Colab and Kaggle badges at the top of the README for quick experimentation.

WhisperJAV is a concrete example of how a general‑purpose speech‑to‑text model (Whisper) can be wrapped with domain‑aware preprocessing, segmentation, and post‑processing to make it usable on a notoriously difficult audio domain. It stays entirely on‑device, giving privacy‑conscious users a practical way to generate Japanese subtitles for long‑form, noisy video content.

TL;DR – Install the Windows .exe (or run the Colab notebook), drop a JAV file into the GUI, pick a mode (e.g., balanced), and you’ll get a clean .srt subtitle file without ever uploading the video anywhere.

meizhong986/WhisperJAV